To answer your CSV import config file questions first:
interval. The description for the interval config option in the example CSV
import config file reads:
# If there is no mapped interval field how will the interval field be
# determined for the imported records. Available options are:
# derive - Derive the interval field from the timestamp of successive
# records. This setting is best used when there are no
missing
# records from period being imported. Missing records will
cause
# the interval field to be incorrectly calculated for some
# records.
# conf - Use the interval setting from weewx.conf. This setting is
best
# used if the records to be imported have been produced by
WeeWX
# or some other means with the same archive interval as set
in
# weewx.conf on this machine.
# x - Use a fixed interval of 'x' minutes for every record where
'x'
# is a number. This setting is best used if the records to be
# imported are equally spaced in time but there are some
missing
# records.
#
# Note: If there is a mapped interval field this setting will be
ignored.
# Format is:
# interval = (derive | conf | x)
# Default is derive.
If you use interval = derive the interval field in a given imported record
will be calculated as the difference in timestamps of the current record
being imported and the previous record that was imported. Given your
irregular spacing in time of records to be imported interval = derive will
not give you the desired result (1 minute). interval = conf will set the
interval field for all imported records to the archive_interval setting
from [StdArchive] in weewx.conf. If archive_interval is set to one minute
you will get the desired result. If you set interval = 1 the interval field
for all imported records will be set to one minute, again the desired
result. Sounds like you should be using interval = 1 or maybe interval =
conf.
is_cumulative. is_cumulative doesn't expect data in a particular format,
rather the is_cumulative setting (True or False) specifies the format the
data should be in (cumulative or per-archive period respectively). Again
reading the description in the example CSV import config file:
# is_cumulative - Config option specifying whether the CSV
# field being mapped is cumulative,
# e.g: dayrain. Optional, default value is
# False.
WeeWX records the likes of rain and lightning strike count etc as
per-archive period values rather than as cumulative values; however, many
stations provides such data as cumulative values. The is_cumulative setting
is used to tell weectl import that the field being imported is a cumulative
value (is_cumulative = True) in which case weectl import calculates the
value for the field concerned as the difference between successive imported
values. If the field being imported is already a per-archive period value
setting is_cumulative = False will cause the value to be imported without
change. It sounds as though in your case the original rainfall data was
cumulative but the data you intend to import has been converted to
per-period values. In that case you should be using is_cumulative = False.
(An aside: in your case with imported records needing to be slotted in
between existing archive records you have to import per-period rainfall
data; if you were to import using cumulative values the resulting
per-archive period value calculated by weectl import would be incorrect due
to the interspersed existing archive records).
You should also keep in mind the manner in which weectl import (nee
weeimport) operates. It was originally designed to import large blocks of
archive record data from other sources; eg other weather station software
or (shudder) WeatherUnderground etc. Essentially archive record like data
was read from a source, the data was massaged into a format compatible with
a WeeWX archive record and the re-formatted data written to WeeWX archive.
Initially weeimport only allowed new archive records to be added; existing
records could not be modified and any attempt to do so resulted in the
imported record being discarded. As of WeeWX v5.1.0 existing records can be
updated, but the update performed is the overwriting of an existing archive
record with the imported archive record. The existing archive record is not
merged with the imported record and the merged record written to the WeeWX
database. So what? If you want to use weectl import to import data into
existing archive records, each record in your import source data (in your
case your CSV data) needs to include the existing data for all fields as
well as the updated rainfall data. You then need to use the (as yet
undocumented) --update command line option when running weectl import.
Alternatively, you could avoid using the --update command line option by
first deleting the records to be imported from the WeeWX database and then
importing your merged/updated CSV data.
Gary
On Friday, 14 March 2025 at 02:58:14 UTC+10 Andrew McGinnis wrote:
> I had a rain event on Monday that saw 0.66" fall on my station. All of
> the consoles that receive direct from the sensor (433mhz) recorded the same
> 0.66".
>
> My weewx install uses MQTTSubscribe for all of my observations, via
> rtl_433's mqtt output. It recorded 0.48" in total.
>
> My sensor is a cumulative output, so the mqtt topic would read something
> like this
> rtl_433/pi4b8/devices/Acurite-5n1/A/3030/rain_mm 510.28601
> rtl_433/pi4b8/devices/Acurite-5n1/A/3030/rain_mm 511.5
> rtl_433/pi4b8/devices/Acurite-5n1/A/3030/rain_mm 516.2
>
> The rain observation is set this way:
> [[topics]]
> [[[rtl_433/pi4b8/devices/Acurite-5n1/A/3030/rain_mm]]]
> name = rain
> contains_total = true
>
> I dumped out the timeframe of the rain event from my weewx.sdb, and looked
> at where the missing data might've been:
> echo "select datetime(dateTime, 'unixepoch', 'localtime'), rain from
> archive where datetime(dateTime,'unixepoch','localtime') > "2025-03-10
> 09:00:00" limit 120;" | sqlite3 weewx.sdb
> ...
> 2025-03-10 09:49:00|
> 2025-03-10 09:50:00|0.0199988188976378
> 2025-03-10 09:51:00|
> 2025-03-10 09:52:00|0.0199996062992129
> 2025-03-10 09:53:00|0.0499996062992106
> 2025-03-10 09:54:00|0.0599996062992136
> 2025-03-10 09:55:00|0.0300007874015751
> 2025-03-10 09:56:00|0.0499984251968492
> 2025-03-10 09:57:00|0.0400007874015758
> 2025-03-10 09:58:00|
> 2025-03-10 09:59:00|0.0100000000000008
> 2025-03-10 10:00:00|0.0
> 2025-03-10 10:01:00|0.0
> 2025-03-10 10:02:00|0.00999999999999852
> 2025-03-10 10:03:00|
> 2025-03-10 10:04:00|0.0
> ...
>
> I've pieced together the 15 missing data points to get that day's total
> rainfall to balance with the other consoles.
>
> So with a csv file that looks like this:
> Time,Rain
> 3/10/2025 09:14:00,0.00999881889763705
> 3/10/2025 09:19:00,0.0100000000000008
> 3/10/2025 09:24:00,0.0100000000000008
> 3/10/2025 09:31:00,0.0
> 3/10/2025 09:38:00,0.0100000000000008
> 3/10/2025 09:42:00,0.0199984251968514
> 3/10/2025 09:49:00,0.04
> 3/10/2025 09:51:00,0.0200000000000016
> 3/10/2025 10:03:00,0.0
> 3/10/2025 10:06:00,0.0200000000000015
> 3/10/2025 10:09:00,0.0
> 3/10/2025 10:12:00,0.0
> 3/10/2025 10:17:00,0.0
> 3/10/2025 10:18:00,0.00999999999999852
> 3/10/2025 10:19:00,0.0
>
> I'm not confident that my csv.conf import file is correctly configured.
> Here's what I've got:
> source = CSV
>
> [CSV]
> file = /var/lib/weewx/rain.csv
> delimiter = ','
> decimal = '.'
> interval = conf
> qc = True
> calc_missing = True
> ignore_invalid_data = True
> tranche = 250
> raw_datetime_format = %m/%d/%Y %H:%M:%S
> wind_direction = -360,360
>
> # is_cumulative - Config option specifying whether the CSV
> # field being mapped is cumulative,
> # e.g: dayrain. Optional, default value is
> # False.
> [[FieldMap]]
> [[[dateTime]]]
> source_field = Time
> unit = unix_epoch
> [[[rain]]]
> source_field = Rain
> unit = mm
> is_cumulative = False
>
> I'm unsure of the interval = conf setting. My weewx.conf uses a 60-second
> archive_interval, and the missing data points are what would've been
> recorded in those missing periods... Or should it be interval = x (in my
> case, 1 minute)?
>
> For [[FieldMap]] > [[[rain]]] > is_cumulative, I'm unsure what it should
> be, and subsequently what my csv data should be. My data points are the
> values of what fell in those minute intervals, the current.[running total]
> at each of those periods. is is_cumulative expecting data that is a
> running total (510.2,511.5,516.2), or just the delta (1.3,0.7,etc)?
>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/weewx-user/23be3120-1e80-4e80-93c4-4249428864ecn%40googlegroups.com.
