set value to None if it does not exist, as opposed to 0 that does exist as value 0 ⊣GE⊢
> On 15 Mar 2025, at 4:47 am, Andrew McGinnis <[email protected]> wrote: > >> If you want to use weectl import to import data into existing archive >> records, each record in your import source data (in your case your CSV data) >> needs to include the existing data for all fields as well as the updated >> rainfall data. You then need to use the (as yet undocumented) --update >> command line option when running weectl import. > > So rather than just the records I'm missing, I need to go from the first > missing record to the last missing record, inclusive of all in-between > records? I can easily do that, just not 100% on whether it's required. > >> Alternatively, you could avoid using the --update command line option by >> first deleting the records to be imported from the WeeWX database and then >> importing your merged/updated CSV data. > > If you'll save me the research, how do I delete specific records/periods, > either a single period record, or a range? I just recently noticed that the > soilMoist1 observation, on my weewx testing install, recorded about 84 hours > of very wrong values, that my production weewx install didn't (both getting > the data from the same mqtt topic). I know how to drop entire observation > columns via weectl database drop-columns NAME, but in this case it's just > range of periods within the column that need purging. Or is it as simple as, > zeroing out the values in a range via sqlite> update archive set > soilMoist1=0.0 where dateTime > {startepochtime} and dateTime <= > {endepochtime]; ? > > On Fri, Mar 14, 2025 at 1:17 AM gjr80 <[email protected] > <mailto:[email protected]>> wrote: >> To answer your CSV import config file questions first: >> >> interval. The description for the interval config option in the example CSV >> import config file reads: >> >> # If there is no mapped interval field how will the interval field be >> # determined for the imported records. Available options are: >> # derive - Derive the interval field from the timestamp of successive >> # records. This setting is best used when there are no missing >> # records from period being imported. Missing records will >> cause >> # the interval field to be incorrectly calculated for some >> # records. >> # conf - Use the interval setting from weewx.conf. This setting is >> best >> # used if the records to be imported have been produced by >> WeeWX >> # or some other means with the same archive interval as set in >> # weewx.conf on this machine. >> # x - Use a fixed interval of 'x' minutes for every record where >> 'x' >> # is a number. This setting is best used if the records to be >> # imported are equally spaced in time but there are some >> missing >> # records. >> # >> # Note: If there is a mapped interval field this setting will be ignored. >> # Format is: >> # interval = (derive | conf | x) >> # Default is derive. >> >> If you use interval = derive the interval field in a given imported record >> will be calculated as the difference in timestamps of the current record >> being imported and the previous record that was imported. Given your >> irregular spacing in time of records to be imported interval = derive will >> not give you the desired result (1 minute). interval = conf will set the >> interval field for all imported records to the archive_interval setting from >> [StdArchive] in weewx.conf. If archive_interval is set to one minute you >> will get the desired result. If you set interval = 1 the interval field for >> all imported records will be set to one minute, again the desired result. >> Sounds like you should be using interval = 1 or maybe interval = conf. >> >> is_cumulative. is_cumulative doesn't expect data in a particular format, >> rather the is_cumulative setting (True or False) specifies the format the >> data should be in (cumulative or per-archive period respectively). Again >> reading the description in the example CSV import config file: >> >> # is_cumulative - Config option specifying whether the CSV >> # field being mapped is cumulative, >> # e.g: dayrain. Optional, default value is >> # False. >> >> WeeWX records the likes of rain and lightning strike count etc as >> per-archive period values rather than as cumulative values; however, many >> stations provides such data as cumulative values. The is_cumulative setting >> is used to tell weectl import that the field being imported is a cumulative >> value (is_cumulative = True) in which case weectl import calculates the >> value for the field concerned as the difference between successive imported >> values. If the field being imported is already a per-archive period value >> setting is_cumulative = False will cause the value to be imported without >> change. It sounds as though in your case the original rainfall data was >> cumulative but the data you intend to import has been converted to >> per-period values. In that case you should be using is_cumulative = False. >> (An aside: in your case with imported records needing to be slotted in >> between existing archive records you have to import per-period rainfall >> data; if you were to import using cumulative values the resulting >> per-archive period value calculated by weectl import would be incorrect due >> to the interspersed existing archive records). >> >> You should also keep in mind the manner in which weectl import (nee >> weeimport) operates. It was originally designed to import large blocks of >> archive record data from other sources; eg other weather station software or >> (shudder) WeatherUnderground etc. Essentially archive record like data was >> read from a source, the data was massaged into a format compatible with a >> WeeWX archive record and the re-formatted data written to WeeWX archive. >> Initially weeimport only allowed new archive records to be added; existing >> records could not be modified and any attempt to do so resulted in the >> imported record being discarded. As of WeeWX v5.1.0 existing records can be >> updated, but the update performed is the overwriting of an existing archive >> record with the imported archive record. The existing archive record is not >> merged with the imported record and the merged record written to the WeeWX >> database. So what? If you want to use weectl import to import data into >> existing archive records, each record in your import source data (in your >> case your CSV data) needs to include the existing data for all fields as >> well as the updated rainfall data. You then need to use the (as yet >> undocumented) --update command line option when running weectl import. >> Alternatively, you could avoid using the --update command line option by >> first deleting the records to be imported from the WeeWX database and then >> importing your merged/updated CSV data. >> >> Gary >> On Friday, 14 March 2025 at 02:58:14 UTC+10 Andrew McGinnis wrote: >>> I had a rain event on Monday that saw 0.66" fall on my station. All of the >>> consoles that receive direct from the sensor (433mhz) recorded the same >>> 0.66". >>> >>> My weewx install uses MQTTSubscribe for all of my observations, via >>> rtl_433's mqtt output. It recorded 0.48" in total. >>> >>> My sensor is a cumulative output, so the mqtt topic would read something >>> like this >>> rtl_433/pi4b8/devices/Acurite-5n1/A/3030/rain_mm 510.28601 >>> rtl_433/pi4b8/devices/Acurite-5n1/A/3030/rain_mm 511.5 >>> rtl_433/pi4b8/devices/Acurite-5n1/A/3030/rain_mm 516.2 >>> >>> The rain observation is set this way: >>> [[topics]] >>> [[[rtl_433/pi4b8/devices/Acurite-5n1/A/3030/rain_mm]]] >>> name = rain >>> contains_total = true >>> >>> I dumped out the timeframe of the rain event from my weewx.sdb, and looked >>> at where the missing data might've been: >>> echo "select datetime(dateTime, 'unixepoch', 'localtime'), rain from >>> archive where datetime(dateTime,'unixepoch','localtime') > "2025-03-10 >>> 09:00:00" limit 120;" | sqlite3 weewx.sdb >>> ... >>> 2025-03-10 09:49:00| >>> 2025-03-10 09:50:00|0.0199988188976378 >>> 2025-03-10 09:51:00| >>> 2025-03-10 09:52:00|0.0199996062992129 >>> 2025-03-10 09:53:00|0.0499996062992106 >>> 2025-03-10 09:54:00|0.0599996062992136 >>> 2025-03-10 09:55:00|0.0300007874015751 >>> 2025-03-10 09:56:00|0.0499984251968492 >>> 2025-03-10 09:57:00|0.0400007874015758 >>> 2025-03-10 09:58:00| >>> 2025-03-10 09:59:00|0.0100000000000008 >>> 2025-03-10 10:00:00|0.0 >>> 2025-03-10 10:01:00|0.0 >>> 2025-03-10 10:02:00|0.00999999999999852 >>> 2025-03-10 10:03:00| >>> 2025-03-10 10:04:00|0.0 >>> ... >>> >>> I've pieced together the 15 missing data points to get that day's total >>> rainfall to balance with the other consoles. >>> >>> So with a csv file that looks like this: >>> Time,Rain >>> 3/10/2025 09:14:00,0.00999881889763705 >>> 3/10/2025 09:19:00,0.0100000000000008 >>> 3/10/2025 09:24:00,0.0100000000000008 >>> 3/10/2025 09:31:00,0.0 >>> 3/10/2025 09:38:00,0.0100000000000008 >>> 3/10/2025 09:42:00,0.0199984251968514 >>> 3/10/2025 09:49:00,0.04 >>> 3/10/2025 09:51:00,0.0200000000000016 >>> 3/10/2025 10:03:00,0.0 >>> 3/10/2025 10:06:00,0.0200000000000015 >>> 3/10/2025 10:09:00,0.0 >>> 3/10/2025 10:12:00,0.0 >>> 3/10/2025 10:17:00,0.0 >>> 3/10/2025 10:18:00,0.00999999999999852 >>> 3/10/2025 10:19:00,0.0 >>> >>> I'm not confident that my csv.conf import file is correctly configured. >>> Here's what I've got: >>> source = CSV >>> >>> [CSV] >>> file = /var/lib/weewx/rain.csv >>> delimiter = ',' >>> decimal = '.' >>> interval = conf >>> qc = True >>> calc_missing = True >>> ignore_invalid_data = True >>> tranche = 250 >>> raw_datetime_format = %m/%d/%Y %H:%M:%S >>> wind_direction = -360,360 >>> >>> # is_cumulative - Config option specifying whether the CSV >>> # field being mapped is cumulative, >>> # e.g: dayrain. Optional, default value is >>> # False. >>> [[FieldMap]] >>> [[[dateTime]]] >>> source_field = Time >>> unit = unix_epoch >>> [[[rain]]] >>> source_field = Rain >>> unit = mm >>> is_cumulative = False >>> >>> I'm unsure of the interval = conf setting. My weewx.conf uses a 60-second >>> archive_interval, and the missing data points are what would've been >>> recorded in those missing periods... Or should it be interval = x (in my >>> case, 1 minute)? >>> >>> For [[FieldMap]] > [[[rain]]] > is_cumulative, I'm unsure what it should >>> be, and subsequently what my csv data should be. My data points are the >>> values of what fell in those minute intervals, the current.[running total] >>> at each of those periods. is is_cumulative expecting data that is a running >>> total (510.2,511.5,516.2), or just the delta (1.3,0.7,etc)? >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "weewx-user" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/weewx-user/1KX3-8lbU34/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected] >> <mailto:[email protected]>. >> To view this discussion visit >> https://groups.google.com/d/msgid/weewx-user/23be3120-1e80-4e80-93c4-4249428864ecn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/weewx-user/23be3120-1e80-4e80-93c4-4249428864ecn%40googlegroups.com?utm_medium=email&utm_source=footer>. > > > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion visit > https://groups.google.com/d/msgid/weewx-user/CAH3iRs1GPVORbFDZ5qs0zg2uj0PEHBL2v-g-C8w0L6Qy5i_bEw%40mail.gmail.com > > <https://groups.google.com/d/msgid/weewx-user/CAH3iRs1GPVORbFDZ5qs0zg2uj0PEHBL2v-g-C8w0L6Qy5i_bEw%40mail.gmail.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/weewx-user/800BEFD5-CED4-4682-B52F-6A9F9F512CC2%40geddy.au.
