Unsubscribe doesn’t work, unsubscribe me On Fri, Mar 14, 2025 at 8:08 PM Graham Eddy <[email protected]> wrote:
> set value to None if it does not exist, as opposed to 0 that does exist as > value 0 > *⊣GE⊢* > > On 15 Mar 2025, at 4:47 am, Andrew McGinnis <[email protected]> > wrote: > > If you want to use weectl import to import data into existing archive >> records, each record in your import source data (in your case your CSV >> data) needs to include the existing data for all fields as well as the >> updated rainfall data. You then need to use the (as yet undocumented) >> --update command line option when running weectl import. > > > So rather than just the records I'm *missing*, I need to go from the > first missing record to the last missing record, inclusive of all > in-between records? I can easily do that, just not 100% on whether it's > required. > > Alternatively, you could avoid using the --update command line option by >> first deleting the records to be imported from the WeeWX database and then >> importing your merged/updated CSV data. > > > If you'll save me the research, how do I delete specific records/periods, > either a single period record, or a range? I just recently noticed that the > soilMoist1 observation, on my weewx testing install, recorded about 84 > hours of very wrong values, that my production weewx install didn't (both > getting the data from the same mqtt topic). I know how to drop entire > observation columns via weectl database drop-columns NAME, but in this > case it's just range of periods within the column that need purging. Or is > it as simple as, zeroing out the values in a range via sqlite> update > archive set soilMoist1=0.0 where dateTime > {startepochtime} and dateTime > <= {endepochtime]; ? > > On Fri, Mar 14, 2025 at 1:17 AM gjr80 <[email protected]> wrote: > >> To answer your CSV import config file questions first: >> >> interval. The description for the interval config option in the example >> CSV import config file reads: >> >> # If there is no mapped interval field how will the interval field be >> # determined for the imported records. Available options are: >> # derive - Derive the interval field from the timestamp of >> successive >> # records. This setting is best used when there are no >> missing >> # records from period being imported. Missing records will >> cause >> # the interval field to be incorrectly calculated for some >> # records. >> # conf - Use the interval setting from weewx.conf. This setting >> is best >> # used if the records to be imported have been produced by >> WeeWX >> # or some other means with the same archive interval as >> set in >> # weewx.conf on this machine. >> # x - Use a fixed interval of 'x' minutes for every record >> where 'x' >> # is a number. This setting is best used if the records to >> be >> # imported are equally spaced in time but there are some >> missing >> # records. >> # >> # Note: If there is a mapped interval field this setting will be >> ignored. >> # Format is: >> # interval = (derive | conf | x) >> # Default is derive. >> >> If you use interval = derive the interval field in a given imported >> record will be calculated as the difference in timestamps of the current >> record being imported and the previous record that was imported. Given your >> irregular spacing in time of records to be imported interval = derive >> will not give you the desired result (1 minute). interval = conf will >> set the interval field for all imported records to the archive_interval >> setting from [StdArchive] in weewx.conf. If archive_interval is set to >> one minute you will get the desired result. If you set interval = 1 the >> interval field for all imported records will be set to one minute, again >> the desired result. Sounds like you should be using interval = 1 or >> maybe interval = conf. >> >> is_cumulative. is_cumulative doesn't expect data in a particular format, >> rather the is_cumulative setting (True or False) specifies the format >> the data should be in (cumulative or per-archive period respectively). >> Again reading the description in the example CSV import config file: >> >> # is_cumulative - Config option specifying whether the >> CSV >> # field being mapped is cumulative, >> # e.g: dayrain. Optional, default value >> is >> # False. >> >> WeeWX records the likes of rain and lightning strike count etc as >> per-archive period values rather than as cumulative values; however, many >> stations provides such data as cumulative values. The is_cumulative >> setting is used to tell weectl import that the field being imported is a >> cumulative value (is_cumulative = True) in which case weectl import >> calculates the value for the field concerned as the difference between >> successive imported values. If the field being imported is already a >> per-archive period value setting is_cumulative = False will cause the >> value to be imported without change. It sounds as though in your case the >> original rainfall data was cumulative but the data you intend to import has >> been converted to per-period values. In that case you should be using >> is_cumulative >> = False. (An aside: in your case with imported records needing to be >> slotted in between existing archive records you have to import per-period >> rainfall data; if you were to import using cumulative values the resulting >> per-archive period value calculated by weectl import would be incorrect >> due to the interspersed existing archive records). >> >> You should also keep in mind the manner in which weectl import (nee >> weeimport) operates. It was originally designed to import large blocks >> of archive record data from other sources; eg other weather station >> software or (shudder) WeatherUnderground etc. Essentially archive record >> like data was read from a source, the data was massaged into a format >> compatible with a WeeWX archive record and the re-formatted data written to >> WeeWX archive. Initially weeimport only allowed new archive records to >> be added; existing records could not be modified and any attempt to do so >> resulted in the imported record being discarded. As of WeeWX v5.1.0 >> existing records can be updated, but the update performed is the >> overwriting of an existing archive record with the imported archive record. >> The existing archive record is not merged with the imported record and the >> merged record written to the WeeWX database. So what? If you want to use >> weectl >> import to import data into existing archive records, each record in your >> import source data (in your case your CSV data) needs to include the >> existing data for all fields as well as the updated rainfall data. You then >> need to use the (as yet undocumented) --update command line option when >> running weectl import. Alternatively, you could avoid using the --update >> command >> line option by first deleting the records to be imported from the WeeWX >> database and then importing your merged/updated CSV data. >> >> Gary >> On Friday, 14 March 2025 at 02:58:14 UTC+10 Andrew McGinnis wrote: >> >>> I had a rain event on Monday that saw 0.66" fall on my station. All of >>> the consoles that receive direct from the sensor (433mhz) recorded the same >>> 0.66". >>> >>> My weewx install uses MQTTSubscribe for all of my observations, via >>> rtl_433's mqtt output. It recorded 0.48" in total. >>> >>> My sensor is a cumulative output, so the mqtt topic would read something >>> like this >>> rtl_433/pi4b8/devices/Acurite-5n1/A/3030/rain_mm 510.28601 >>> rtl_433/pi4b8/devices/Acurite-5n1/A/3030/rain_mm 511.5 >>> rtl_433/pi4b8/devices/Acurite-5n1/A/3030/rain_mm 516.2 >>> >>> The rain observation is set this way: >>> [[topics]] >>> [[[rtl_433/pi4b8/devices/Acurite-5n1/A/3030/rain_mm]]] >>> name = rain >>> contains_total = true >>> >>> I dumped out the timeframe of the rain event from my weewx.sdb, and >>> looked at where the missing data might've been: >>> echo "select datetime(dateTime, 'unixepoch', 'localtime'), rain from >>> archive where datetime(dateTime,'unixepoch','localtime') > "2025-03-10 >>> 09:00:00" limit 120;" | sqlite3 weewx.sdb >>> ... >>> 2025-03-10 09:49:00| >>> 2025-03-10 09:50:00|0.0199988188976378 >>> 2025-03-10 09:51:00| >>> 2025-03-10 09:52:00|0.0199996062992129 >>> 2025-03-10 09:53:00|0.0499996062992106 >>> 2025-03-10 09:54:00|0.0599996062992136 >>> 2025-03-10 09:55:00|0.0300007874015751 >>> 2025-03-10 09:56:00|0.0499984251968492 >>> 2025-03-10 09:57:00|0.0400007874015758 >>> 2025-03-10 09:58:00| >>> 2025-03-10 09:59:00|0.0100000000000008 >>> 2025-03-10 10:00:00|0.0 >>> 2025-03-10 10:01:00|0.0 >>> 2025-03-10 10:02:00|0.00999999999999852 >>> 2025-03-10 10:03:00| >>> 2025-03-10 10:04:00|0.0 >>> ... >>> >>> I've pieced together the 15 missing data points to get that day's total >>> rainfall to balance with the other consoles. >>> >>> So with a csv file that looks like this: >>> Time,Rain >>> 3/10/2025 09:14:00,0.00999881889763705 >>> 3/10/2025 09:19:00,0.0100000000000008 >>> 3/10/2025 09:24:00,0.0100000000000008 >>> 3/10/2025 09:31:00,0.0 >>> 3/10/2025 09:38:00,0.0100000000000008 >>> 3/10/2025 09:42:00,0.0199984251968514 >>> 3/10/2025 09:49:00,0.04 >>> 3/10/2025 09:51:00,0.0200000000000016 >>> 3/10/2025 10:03:00,0.0 >>> 3/10/2025 10:06:00,0.0200000000000015 >>> 3/10/2025 10:09:00,0.0 >>> 3/10/2025 10:12:00,0.0 >>> 3/10/2025 10:17:00,0.0 >>> 3/10/2025 10:18:00,0.00999999999999852 >>> 3/10/2025 10:19:00,0.0 >>> >>> I'm not confident that my csv.conf import file is correctly configured. >>> Here's what I've got: >>> source = CSV >>> >>> [CSV] >>> file = /var/lib/weewx/rain.csv >>> delimiter = ',' >>> decimal = '.' >>> interval = conf >>> qc = True >>> calc_missing = True >>> ignore_invalid_data = True >>> tranche = 250 >>> raw_datetime_format = %m/%d/%Y %H:%M:%S >>> wind_direction = -360,360 >>> >>> # is_cumulative - Config option specifying whether the >>> CSV >>> # field being mapped is cumulative, >>> # e.g: dayrain. Optional, default value >>> is >>> # False. >>> [[FieldMap]] >>> [[[dateTime]]] >>> source_field = Time >>> unit = unix_epoch >>> [[[rain]]] >>> source_field = Rain >>> unit = mm >>> is_cumulative = False >>> >>> I'm unsure of the interval = conf setting. My weewx.conf uses a >>> 60-second archive_interval, and the missing data points are what would've >>> been recorded in those missing periods... Or should it be interval = x >>> (in my case, 1 minute)? >>> >>> For [[FieldMap]] > [[[rain]]] > is_cumulative, I'm unsure what it >>> should be, and subsequently what my csv data should be. My data points are >>> the values of what fell in those minute intervals, the current.[running >>> total] at each of those periods. is is_cumulative expecting data that >>> is a running total (510.2,511.5,516.2), or just the delta (1.3,0.7,etc)? >>> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "weewx-user" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/weewx-user/1KX3-8lbU34/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion visit >> https://groups.google.com/d/msgid/weewx-user/23be3120-1e80-4e80-93c4-4249428864ecn%40googlegroups.com >> <https://groups.google.com/d/msgid/weewx-user/23be3120-1e80-4e80-93c4-4249428864ecn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion visit > https://groups.google.com/d/msgid/weewx-user/CAH3iRs1GPVORbFDZ5qs0zg2uj0PEHBL2v-g-C8w0L6Qy5i_bEw%40mail.gmail.com > <https://groups.google.com/d/msgid/weewx-user/CAH3iRs1GPVORbFDZ5qs0zg2uj0PEHBL2v-g-C8w0L6Qy5i_bEw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion visit > https://groups.google.com/d/msgid/weewx-user/800BEFD5-CED4-4682-B52F-6A9F9F512CC2%40geddy.au > <https://groups.google.com/d/msgid/weewx-user/800BEFD5-CED4-4682-B52F-6A9F9F512CC2%40geddy.au?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/weewx-user/CACyhQtQKfGzHHvhmHGsSvNd6%2BCf5ftwFyRkaSL4iCVveX24fQg%40mail.gmail.com.
