Your import config file is telling wee_import the input rainfall field contains cumulative data, so wee_import uses the difference between the current row rainfall data and the previous row rainfall data to calculate the WeeWX field rain. For the first row there is no previous row so rain is set to None (also note that the 1January 2019 00:00 rain value is considered by WeeWX as containing 2018 data (ie data covering the period from 23:59:00 to 23:59:59 + 1 second on 31 December 2018). The second row is is the first record containing 2019 data. It should calculate the rain field as being 0.2 - 0.2 = 0.0, though from your database extract it appears to have been set to None. This could be a corner case involving the change of year, but as far as I know wee_import should handle change of year without problem. I will need to look further at the wee_import code. The third row will calculate rain as 0 - 0.2 = -0.2 which of course is a nonsense and I believe wee_import would assume there has been a counter overflow and set rain to 0.0. For the fourth row rain is calculated as 0 - 0 = 0.0 which is what you see.
So other than the minor discrepancy at row two (for rain 0 and None are essentially treated the same in WeeWX output) I think wee_import is working as expected given the import data and import config settings. Analysis of a larger sample of data involving non-zero rainfall data may throw up more information. Gary On Sunday, 15 January 2023 at 08:40:05 UTC+10 [email protected] wrote: > I think I should add a period to all values, not a comma. > > David Schulz schrieb am Samstag, 14. Januar 2023 um 23:35:21 UTC+1: > >> There should actually be values in there. >> >> echo "SELECT dateTime, rain FROM archive" | sqlite3 >> /var/lib/weewx/weewx.sdb >> 1546297200| >> 1546297260| >> 1546297320|0.0 >> 1546297380|0.0 >> >> DatumZeit;Temp;Feuchte;Luftdruck;Regen;Wind;Richtung;Windboeen >> 01.01.2019 00:00;6,9;97;1026,6;0,2;4,7;242;9,7 >> 01.01.2019 00:01;6,9;97;1026,6;0,2;4,7;251;7,9 >> 01.01.2019 00:02;6,9;97;1026,6;0;4,7;250;7,9 >> 01.01.2019 00:03;6,9;97;1026,6;0;4,7;245;4,7 >> >> >> gjr80 schrieb am Samstag, 14. Januar 2023 um 23:19:12 UTC+1: >> >>> OK. Let's go straight to the horses head, what happens when you manually >>> query the database archive table for the period concerned. Do each of the >>> rows show the correct rainfall data or is it missing? >>> >>> Gary >>> >>> On Sunday, 15 January 2023 at 08:06:20 UTC+10 [email protected] >>> wrote: >>> >>>> Thanks for the info. But for testing I delete every time the whole DB >>>> and the NOAA reports. I have a Weewx test environment specifically for >>>> this. >>>> >>>> gjr80 schrieb am Samstag, 14. Januar 2023 um 23:03:41 UTC+1: >>>> >>>>> You need to be very careful when using the NOAA format reports to >>>>> verify the import of historical data, this is because only the current >>>>> month and current year reports are updated each report cycle. Historical >>>>> reports are not updated so will not reflect and changes to earlier data. >>>>> The sample data you provided is from January 2019 so I am guessing you >>>>> did >>>>> not import into the current month and year on your system. Try deleting >>>>> all >>>>> of your NOAA format reports from your WeeWX system, they will all be >>>>> re-generated on the next report cycle based on the current database data. >>>>> >>>>> Gary >>>>> >>>>> On Sunday, 15 January 2023 at 07:06:08 UTC+10 [email protected] >>>>> wrote: >>>>> >>>>>> [image: 2023-01-14 22_05_47-Blub.png] >>>>>> >>>>>> David Schulz schrieb am Samstag, 14. Januar 2023 um 22:04:28 UTC+1: >>>>>> >>>>>>> wee_import --verbose --import-config=/etc/weewx/import/csv.conf >>>>>>> /usr/share/weewx/wee_import:719: DeprecationWarning: The distutils >>>>>>> package is deprecated and slated for removal in Python 3.12. Use >>>>>>> setuptools >>>>>>> or check PEP 632 for potential alternatives >>>>>>> from distutils.version import StrictVersion >>>>>>> Using WeeWX configuration file /etc/weewx/weewx.conf >>>>>>> Starting wee_import... >>>>>>> A CSV import from source file '/etc/weewx/import/EXP201901A.CSV' has >>>>>>> been requested. >>>>>>> The following options will be used: >>>>>>> config=/etc/weewx/weewx.conf, >>>>>>> import-config=/etc/weewx/import/csv.conf >>>>>>> source=/etc/weewx/import/EXP201901A.CSV, from=None, to=None >>>>>>> dry-run=None, calc_missing=True, ignore_invalid_data=True >>>>>>> tranche=1000, interval=derive, date/time_string_format=%d.%m.%Y >>>>>>> %H:%M >>>>>>> delimiter=';', rain=cumulative, wind_direction=[-360.0, 360.0] >>>>>>> UV=False, radiation=False >>>>>>> Using database binding 'wx_binding', which is bound to database >>>>>>> 'weewx.sdb' >>>>>>> Destination table 'archive' unit system is '0x01' (US). >>>>>>> Missing derived observations will be calculated. >>>>>>> All WeeWX UV fields will be set to None. >>>>>>> All WeeWX radiation fields will be set to None. >>>>>>> Starting import ... >>>>>>> Obtaining raw import data for period 1 ... >>>>>>> The following imported field-to-WeeWX field map will be used: >>>>>>> source field 'DatumZeit' in units 'unix_epoch' --> WeeWX field >>>>>>> 'dateTime' >>>>>>> source field 'Temp' in units 'degree_C' --> WeeWX field >>>>>>> 'outTemp' >>>>>>> source field 'Feuchte' in units 'percent' --> WeeWX field >>>>>>> 'outHumidity' >>>>>>> source field 'Luftdruck' in units 'hPa' --> WeeWX field >>>>>>> 'barometer' >>>>>>> source field 'Regen' in units 'mm' --> WeeWX field 'rain' >>>>>>> source field 'Wind' in units 'km_per_hour' --> WeeWX field >>>>>>> 'windSpeed' >>>>>>> source field 'Richtung' in units 'degree_compass' --> WeeWX >>>>>>> field 'windDir' >>>>>>> source field 'Windboeen' in units 'km_per_hour' --> WeeWX field >>>>>>> 'windGust' >>>>>>> Raw import data read successfully for period 1. >>>>>>> Mapping raw import data for period 1 ... >>>>>>> Mapped 1440 records. >>>>>>> Raw import data mapped successfully for period 1. >>>>>>> Saving mapped data to archive for period 1 ... >>>>>>> 1440 records identified for import. >>>>>>> Proceeding will save all imported records in the WeeWX archive. >>>>>>> Are you sure you want to proceed (y/n)? y >>>>>>> Unique records processed: 1440; Last timestamp: 2019-01-01 23:59:00 >>>>>>> CET (1546383540) >>>>>>> Mapped data saved to archive successfully ""for period 1. >>>>>>> Calculating missing derived observations ... >>>>>>> Processing record: 1440; Last record: 2019-01-02 00:00:00 CET >>>>>>> (1546383600) >>>>>>> Recalculating daily summaries... >>>>>>> Records processed: 1000; time: 2019-01-01 16:39:00 CET (1546357140) >>>>>>> Finished recalculating daily summaries >>>>>>> Finished calculating missing derived observations >>>>>>> Finished import >>>>>>> 1440 records were processed and 1440 unique records imported in >>>>>>> 40.49 seconds. >>>>>>> Those records with a timestamp already in the archive will not have >>>>>>> been >>>>>>> imported. Confirm successful import in the WeeWX log file. >>>>>>> >>>>>>> David Schulz schrieb am Samstag, 14. Januar 2023 um 22:03:34 UTC+1: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> i want to import some old weatherdata via csv file, but after the >>>>>>>> import, the raindata is missing. i tried the import with "rain = >>>>>>>> cumulative >>>>>>>> / discrete". >>>>>>>> >>>>>>>> Here my config: >>>>>>>> >>>>>>>> source = CSV >>>>>>>> ####################################### >>>>>>>> [CSV] >>>>>>>> file = /etc/weewx/import/EXP201901A.CSV >>>>>>>> delimiter = ; >>>>>>>> interval = derive >>>>>>>> qc = True >>>>>>>> calc_missing = True >>>>>>>> ignore_invalid_data = True >>>>>>>> tranche = 1000 >>>>>>>> UV_sensor = False >>>>>>>> solar_sensor = False >>>>>>>> raw_datetime_format = %d.%m.%Y %H:%M >>>>>>>> # rain = (discrete | cumulative) >>>>>>>> rain = cumulative >>>>>>>> wind_direction = -360,360 >>>>>>>> [[FieldMap]] >>>>>>>> dateTime = DatumZeit, unix_epoch >>>>>>>> outTemp = Temp, degree_C >>>>>>>> outHumidity = Feuchte, percent >>>>>>>> barometer = Luftdruck, hPa >>>>>>>> rain = Regen, mm >>>>>>>> windSpeed = Wind, km_per_hour >>>>>>>> windDir = Richtung, degree_compass >>>>>>>> windGust = Windboeen, km_per_hour >>>>>>>> >>>>>>>> And here some sample data. >>>>>>>> >>>>>>>> DatumZeit;Temp;Feuchte;Luftdruck;Regen;Wind;Richtung;Windboeen >>>>>>>> 01.01.2019 00:00;6,9;97;1026,6;0,2;4,7;242;9,7 >>>>>>>> 01.01.2019 00:01;6,9;97;1026,6;0,2;4,7;251;7,9 >>>>>>>> 01.01.2019 00:02;6,9;97;1026,6;0,2;4,7;250;7,9 >>>>>>>> 01.01.2019 00:03;6,9;97;1026,6;0,2;4,7;245;4,7 >>>>>>>> 01.01.2019 00:04;6,9;97;1026,6;0;4,7;250;7,9 >>>>>>>> 01.01.2019 00:05;6,9;97;1026,5;0;4,7;234;7,9 >>>>>>>> 01.01.2019 00:06;6,9;97;1026,5;0;4,7;232;11,2 >>>>>>>> 01.01.2019 00:07;7;97;1026,5;0;4,7;234;4,7 >>>>>>>> 01.01.2019 00:08;7;97;1026,5;0;4,7;241;7,9 >>>>>>>> 01.01.2019 00:09;7;97;1026,5;0;4,7;240;6,5 >>>>>>>> >>>>>>> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/27450a8e-e476-4963-87c5-726672111743n%40googlegroups.com.
