I just setup a WeeWX instance with your import config file in your first
post but using rain = discrete and a csv data file using the two records in
your last post. I ran wee_import using these files and --dry-run and
--verbose and received somewhat different output to what you posted, in my
case there were warnings of missing fields in the csv data file:
$ sudo /home/weewx/bin/wee_import --import-config=/var/tmp/csv.conf
--verbose
/home/weewx/bin/wee_import:719: DeprecationWarning: The distutils package
is deprecated and slated for removal in Python 3.12. Use setuptools or
check PEP 632 for potential alternatives
from distutils.version import StrictVersion
Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
A CSV import from source file '/var/tmp/remy.csv' has been requested.
The following options will be used:
config=/home/weewx/weewx.conf, import-config=/var/tmp/csv.conf
source=/var/tmp/remy.csv, from=None, to=None
dry-run=None, calc_missing=True, ignore_invalid_data=True
tranche=250, interval=conf, date/time_string_format=%Y-%m-%d %H:%M:%S
delimiter=',', rain=discrete, wind_direction=[-360.0, 360.0]
UV=True, radiation=True
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
Starting import ...
Obtaining raw import data for period 1 ...
The following imported field-to-WeeWX field map will be used:
source field 'dateTime' in units 'unix_epoch' --> WeeWX field
'dateTime'
source field 'barometer' in units 'hPa' --> WeeWX field 'barometer'
source field 'inTemp' in units 'degree_C' --> WeeWX field 'inTemp'
source field 'outTemp' in units 'degree_C' --> WeeWX field 'outTemp'
source field 'inHumidity' in units 'percent' --> WeeWX field
'inHumidity'
source field 'outHumidity' in units 'percent' --> WeeWX field
'outHumidity'
source field 'windSpeed' in units 'km_per_hour' --> WeeWX field
'windSpeed'
source field 'windDir' in units 'degree_compass' --> WeeWX field
'windDir'
source field 'windGust' in units 'km_per_hour' --> WeeWX field
'windGust'
source field 'rainRate' in units 'mm_per_hour' --> WeeWX field
'rainRate'
source field 'rain' in units 'mm' --> WeeWX field 'rain'
source field 'dewpoint' in units 'degree_C' --> WeeWX field 'dewpoint'
source field 'windchill' in units 'degree_C' --> WeeWX field
'windchill'
source field 'radiation' in units 'watt_per_meter_squared' --> WeeWX
field 'radiation'
source field 'UV' in units 'uv_index' --> WeeWX field 'UV'
source field 'extraTemp1' in units 'degree_C' --> WeeWX field
'extraTemp1'
Raw import data read successfully for period 1.
Mapping raw import data for period 1 ...
Warning: Import field 'inTemp' is mapped to WeeWX field 'inTemp' but the
import field 'inTemp' could not be found in one or more records.
WeeWX field 'inTemp' will be set to 'None' in these records.
Warning: Import field 'inHumidity' is mapped to WeeWX field 'inHumidity'
but the
import field 'inHumidity' could not be found in one or more
records.
WeeWX field 'inHumidity' will be set to 'None' in these records.
Warning: Import field 'radiation' is mapped to WeeWX field 'radiation' but
the
import field 'radiation' could not be found in one or more records.
WeeWX field 'radiation' will be set to 'None' in these records.
Warning: Import field 'UV' is mapped to WeeWX field 'UV' but the
import field 'UV' could not be found in one or more records.
WeeWX field 'UV' will be set to 'None' in these records.
Warning: Import field 'extraTemp1' is mapped to WeeWX field 'extraTemp1'
but the
import field 'extraTemp1' could not be found in one or more
records.
WeeWX field 'extraTemp1' will be set to 'None' in these records.
Mapped 2 records.
Raw import data mapped successfully for period 1.
Saving mapped data to archive for period 1 ...
2 records identified for import.
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)? y
Unique records processed: 2; Last timestamp: 2023-08-28 18:35:00 AEST
(1693211700)
Mapped data saved to archive successfully ""for period 1.
Calculating missing derived observations ...
Processing record: 2; Last record: 2023-08-29 00:00:00 AEST (1693231200)
Recalculating daily summaries...
Finished recalculating daily summaries
Finished calculating missing derived observations
Finished import
2 records were processed and 2 unique records imported in 2.87 seconds.
Those records with a timestamp already in the archive will not have been
imported. Confirm successful import in the WeeWX log file.
These warnings are inconsequential for your import, but they make me wonder
if you have posted the correct output for the command/config/data
specified? Importantly, in my test the data was imported as expected:
$ sqlite3 /home/weewx/archive/weewx.sdb
SQLite version 3.37.2 2022-01-06 13:25:41
Enter ".help" for usage hints.
sqlite> SELECT dateTime,datetime(dateTime, 'unixepoch',
'localtime'),outTemp,rain,rainRate FROM archive ORDER BY dateTime DESC
LIMIT 6;
1693340400|2023-08-30 06:20:00|32.1130389185491|0.0|0.0
1693340100|2023-08-30 06:15:00|32.3124634933352|0.0|0.0
1693339800|2023-08-30 06:10:00|32.5203063775446|0.0|0.0
1693339500|2023-08-30 06:05:00|32.6532104624026|0.0|0.0
1693211700|2023-08-28 18:35:00|60.8|0.0078740157480315|0.2362204722
1693211400|2023-08-28 18:30:00|63.68|0.0118110236220472|0.14173228332
sqlite>
(note the fields are in the native units (US customary) for my database ie
rain and rainRate values are in inches and inches/hour)
Also, the command and response specified in your posts include the use of
the --dry-run command line option; if --dry-run is used data will never be
saved to database - that is the point of --dry-run. Have you run the same
command without --dry-run? If not I suggest you try that and if it does not
import correctly post the exact command line used and the exact output
received. If you have run wee_import without --dry-run please post the
corresponding wee_import output.
One other thing, how are you determining that the data was not imported?
The most reliable way is to directly query the database, if you are relying
upon WeeWX generated output this can be unreliable/untimely due to caching,
delays in generation etc.
Gary
On Tuesday, 29 August 2023 at 22:34:05 UTC+10 remy.l...@gmail.com wrote:
> Hi Gary,
> I followed your advice:
>
> 1/ I now have in the rain field of my csv file the amount of rain that
> fell over the previous 5 minutes.
> 2/ I changed the config file so that rain=discrete is now the config value
>
> Unfortunately, the integration of the csv file in the weewx database with
> this new configuration did not solve the problem, the rain and rainRate
> fields systematically remain at zero while the integration of all the other
> fields works wonderfully!
> Do you think there is something wrong or is it a weewx bug?
>
>
> *My new configuration file :*
> * rain = discrete*
> wind_direction = 0,360
>
> [[FieldMap]]
> dateTime = dateTime, unix_epoch
> barometer = barometer, hPa
> inTemp = inTemp, degree_C
> outTemp = outTemp, degree_C
> inHumidity = inHumidity, percent
> outHumidity = outHumidity, percent
> windSpeed = windSpeed, km_per_hour
> windDir = windDir, degree_compass
> windGust = windGust, km_per_hour
>
> * rainRate = rainRate, mm_per_hour rain = rain, mm*
> dewpoint = dewpoint, degree_C
> windchill = windchill, degree_C
> radiation = radiation, watt_per_meter_squared
> UV = UV, uv_index
> extraTemp1 = extraTemp1, degree_C
>
> *Example of CSV file with rain & rainRate *:
>
> dateTime,outTemp,outHumidity,dewpoint,appTemp,barometer,windchill,windSpeed,windDir,windGust
> *,rain,rainRate*
> 2023-08-28 18:30:00,17.6,66.0,11.3,13.8,1009.5,17.6,3.8,0,6.1,*0.3,3.6*
> 2023-08-28 18:35:00,16.0,80.0,12.6,13.7,1009.4,16.0,3.8,1,7.1,*0.2,6.0*
>
>
> *Result of the test :*
> sudo wee_import --config=/etc/weewx/weewx.conf
> --import-config=/home/pi/Documents/Bresser7in1/csv-Bresser.conf --dry-run
> --verbose
> Using WeeWX configuration file /etc/weewx/weewx.conf
> Starting wee_import...
> A CSV import from source file '/home/pi/Documents/Bresser7in1/data5.csv'
> has been requested.
> The following options will be used:
> config=/etc/weewx/weewx.conf,
> import-config=/home/pi/Documents/Bresser7in1/csv-Bresser.conf
> source=/home/pi/Documents/Bresser7in1/data5.csv, from=None, to=None
> dry-run=True, calc_missing=True, ignore_invalid_data=True
> tranche=250, interval=conf, date/time_string_format=%Y-%m-%d %H:%M:%S
> delimiter=',', *rain=discrete,* wind_direction=[0.0, 360.0]
>
> UV=True, radiation=True
> Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
> Destination table 'archive' unit system is '0x01' (US).
> Missing derived observations will be calculated.
> This is a dry run, imported data will not be saved to archive.
> Starting dry run import ...
> Obtaining raw import data for period 1 ...
> The following imported field-to-WeeWX field map will be used:
> source field 'dateTime' in units 'unix_epoch' --> WeeWX field
> 'dateTime'
> source field 'barometer' in units 'hPa' --> WeeWX field 'barometer'
> source field 'inTemp' in units 'degree_C' --> WeeWX field 'inTemp'
> source field 'outTemp' in units 'degree_C' --> WeeWX field 'outTemp'
> source field 'inHumidity' in units 'percent' --> WeeWX field
> 'inHumidity'
> source field 'outHumidity' in units 'percent' --> WeeWX field
> 'outHumidity'
> source field 'windSpeed' in units 'km_per_hour' --> WeeWX field
> 'windSpeed'
> source field 'windDir' in units 'degree_compass' --> WeeWX field
> 'windDir'
> source field 'windGust' in units 'km_per_hour' --> WeeWX field
> 'windGust'
>
>
> * source field 'rainRate' in units 'mm_per_hour' --> WeeWX field
> 'rainRate' source field 'rain' in units 'mm' --> WeeWX field 'rain'*
> source field 'dewpoint' in units 'degree_C' --> WeeWX field 'dewpoint'
> source field 'windchill' in units 'degree_C' --> WeeWX field
> 'windchill'
> source field 'radiation' in units 'watt_per_meter_squared' --> WeeWX
> field 'radiation'
> source field 'UV' in units 'uv_index' --> WeeWX field 'UV'
> source field 'extraTemp1' in units 'degree_C' --> WeeWX field
> 'extraTemp1'
> source field 'extraTemp2' in units 'degree_C' --> WeeWX field
> 'extraTemp2'
> source field 'extraTemp3' in units 'degree_C' --> WeeWX field
> 'extraTemp3'
> source field 'extraTemp4' in units 'degree_C' --> WeeWX field
> 'extraTemp4'
> source field 'extraTemp6' in units 'degree_C' --> WeeWX field
> 'extraTemp6'
> source field 'extraHumid1' in units 'percent' --> WeeWX field
> 'extraHumid1'
> source field 'extraHumid2' in units 'percent' --> WeeWX field
> 'extraHumid2'
> source field 'extraHumid3' in units 'percent' --> WeeWX field
> 'extraHumid3'
> source field 'extraHumid4' in units 'percent' --> WeeWX field
> 'extraHumid4'
> *Raw import data read successfully for period 1.*
> Mapping raw import data for period 1 ...
> Mapped 1 records.
> *Raw import data mapped successfully for period 1.*
> 1 records identified for import.
> Unique records processed: 1; Last timestamp: 2023-08-29 14:30:00 CEST
> (1693312200)
>
> Finished dry run import
> 1 records were processed and 1 unique records would have been imported.
>
> *Rémy LAVABRE*
>
>
> Le dim. 27 août 2023 à 05:02, gjr80 <gjrod...@gmail.com> a écrit :
>
>> If you are importing a single record and using rain = cumulative the
>> imported rain and rainRate values will always be zero, this is expected
>> behaviour. The reason being that WeeWX records rainfall data on a per
>> archive interval basis; in other words each archive record contains the
>> rainfall that was recorded in that archive period, for example in the last
>> five minutes if using a five minute archive interval. When the import
>> source rainfall data is cumulative or a running total wee_import
>> calculates the rain field value as the difference in this cumulative or
>> running total value between successive records. The If there is only one
>> record then this difference cannot be calculated and hence rain is
>> recorded as zero. Likewise for rainRate; if there is no rain data then
>> there can be no rainRate.
>>
>> The use of the rain config option in a CSV import config file is covered
>> here <http://weewx.com/docs.html/latest/utilities.htm#csv_rain> in the
>> Utilities
>> Guide <http://weewx.com/docs.html/latest/utilities.htm>.
>>
>> If you must import a single record at a time you must convert your source
>> rain data to be a per-record value and import using rain = discrete.
>> wee_import was not designed with the regular import of single records in
>> mind, rather it was designed for the bulk import of days/months/years of
>> data.
>>
>> Gary
>> On Sunday, 27 August 2023 at 04:34:16 UTC+10 remy.l...@gmail.com wrote:
>>
>>>
>>>
>>>
>>> *Good morning,I am using the wee_import.py utility to integrate AWEKAS
>>> data into my weewx database.command executed: *sudo wee_import
>>> --config=/etc/weewx/weewx.conf
>>> --import-config=/home/pi/Documents/Bresser7in1/csv-Bresser.conf --dry-run
>>> --verbose
>>>
>>> *My CSVFile :*
>>>
>>>
>>> dateTime,outTemp,outHumidity,dewpoint,appTemp,barometer,windchill,windSpeed,windDir,windGust,
>>> *rain,rainRate*,UV,radiation,inTemp,inHumidity,extraTemp1
>>> 2023-08-26 16:35:00,23.6,61.0,15.7,18.4,1010.6,23.6,3.2,214,3.9,*6.9,0,*
>>> 1.1,101.0,26.1,53.0,24.9,22.0
>>>
>>>
>>>
>>> *My wee_import configuration file:*
>>>
>>> source = CSV
>>>
>>>
>>> ##############################################################################
>>>
>>> [CSV]
>>> file = /home/pi/Documents/Bresser7in1/data5.csv
>>> delimiter = ','
>>> decimal = '.'
>>> interval = conf
>>> qc = True
>>> calc_missing = True
>>> ignore_invalid_data = True
>>> tranche = 250
>>> UV_sensor = True
>>> solar_sensor = True
>>> raw_datetime_format = %Y-%m-%d %H:%M:%S
>>>
>>> rain = cumulative
>>> wind_direction = 0,360
>>>
>>>
>>> [[FieldMap]]
>>> dateTime = dateTime, unix_epoch
>>> barometer = barometer, hPa
>>> inTemp = inTemp, degree_C
>>> outTemp = outTemp, degree_C
>>> inHumidity = inHumidity, percent
>>> outHumidity = outHumidity, percent
>>> windSpeed = windSpeed, km_per_hour
>>> windDir = windDir, degree_compass
>>> windGust = windGust, km_per_hour
>>>
>>> * rainRate = rainRate, mm_per_hour rain = rain, mm*
>>> dewpoint = dewpoint, degree_C
>>> windchill = windchill, degree_C
>>> radiation = radiation, watt_per_meter_squared
>>> UV = UV, uv_index
>>> extraTemp1 = extraTemp1, degree_C
>>> extraTemp2 = extraTemp2, degree_C
>>> extraTemp3 = extraTemp3, degree_C
>>> extraTemp4 = extraTemp4, degree_C
>>> extraTemp6 = extraTemp6, degree_C
>>> extraHumid1 = extraHumid1, percent
>>> extraHumid2 = extraHumid2, percent
>>> extraHumid3 = extraHumid3, percent
>>> extraHumid4 = extraHumid4, percent
>>>
>>>
>>>
>>> *Response from WeeWX :*
>>>
>>> Using WeeWX configuration file /etc/weewx/weewx.conf
>>> Starting wee_import...
>>> A CSV import from source file '/home/pi/Documents/Bresser7in1/data5.csv'
>>> has been requested.
>>> The following options will be used:
>>> config=/etc/weewx/weewx.conf,
>>> import-config=/home/pi/Documents/Bresser7in1/csv-Bresser.conf
>>> source=/home/pi/Documents/Bresser7in1/data5.csv, from=None, to=None
>>> dry-run=True, calc_missing=True, ignore_invalid_data=True
>>> tranche=250, interval=conf, date/time_string_format=%Y-%m-%d
>>> %H:%M:%S
>>> delimiter=',', rain=cumulative, wind_direction=[0.0, 360.0]
>>> UV=True, radiation=True
>>> Using database binding 'wx_binding', which is bound to database
>>> 'weewx.sdb'
>>> Destination table 'archive' unit system is '0x01' (US).
>>> Missing derived observations will be calculated.
>>> This is a dry run, imported data will not be saved to archive.
>>> Starting dry run import ...
>>> Obtaining raw import data for period 1 ...
>>> The following imported field-to-WeeWX field map will be used:
>>> source field 'dateTime' in units 'unix_epoch' --> WeeWX field
>>> 'dateTime'
>>> source field 'barometer' in units 'hPa' --> WeeWX field 'barometer'
>>> source field 'inTemp' in units 'degree_C' --> WeeWX field 'inTemp'
>>> source field 'outTemp' in units 'degree_C' --> WeeWX field 'outTemp'
>>> source field 'inHumidity' in units 'percent' --> WeeWX field
>>> 'inHumidity'
>>> source field 'outHumidity' in units 'percent' --> WeeWX field
>>> 'outHumidity'
>>> source field 'windSpeed' in units 'km_per_hour' --> WeeWX field
>>> 'windSpeed'
>>> source field 'windDir' in units 'degree_compass' --> WeeWX field
>>> 'windDir'
>>> source field 'windGust' in units 'km_per_hour' --> WeeWX field
>>> 'windGust'
>>>
>>> * source field 'rainRate' in units 'mm_per_hour' --> WeeWX field
>>> 'rainRate' source field 'rain' in units 'mm' --> WeeWX field 'rain'*
>>> source field 'dewpoint' in units 'degree_C' --> WeeWX field
>>> 'dewpoint'
>>> source field 'windchill' in units 'degree_C' --> WeeWX field
>>> 'windchill'
>>> source field 'radiation' in units 'watt_per_meter_squared' -->
>>> WeeWX field 'radiation'
>>> source field 'UV' in units 'uv_index' --> WeeWX field 'UV'
>>> source field 'extraTemp1' in units 'degree_C' --> WeeWX field
>>> 'extraTemp1'
>>> source field 'extraTemp2' in units 'degree_C' --> WeeWX field
>>> 'extraTemp2'
>>> source field 'extraTemp3' in units 'degree_C' --> WeeWX field
>>> 'extraTemp3'
>>> source field 'extraTemp4' in units 'degree_C' --> WeeWX field
>>> 'extraTemp4'
>>> source field 'extraTemp6' in units 'degree_C' --> WeeWX field
>>> 'extraTemp6'
>>> source field 'extraHumid1' in units 'percent' --> WeeWX field
>>> 'extraHumid1'
>>> source field 'extraHumid2' in units 'percent' --> WeeWX field
>>> 'extraHumid2'
>>> source field 'extraHumid3' in units 'percent' --> WeeWX field
>>> 'extraHumid3'
>>> source field 'extraHumid4' in units 'percent' --> WeeWX field
>>> 'extraHumid4'
>>> Raw import data read successfully for period 1.
>>> Mapping raw import data for period 1 ...
>>> Mapped 1 records.
>>> Raw import data mapped successfully for period 1.
>>> 1 records identified for import.
>>> Unique records processed: 1; Last timestamp: 2023-08-26 16:55:00 CEST
>>> (1693061700)
>>> Finished dry run import
>>> 1 records were processed and 1 unique records would have been imported.
>>>
>>>
>>>
>>>
>>> *Everything seems to be correct.However, when I run the command without
>>> the --dry run option, ALL the fields are correctly entered in the weewx
>>> database except SYSTEMATIC "rain" and "rainRate" which always remain at 0!I
>>> don't understand why these two fields are never updated in weewx database.
>>> Here for example rain is at 6.9 (since 00H00) and in the database rain
>>> remains at 0 (option cumulative actived). Same thing for rainRate! And
>>> these are the only two fields that pose a problem!Is it a wee_import.py
>>> bug? Does anyone have a solution... Thanks a lot!*
>>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "weewx-user" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/weewx-user/nWTJnx2UNlU/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> weewx-user+...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/weewx-user/0a5739b7-436c-4ccf-b170-576969ae6357n%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/weewx-user/0a5739b7-436c-4ccf-b170-576969ae6357n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/80122a9d-9540-41a1-b6b4-77f50fb4448dn%40googlegroups.com.
