Gary Removed usUnits and interval from the csv.conf and it worked like a champ.
Thanks Rich On Tuesday, June 18, 2019 at 11:29:55 PM UTC-4, gjr80 wrote: > Rich, > > Try deleting the mapping for usUnits from the field map. Long story > short, it does not make much sense to be importing/converting a unit system > code when units are specified/implied for all the fields. I know this isn't > covered in the wee_import guide, will fix that up later. > > Gary > > On Wednesday, 19 June 2019 13:10:13 UTC+10, rich T wrote: >> >> Gary >> >> I have the raw_datetime_format as "%m/%d/%Y %H:%M", but forgot to >> include it in previous post. Now I receive the following error: >> >> pi@raspberrypi:/home/weewx/bin $ sudo ./wee_import >> --import-config=/home/pi/Desktop/clog/csv.conf --dry-run >> Starting wee_import... >> A CSV import from source file >> '/home/pi/Desktop/clog/archive_updated1.csv' has been requested. >> Using database binding 'wx_binding', which is bound to database >> 'weewx.sdb' >> Destination table 'archive' unit system is '0x01' (US). >> Missing derived observations will be calculated. >> All WeeWX UV fields will be set to None. >> All WeeWX radiation fields will be set to None. >> This is a dry run, imported data will not be saved to archive. >> **** Unable to map source data. >> **** Field 'units' not found in source data. >> **** Nothing done, exiting. >> >> Thanks >> Rich >> >> On Tuesday, June 18, 2019 at 10:46:20 PM UTC-4, gjr80 wrote: >>> >>> The error in the original post is due to one of the [[FieldMap]] >>> mappings not having WeeWX unit name. You do not need to include mappings >>> for all fields in the CSV source, only those you wish to import, but those >>> you do include must be of the form: >>> >>> WeeWX field = CSV field name, WeeWX unit name >>> >>> You are correct in that the raw_datetime_format setting determines how >>> the CSV date-time field is decoded. For the data shown I would expect >>> something like (untested): >>> >>> raw_datetime_format = "%m/%d/%Y %H:%M" >>> >>> this assumes the date is in the format month/day/year, just transpose %m >>> and %d if it id day/month/year. >>> >>> Gary >>> >>> On Wednesday, 19 June 2019 12:33:03 UTC+10, rich T wrote: >>>> >>>> I could be wrong, but the raw_datetime_format is the datetime format in >>>> the CSV file (i.e. %m/%d%Y). The dateTime = timestamp, unix_epoch in the >>>> CSV Map basically means convert the timestamp column into unix_epoch time >>>> format. >>>> >>>> On Tuesday, June 18, 2019 at 10:06:49 PM UTC-4, Andrew Milner wrote: >>>>> >>>>> shouldn't it just be dateTime = timestamp and the format of timestamp >>>>> be specified in raw_datetime_format ?? >>>>> >>>>> >>>>> >>>>> On Wednesday, 19 June 2019 04:32:30 UTC+3, rich T wrote: >>>>>> >>>>>> I'm trying to import a large CSV file into the database and I'm >>>>>> receiving the following error the the dry run. >>>>>> >>>>>> pi@raspberrypi:/home/weewx/bin $ sudo ./wee_import >>>>>> --import-config=/home/pi/Desktop/clog/csv.conf --dry-run >>>>>> Starting wee_import... >>>>>> A CSV import from source file >>>>>> '/home/pi/Desktop/clog/archive_updated1.csv' has been requested. >>>>>> Using database binding 'wx_binding', which is bound to database >>>>>> 'weewx.sdb' >>>>>> Destination table 'archive' unit system is '0x01' (US). >>>>>> Missing derived observations will be calculated. >>>>>> All WeeWX UV fields will be set to None. >>>>>> All WeeWX radiation fields will be set to None. >>>>>> This is a dry run, imported data will not be saved to archive. >>>>>> Traceback (most recent call last): >>>>>> File "./wee_import", line 703, in <module> >>>>>> main() >>>>>> File "./wee_import", line 653, in main >>>>>> source_obj.run() >>>>>> File "/home/weewx/bin/weeimport/weeimport.py", line 350, in run >>>>>> _mapped_data = self.mapRawData(_raw_data, self.archive_unit_sys) >>>>>> File "/home/weewx/bin/weeimport/weeimport.py", line 721, in >>>>>> mapRawData >>>>>> self.map[_field]['units'], >>>>>> KeyError: 'units' >>>>>> >>>>>> Here is the mapping in the csv.conf: >>>>>> >>>>>> [[FieldMap]] >>>>>> dateTime = timestamp, unix_epoch >>>>>> usUnits = units >>>>>> interval = interval >>>>>> barometer = barometer, inHg >>>>>> pressure = pressure >>>>>> altimeter = altimeter >>>>>> inTemp = inTemp >>>>>> outTemp = outTemp, degree_F >>>>>> inHumidity = inHumidity >>>>>> outHumidity = outHumidity, percent >>>>>> windSpeed = windSpeed, mile_per_hour >>>>>> windDir = windDir, degree_compass >>>>>> windGust = windGust, mile_per_hour >>>>>> windGustDir = windGustDir, degree_compass >>>>>> rainRate = rainRate, inch_per_hour >>>>>> rain = rain, inch >>>>>> dewpoint = dewpoint >>>>>> windchill = windchill >>>>>> heatindex = heatindex >>>>>> ET = >>>>>> radiation = >>>>>> UV = >>>>>> rxCheckPercent = rxCheckPercent >>>>>> >>>>>> Here is a small portion of the CSV file: >>>>>> >>>>>> >>>>>> timestamp,units,interval,barometer,pressure,altimeter,inTemp,outTemp,inHumidity,outHumidity,windSpeed,windDir,windGust,windGustDir,rainRate,rain,dewpoint,windchill,heatindex,ET,radiation,UV,extraTemp1,extraTemp2,extraTemp3,soilTemp1,soilTemp2,soilTemp3,soilTemp4,leafTemp1,leafTemp2,extraHumid1,extraHumid2,soilMoist1,soilMoist2,soilMoist3,soilMoist4,leafWet1,leafWet2,rxCheckPercent,txBatteryStatus,consBatteryVoltage,hail,hailRate,heatingTemp,heatingVoltage,supplyVoltage,referenceVoltage,windBatteryStatus,rainBatteryStatus,outTempBatteryStatus,inTempBatteryStatus >>>>>> 04/09/2018 >>>>>> 00:05,1,5,30.101,,,77.2,24.2,20,62,0,,0,,0,0,13.08666521,24.2,24.2,0,,,,,,,,,,,,,,,,,,,,99.9375,,,,,,,,,,,, >>>>>> 04/09/2018 >>>>>> 00:10,1,5,30.102,,,77,24,20,63,0,,0,,0,0,13.25922371,24,24,0,,,,,,,,,,,,,,,,,,,,97.375,,,,,,,,,,,, >>>>>> 04/09/2018 >>>>>> 00:15,1,5,30.108,,,76.9,23.8,20,63,0,,0,,0,0,13.06932323,23.8,23.8,0,,,,,,,,,,,,,,,,,,,,96.52083333,,,,,,,,,,,, >>>>>> 04/09/2018 >>>>>> 00:20,1,5,30.11,,,76.7,23.7,20,63,0,,0,,0,0,12.97437126,23.7,23.7,0,,,,,,,,,,,,,,,,,,,,97.375,,,,,,,,,,,, >>>>>> 04/09/2018 >>>>>> 00:25,1,5,30.112,,,76.6,23.5,20,63,0,,0,,0,0,12.78446384,23.5,23.5,0,,,,,,,,,,,,,,,,,,,,77.72916667,,,,,,,,,,,, >>>>>> 04/09/2018 >>>>>> 00:30,1,5,30.11,,,76.4,23.5,21,63,0,,0,,0,0,12.78446384,23.5,23.5,0,,,,,,,,,,,,,,,,,,,,99.9375,,,,,,,,,,,, >>>>>> 04/09/2018 >>>>>> 00:35,1,5,30.111,,,76.7,23.3,20,64,0,,3,225,0,0,12.95043454,23.3,23.3,0,,,,,,,,,,,,,,,,,,,,99.08333333,,,,,,,,,,,, >>>>>> 04/09/2018 >>>>>> 00:40,1,5,30.106,,,76.9,23.3,20,64,0,,0,,0,0,12.95043454,23.3,23.3,0,,,,,,,,,,,,,,,,,,,,99.08333333,,,,,,,,,,,, >>>>>> 04/09/2018 >>>>>> 00:45,1,5,30.11,,,77,23.1,20,64,0,,0,,0,0,12.76018685,23.1,23.1,0,,,,,,,,,,,,,,,,,,,,99.08333333,,,,,,,,,,,, >>>>>> 04/09/2018 >>>>>> 00:50,1,5,30.114,,,77,23,20,65,0,,0,,0,0,13.01554258,23,23,0,,,,,,,,,,,,,,,,,,,,100,,,,,,,,,,,, >>>>>> >>>>>> >>>>>> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/d80c1854-1f72-40e6-befa-c82d8e10279a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
