Gary I have the raw_datetime_format as "%m/%d/%Y %H:%M", but forgot to include it in previous post. Now I receive the following error:
pi@raspberrypi:/home/weewx/bin $ sudo ./wee_import --import-config=/home/pi/Desktop/clog/csv.conf --dry-run Starting wee_import... A CSV import from source file '/home/pi/Desktop/clog/archive_updated1.csv' has been requested. Using database binding 'wx_binding', which is bound to database 'weewx.sdb' Destination table 'archive' unit system is '0x01' (US). Missing derived observations will be calculated. All WeeWX UV fields will be set to None. All WeeWX radiation fields will be set to None. This is a dry run, imported data will not be saved to archive. **** Unable to map source data. **** Field 'units' not found in source data. **** Nothing done, exiting. Thanks Rich On Tuesday, June 18, 2019 at 10:46:20 PM UTC-4, gjr80 wrote: > > The error in the original post is due to one of the [[FieldMap]] mappings > not having WeeWX unit name. You do not need to include mappings for all > fields in the CSV source, only those you wish to import, but those you do > include must be of the form: > > WeeWX field = CSV field name, WeeWX unit name > > You are correct in that the raw_datetime_format setting determines how > the CSV date-time field is decoded. For the data shown I would expect > something like (untested): > > raw_datetime_format = "%m/%d/%Y %H:%M" > > this assumes the date is in the format month/day/year, just transpose %m > and %d if it id day/month/year. > > Gary > > On Wednesday, 19 June 2019 12:33:03 UTC+10, rich T wrote: >> >> I could be wrong, but the raw_datetime_format is the datetime format in >> the CSV file (i.e. %m/%d%Y). The dateTime = timestamp, unix_epoch in the >> CSV Map basically means convert the timestamp column into unix_epoch time >> format. >> >> On Tuesday, June 18, 2019 at 10:06:49 PM UTC-4, Andrew Milner wrote: >>> >>> shouldn't it just be dateTime = timestamp and the format of timestamp be >>> specified in raw_datetime_format ?? >>> >>> >>> >>> On Wednesday, 19 June 2019 04:32:30 UTC+3, rich T wrote: >>>> >>>> I'm trying to import a large CSV file into the database and I'm >>>> receiving the following error the the dry run. >>>> >>>> pi@raspberrypi:/home/weewx/bin $ sudo ./wee_import >>>> --import-config=/home/pi/Desktop/clog/csv.conf --dry-run >>>> Starting wee_import... >>>> A CSV import from source file >>>> '/home/pi/Desktop/clog/archive_updated1.csv' has been requested. >>>> Using database binding 'wx_binding', which is bound to database >>>> 'weewx.sdb' >>>> Destination table 'archive' unit system is '0x01' (US). >>>> Missing derived observations will be calculated. >>>> All WeeWX UV fields will be set to None. >>>> All WeeWX radiation fields will be set to None. >>>> This is a dry run, imported data will not be saved to archive. >>>> Traceback (most recent call last): >>>> File "./wee_import", line 703, in <module> >>>> main() >>>> File "./wee_import", line 653, in main >>>> source_obj.run() >>>> File "/home/weewx/bin/weeimport/weeimport.py", line 350, in run >>>> _mapped_data = self.mapRawData(_raw_data, self.archive_unit_sys) >>>> File "/home/weewx/bin/weeimport/weeimport.py", line 721, in mapRawData >>>> self.map[_field]['units'], >>>> KeyError: 'units' >>>> >>>> Here is the mapping in the csv.conf: >>>> >>>> [[FieldMap]] >>>> dateTime = timestamp, unix_epoch >>>> usUnits = units >>>> interval = interval >>>> barometer = barometer, inHg >>>> pressure = pressure >>>> altimeter = altimeter >>>> inTemp = inTemp >>>> outTemp = outTemp, degree_F >>>> inHumidity = inHumidity >>>> outHumidity = outHumidity, percent >>>> windSpeed = windSpeed, mile_per_hour >>>> windDir = windDir, degree_compass >>>> windGust = windGust, mile_per_hour >>>> windGustDir = windGustDir, degree_compass >>>> rainRate = rainRate, inch_per_hour >>>> rain = rain, inch >>>> dewpoint = dewpoint >>>> windchill = windchill >>>> heatindex = heatindex >>>> ET = >>>> radiation = >>>> UV = >>>> rxCheckPercent = rxCheckPercent >>>> >>>> Here is a small portion of the CSV file: >>>> >>>> >>>> timestamp,units,interval,barometer,pressure,altimeter,inTemp,outTemp,inHumidity,outHumidity,windSpeed,windDir,windGust,windGustDir,rainRate,rain,dewpoint,windchill,heatindex,ET,radiation,UV,extraTemp1,extraTemp2,extraTemp3,soilTemp1,soilTemp2,soilTemp3,soilTemp4,leafTemp1,leafTemp2,extraHumid1,extraHumid2,soilMoist1,soilMoist2,soilMoist3,soilMoist4,leafWet1,leafWet2,rxCheckPercent,txBatteryStatus,consBatteryVoltage,hail,hailRate,heatingTemp,heatingVoltage,supplyVoltage,referenceVoltage,windBatteryStatus,rainBatteryStatus,outTempBatteryStatus,inTempBatteryStatus >>>> 04/09/2018 >>>> 00:05,1,5,30.101,,,77.2,24.2,20,62,0,,0,,0,0,13.08666521,24.2,24.2,0,,,,,,,,,,,,,,,,,,,,99.9375,,,,,,,,,,,, >>>> 04/09/2018 >>>> 00:10,1,5,30.102,,,77,24,20,63,0,,0,,0,0,13.25922371,24,24,0,,,,,,,,,,,,,,,,,,,,97.375,,,,,,,,,,,, >>>> 04/09/2018 >>>> 00:15,1,5,30.108,,,76.9,23.8,20,63,0,,0,,0,0,13.06932323,23.8,23.8,0,,,,,,,,,,,,,,,,,,,,96.52083333,,,,,,,,,,,, >>>> 04/09/2018 >>>> 00:20,1,5,30.11,,,76.7,23.7,20,63,0,,0,,0,0,12.97437126,23.7,23.7,0,,,,,,,,,,,,,,,,,,,,97.375,,,,,,,,,,,, >>>> 04/09/2018 >>>> 00:25,1,5,30.112,,,76.6,23.5,20,63,0,,0,,0,0,12.78446384,23.5,23.5,0,,,,,,,,,,,,,,,,,,,,77.72916667,,,,,,,,,,,, >>>> 04/09/2018 >>>> 00:30,1,5,30.11,,,76.4,23.5,21,63,0,,0,,0,0,12.78446384,23.5,23.5,0,,,,,,,,,,,,,,,,,,,,99.9375,,,,,,,,,,,, >>>> 04/09/2018 >>>> 00:35,1,5,30.111,,,76.7,23.3,20,64,0,,3,225,0,0,12.95043454,23.3,23.3,0,,,,,,,,,,,,,,,,,,,,99.08333333,,,,,,,,,,,, >>>> 04/09/2018 >>>> 00:40,1,5,30.106,,,76.9,23.3,20,64,0,,0,,0,0,12.95043454,23.3,23.3,0,,,,,,,,,,,,,,,,,,,,99.08333333,,,,,,,,,,,, >>>> 04/09/2018 >>>> 00:45,1,5,30.11,,,77,23.1,20,64,0,,0,,0,0,12.76018685,23.1,23.1,0,,,,,,,,,,,,,,,,,,,,99.08333333,,,,,,,,,,,, >>>> 04/09/2018 >>>> 00:50,1,5,30.114,,,77,23,20,65,0,,0,,0,0,13.01554258,23,23,0,,,,,,,,,,,,,,,,,,,,100,,,,,,,,,,,, >>>> >>>> >>>> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/97dff43d-7970-42b1-824f-4db6da8d389b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
# EXAMPLE CONFIGURATION FILE FOR IMPORTING FROM CSV FILES # # Copyright (c) 2009-2016 Tom Keffer <[email protected]> and Gary Roderick. # See the file LICENSE.txt for your rights. ############################################################################## # Specify the source. Available options are: # CSV - import obs from a single CSV format file # WU - import obs from a Weather Underground PWS history # Cumulus - import obs from a one or more Cumulus monthly log files # Format is: # source = (CSV | WU | Cumulus) source = CSV ############################################################################## [CSV] # Parameters used when importing from a CSV file # Path and name of our CSV source file. Format is: # file = full path and filename file = /home/pi/Desktop/clog/archive_updated1.csv # If there is no mapped interval field how will the interval field be # determined for the imported records. Available options are: # derive - Derive the interval field from the timestamp of successive # records. This setting is best used when the imported records # are equally spaced in time and there are no missing records. # conf - Use the interval setting from weewx.conf. This setting is # best used if the records to be imported have been produced by # WeeWX using the same archive interval as set in weewx.conf on # this machine. # x - Use a fixed interval of x minutes for every record. This # setting is best used if the records to be imported are # equally based in time but there are some missing records. # # Note: If there is a mapped interval field then this setting will be # ignored. # Format is: # interval = (derive | conf | x) interval = 5 # Should the [StdQC] max/min limits in weewx.conf be applied to the # imported data. This may be useful if the source has extreme values that # are clearly incorrect for some observations. Available options are: # True - weewx.conf [StdQC] max/min limits are applied. # False - weewx.conf [StdQC] max/min limits are not applied. # Format is: # qc = (True | False) qc = True # Should any missing derived observations be calculated from the imported # data if possible. Available options are: # True - Any missing derived observations are calculated. # False - Any missing derived observations are not calculated. # Format is: # calc_missing = (True | False) calc_missing = True # Specify how imported data fields that contain invalid data (eg a numeric # field containing non-numeric data) are handled. Available options are: # True - The invalid data is ignored, the WeeWX target field is set to # None and the import continues. # False - The import is halted. # Format is: # ignore_invalid_data = (True | False) # Default is True. ignore_invalid_data = True # Imported records are written to archive in transactions of tranche # records at a time. Increase for faster throughput, decrease to reduce # memory requirements. Format is: # tranche = x # where x is an integer tranche = 250 # Specify whether a UV sensor was used to produce any UV observations. # Available options are: # True - UV sensor was used and UV data will be imported. # False - UV sensor was not used and any UV data will not be imported. # UV fields will be set to None/NULL. # For a CSV import UV_sensor should be set to False if a UV sensor was # NOT present when the import data was created. Otherwise it may be set to # True or omitted. Format is: # UV_sensor = (True | False) UV_sensor = False # Specify whether a solar radiation sensor was used to produce any solar # radiation observations. Available options are: # True - Solar radiation sensor was used and solar radiation data will # be imported. # False - Solar radiation sensor was not used and any solar radiation # data will not be imported. radiation fields will be set to # None/NULL. # For a CSV import solar_sensor should be set to False if a solar radiation # sensor was NOT present when the import data was created. Otherwise it may # be set to True or omitted. Format is: # solar_sensor = (True | False) solar_sensor = False # Date-time format of CSV field from which the WeeWX archive record # dateTime field is to be extracted. wee_import first attempts to interpret # date/time info in this format, if this fails it then attempts to # interpret it as a timestamp and if this fails it then raises an error. # Uses Python strptime() format codes. # raw_datetime_format = Python strptime() format string #raw_datetime_format = %Y/%m/%d %H:%M raw_datetime_format = %m/%d/%Y %H:%M # Does the imported rain field represent the total rainfall since the last # record or a cumulative value. Available options are: # discrete - rain field represents total rainfall since last record # cumulative - rain field represents a cumulative rainfall reset at # midnight # rain = (discrete | cumulative) rain = cumulative # Lower and upper bounds for imported wind direction. It is possible, # particularly for a calculated direction, to have a value (eg -45) outside # of the WeeWX limits (0 to 360 inclusive). Format is: # # wind_direction = lower,upper # # where : # lower is the lower limit of acceptable wind direction in degrees # (may be negative) # upper is the upper limit of acceptable wind direction in degrees # # Imported values from lower to upper will be normalised to the range 0 to # 360. Values outside of the parameter range will be stored as None. # Default is -360,360. wind_direction = -360,360 # Map CSV record fields to WeeWX archive fields. Format is: # # weewx_archive_field_name = csv_field_name, weewx_unit_name # # where: # weewx_archive_field_name - An observation name in the WeeWX database # schema. # csv_field_name - The name of a field from the CSV file. # weewx_unit_name - The name of the units, as defined in WeeWX, # used by csv_field_name. wee_import will do # the necessary conversions to the unit system # used by the WeeWX archive. # For example, # outTemp = Temp, degree_C # would map the CSV field Temp, in degrees C, to the archive field outTemp. # # If a field mapping exists for the WeeWX usUnits archive field then the # units option may be omitted for each mapped field. # # WeeWX archive fields that do not exist in the CSV data may be omitted. # Any omitted fields that are derived (eg dewpoint) may be calculated # during import using the equivalent of the WeeWX StdWXCalculate service # through setting the calc-missing parameter above. [[FieldMap]] dateTime = timestamp, unix_epoch usUnits = units, US interval = interval, 5 barometer = barometer, inHg pressure = pressure, inHg altimeter = altimeter, inHg inTemp = inTemp, degree_F outTemp = outTemp, degree_F inHumidity = inHumidity, percent outHumidity = outHumidity, percent windSpeed = windSpeed, mile_per_hour windDir = windDir, degree_compass windGust = windGust, mile_per_hour windGustDir = windGustDir, degree_compass rainRate = rainRate, inch_per_hour rain = rain, inch dewpoint = dewpoint, degree_F windchill = windchill, degree_F heatindex = heatindex, degree_F ET = radiation = UV = rxCheckPercent = rxCheckPercent, percent
