Ok, then I will go for the third solution, and use the `wee_database --cal-missing` utility. Thank you so much for your support, I'll let you know about the result.
Il giorno lunedì 15 agosto 2022 alle 22:46:53 UTC+2 gjr80 ha scritto: > Yes, that explains why those five derived obs were not calculated; > wee_import uses [StdWXCalculate] [[Calculations]] to calculate derived > obs and in effect the fields in the data being imported are 'hardware' > sourced obs. So setting an obs to 'hardware' in [StWXCalculate] > [[Calculations]] would prevent those derived obs from being calculated > when the import is performed. > > There are a few possible solutions. One solution is to manually > re-calculate the missing obs - though this is not really a practical > solution. The second is as you propose; delete the imported records and > re-import with updated [StdWXCalculate] settings. The third solution is > to use the wee_database utility with the --calc-missing option > <http://weewx.com/docs/utilities.htm#Action_--calc-missing> and updated > [StdWXCalculate] settings. wee_import and wee_database --calc-missing use > the same internal routines to calculate derived obs so the results should > be identical. wee_database --calc-missing will recalculate derived obs > where the derived obs is either missing or None. In essence, in your case > (where the derived obs were not calculated) the wee_database approach is > identical to re-importing the data but without actually needing to delete > and re-import the data. You can use --date or --from and --to to limit > the time span of records that wee_database --calc-missing will operate > on. Note that if you do not use --date or --from and --to wee_database > --calc-missing will recalculate missing derived obs for the entire > database. > > I would recommend backing up your database before performing either > operation. If you use the wee_database approach I would also recommend > stopping WeeWX before using wee_database. > > Gary > On Tuesday, 16 August 2022 at 03:24:14 UTC+10 [email protected] wrote: > >> Hello gjr80, >> sorry for my late reply, it has been a busy week, and thanks for got >> tested my data. >> >> I was unable to dig out something useful from logs: we can consider them >> as gone. >> Anyway: checking my `/etc/weewx/weewx.conf` I remembered I have this >> configuration: >> >> [StdWXCalculate] >> [[Calculations]] >> pressure = hardware >> altimeter = hardware >> appTemp = prefer_hardware >> barometer = prefer_hardware >> cloudbase = prefer_hardware >> dewpoint = hardware >> ET = prefer_hardware >> heatindex = hardware >> humidex = prefer_hardware >> inDewpoint = prefer_hardware >> maxSolarRad = prefer_hardware >> rainRate = prefer_hardware >> windchill = hardware >> windrun = prefer_hardware >> >> so I'd bet the reason is that one. >> The reason I have such configuration is because I have a Vantage VUE, and >> I've chosen `loop_request=3` (i.e. alternate between LOOP1 and LOOP2) and >> in that case the configuration above is suggested in the documentation: >> https://www.weewx.com/docs/usersguide.htm#[Vantage] >> >> I didn't think about it when I imported my data. >> So, is there a way, now, to fix and recalculate those missing values, or >> I have to >> 1. stop weewx >> 2. remove previously imported records from database >> 3. change the [StdWXCalculate] -> [[Calculations]] setting everything >> to `prefer_hardware` >> 4. re-run the whole import process >> 5. restore everything as before >> ? >> >> Thanks! >> Il giorno martedì 9 agosto 2022 alle 13:58:44 UTC+2 gjr80 ha scritto: >> >>> Sorry for the delay, I wanted to run your data and import config through >>> wee_import before replying. >>> >>> I ran your data and import config through wee_import and the data was >>> imported successfully and dewpoint, windchill and heatindex calculated >>> correctly. altimeter and pressure were not calculated; the reason is a >>> little complicated. Provided one of the pressures (altimeter, barometer >>> or pressure) is present WeeWX can calculate the other two; however, >>> when only altimeter or barometer is present pressure must be calculated >>> first and pressure requires temperature from 12 hours earlier. In my >>> case I did not have temperature from 12 hours earlier so pressure was >>> not calculated. When calculating altimeter (or barometer) pressure must >>> exist (hence why pressure is calculated first) so in my case altimeter >>> could not be calculated either. In a typical import where temperature 12 >>> hours ago is missing from the first 12 hours of import data you often see >>> 12 hours of two missing pressures before all three are calculated once the >>> temperature 12 hours ago kicks in. I digress. >>> >>> I guess the next thing is to find your log entries. You are looking for >>> whatever log file WeeWX logs to. That may be /var/log/syslog or perhaps >>> /var/log/messages unless you have explicitly changed it. If using a RPi >>> it will likely be /var/log/syslog. The wiki page where are my logs? >>> <https://github.com/weewx/weewx/wiki/faq-where-are-my-logs> may help. >>> If WeeWX is running the log should be very easily spotted as the most >>> recent log file (whichever name it is) will contain clearly labelled WeeWX >>> log entries. The wee_import entries will be clearly labelled as such, >>> you should be looking for something like this: >>> >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO __main__: Starting >>> wee_import... >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.csvimport: A >>> CSV import from source file '/var/tmp/test_data.csv' has been requested. >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.csvimport: >>> Using database binding 'wx_binding', which is bound to database >>> 'weewx.sdb' >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.csvimport: >>> Destination table 'archive' unit system is '0x01' (US). >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.csvimport: >>> Missing derived observations will be calculated. >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.csvimport: All >>> WeeWX UV fields will be set to None. >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.csvimport: All >>> WeeWX radiation fields will be set to None. >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: >>> Obtaining raw import data for period 1 ... >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: The >>> following imported field-to-WeeWX field map will be used: >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: >>> source field 'dateTime' in units 'unix_epoch' --> WeeWX field 'dateTime' >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: >>> source field 'barometer' in units 'hPa' --> WeeWX field 'barometer' >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: >>> source field 'insideTemp' in units 'degree_C' --> WeeWX field 'inTemp' >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: >>> source field 'outsideTemp' in units 'degree_C' --> WeeWX field 'outTemp' >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: >>> source field 'insideHum' in units 'percent' --> WeeWX field 'inHumidity' >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: >>> source field 'outsideHum' in units 'percent' --> WeeWX field >>> 'outHumidity' >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: >>> source field 'windSpeed' in units 'km_per_hour' --> WeeWX field >>> 'windSpeed' >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: >>> source field 'windDirection' in units 'degree_compass' --> WeeWX field >>> 'windDir' >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: >>> source field 'hiWindSpeed' in units 'km_per_hour' --> WeeWX field >>> 'windGust' >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: >>> source field 'hiWindDirection' in units 'degree_compass' --> WeeWX >>> field 'windGustDir' >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: >>> source field 'hiRainRate' in units 'mm_per_hour' --> WeeWX field >>> 'rainRate' >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: >>> source field 'rain' in units 'mm' --> WeeWX field 'rain' >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: Raw >>> import data read successfully for period 1. >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: >>> Mapping raw import data for period 1 ... >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: >>> Mapped 5 records. >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: Raw >>> import data mapped successfully for period 1. >>> Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: >>> Saving mapped data to archive for period 1 ... >>> Aug 9 21:50:05 bullseye4 wee_import[1156] ERROR weewx.manager: Unable >>> to add record 2021-08-01 00:01:00 AEST (1627740060) to database >>> 'weewx.sdb': UNIQUE constraint failed: archive.dateTime >>> Aug 9 21:50:05 bullseye4 wee_import[1156] ERROR weewx.manager: Unable >>> to add record 2021-08-01 00:02:00 AEST (1627740120) to database >>> 'weewx.sdb': UNIQUE constraint failed: archive.dateTime >>> Aug 9 21:50:05 bullseye4 wee_import[1156] ERROR weewx.manager: Unable >>> to add record 2021-08-01 00:03:00 AEST (1627740180) to database >>> 'weewx.sdb': UNIQUE constraint failed: archive.dateTime >>> Aug 9 21:50:05 bullseye4 wee_import[1156] ERROR weewx.manager: Unable >>> to add record 2021-08-01 00:04:00 AEST (1627740240) to database >>> 'weewx.sdb': UNIQUE constraint failed: archive.dateTime >>> Aug 9 21:50:05 bullseye4 wee_import[1156] ERROR weewx.manager: Unable >>> to add record 2021-08-01 00:05:00 AEST (1627740300) to database >>> 'weewx.sdb': UNIQUE constraint failed: archive.dateTime >>> Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weeimport.weeimport: >>> Mapped data saved to archive successfully for period 1. >>> Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weeimport.weeimport: >>> Calculating missing derived observations ... >>> Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.engine: StdConvert >>> target unit is 0x1 >>> Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.wxservices: >>> StdWXCalculate will use data binding wx_binding >>> Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.engine: Archive >>> will use data binding wx_binding >>> Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.engine: Record >>> generation will be attempted in 'hardware' >>> Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.engine: Using >>> archive interval of 60 seconds (specified by hardware) >>> Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.restx: >>> StationRegistry: Registration not requested. >>> Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.restx: >>> Wunderground: Posting not enabled. >>> Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.restx: PWSweather: >>> Posting not enabled. >>> Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.restx: CWOP: >>> Posting not enabled. >>> Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.restx: WOW: >>> Posting not enabled. >>> Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.restx: AWEKAS: >>> Posting not enabled. >>> Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.engine: 'pyephem' >>> detected, extended almanac data is available >>> Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.wxservices: >>> StdWXCalculate will use data binding wx_binding >>> Aug 9 21:50:06 bullseye4 wee_import[1156] INFO weewx.manager: Starting >>> backfill of daily summaries >>> Aug 9 21:50:06 bullseye4 wee_import[1156] INFO weewx.manager: Processed >>> 5 records to backfill 1 day summaries in 0.01 seconds >>> Aug 9 21:50:06 bullseye4 wee_import[1156] INFO weecfg.database: >>> Processed 1 day consisting of 5 records. 1 day consisting of 5 records >>> were updated in 0.04 seconds. >>> Aug 9 21:50:06 bullseye4 wee_import[1156] INFO weeimport.weeimport: >>> Finished calculating missing derived observations >>> Aug 9 21:50:06 bullseye4 wee_import[1156] INFO weeimport.weeimport: >>> Finished import >>> Aug 9 21:50:06 bullseye4 wee_import[1156] INFO weeimport.weeimport: 5 >>> records were processed and 5 unique records imported in 2.45 seconds. >>> >>> (Note in this case the error lines are due to there already being >>> records with the imported data timestamps in the archive) Please post all >>> of the import logs, don't leave anything out. >>> >>> Gary >>> >>> On Monday, 8 August 2022 at 02:50:21 UTC+10 [email protected] wrote: >>> >>>> Thanks for your answer. >>>> The two conditions you listed are satisfied: the necessary other values >>>> are present in my csv, and no records were already existing for such >>>> period >>>> in the database. >>>> >>>> Answering your questions: >>>> >>>> - Those are my data (a sample): >>>> CSV data (more fields than used in the import process are present) >>>> >>>> dateTime,outsideTemp,hiOutsideTemp,lowOutsideTemp,outsideHum,windSpeed,windDirection,hiWindSpeed,hiWindDirection,barometer,rain,hiRainRate,insideTemp,insideHum,numWindSamples,archiveInterval >>>> >>>> >>>> 2021-08-01T00:01,27.61,27.61,27.61,60.0,1.61,SW,3.22,SW,1006.30,0.00,0.00,30.33,49.0,22,1 >>>> >>>> >>>> 2021-08-01T00:02,27.56,27.61,27.56,60.0,1.61,W,1.61,SW,1006.16,0.00,0.00,30.33,49.0,24,1 >>>> >>>> >>>> 2021-08-01T00:03,27.50,27.56,27.50,60.0,1.61,W,1.61,W,1006.27,0.00,0.00,30.33,49.0,23,1 >>>> >>>> >>>> 2021-08-01T00:04,27.50,27.50,27.50,60.0,0.00,SW,3.22,SW,1006.33,0.00,0.00,30.33,49.0,22,1 >>>> >>>> >>>> 2021-08-01T00:05,27.50,27.50,27.50,60.0,3.22,WSW,4.83,WSW,1006.37,0.00,0.00,30.33,49.0,24,1 >>>> >>>> Resulting db records (extracted querying the `archive`. I here >>>> removed all null values for reading convenience) >>>> >>>> dateTime,usUnits,interval,appTemp,barometer,cloudbase,humidex,inDewpoint,inHumidity,inTemp,outHumidity,outTemp,rain,rainRate,windDir,windGust,windGustDir,windrun,windSpeed >>>> >>>> >>>> 1627768860,16,1,30.5895137231874,1006.3,1167.77458102164,34.4779454220626,18.4017421866142,49,30.33,60,27.61,0,0,225,3.22,225,0.0268333333333333,1.61 >>>> >>>> >>>> 1627768920,16,1,30.5182757269146,1006.16,1167.3911940789,34.3910793180734,18.4017421866142,49,30.33,60,27.56,0,0,270,1.61,225,0.0268333333333333,1.61 >>>> >>>> >>>> 1627768980,16,1,30.4328612401526,1006.27,1166.93122058363,34.286968264113,18.4017421866142,49,30.33,60,27.5,0,0,270,1.61,270,0.0268333333333333,1.61 >>>> >>>> >>>> 1627769040,16,1,30.7459167959586,1006.33,1166.93122058363,34.286968264113,18.4017421866142,49,30.33,60,27.5,0,0,,3.22,225,0,0 >>>> >>>> >>>> 1627769100,16,1,30.1198056843466,1006.37,1166.93122058363,34.286968264113,18.4017421866142,49,30.33,60,27.5,0,0,247.5,4.83,247.5,0.0536666666666667,3.22 >>>> >>>> - No data were already present for that period >>>> - I checked resulting records by querying the database directly >>>> - The query I ran (for the sample): SELECT * FROM archive WHERE >>>> dateTime >= 1627768860 and dateTime < 1627769160 ORDER BY dateTime >>>> - I imported data few weeks ago, so I cannot exactly say what the >>>> output of the import command was, but - as I can remember - nothing >>>> remarkable was printed out. I noticed just now that some fields are >>>> missing. Can I still check those logs? how to? >>>> >>>> Thanks, >>>> Andrea >>>> Il giorno domenica 7 agosto 2022 alle 02:09:20 UTC+2 gjr80 ha scritto: >>>> >>>>> Your import config file is fine and provided a couple of conditions >>>>> were met the missing fields should have been calculated. The two >>>>> conditions >>>>> are (1) any pre-requisites for calculating the fields concerned must >>>>> exist, for example for heatindex, windchill and dewpoint you must >>>>> have a non-None outTemp value. If a pre-requisite is missing the >>>>> calculated value will be None. (2) Even if the calculated values were >>>>> calculated there must be no record already existing in the archive with >>>>> the >>>>> same timestamp as the record being imported. If such a record already >>>>> exists in the archive the imported record is discarded in it's entirety >>>>> (not just the calculated fields). So if these conditions were met your >>>>> imported data including calculated fields should have been imported and >>>>> saved to archive, if they were not imported then something went wrong. >>>>> >>>>> A few questions to try to narrow down the cause. What does your data >>>>> look like? Can you post some of the records here or if you don't want to >>>>> can you reply privately to this post with your raw import data. Did >>>>> records >>>>> already exist in your archive for the period covered by the data you were >>>>> importing? How did you check the imported data? Did you query the >>>>> database >>>>> archive table directly or rely on WeeWX generated output? If the former >>>>> what command did you use and what results were presented? Did the other >>>>> non-calculated fields import correctly? What was presented on-screen when >>>>> you ran wee_import? Were there any errors or warnings? wee-import >>>>> will have written some entries to the log at the time of the import, they >>>>> should be clearly evident with a 'wee_import' label, what do these >>>>> entries >>>>> say? >>>>> >>>>> Gary >>>>> >>>>> On Sunday, 7 August 2022 at 00:23:31 UTC+10 [email protected] >>>>> wrote: >>>>> >>>>>> I imported ~1 year of data from csv, using this `import.conf` >>>>>> configuration: >>>>>> source = CSV >>>>>> [CSV] >>>>>> file = my-data.csv >>>>>> interval = 1 >>>>>> qc = True >>>>>> calc_missing = True >>>>>> ignore_invalid_data = True >>>>>> tranche = 250 >>>>>> UV_sensor = False >>>>>> solar_sensor = False >>>>>> raw_datetime_format = %Y-%m-%dT%H:%M >>>>>> rain = discrete >>>>>> wind_direction = 0,360 >>>>>> [[FieldMap]] >>>>>> dateTime = dateTime, unix_epoch >>>>>> usUnits = >>>>>> interval = >>>>>> barometer = barometer, hPa >>>>>> pressure = >>>>>> altimeter = >>>>>> inTemp = insideTemp, degree_C >>>>>> outTemp = outsideTemp, degree_C >>>>>> inHumidity = insideHum, percent >>>>>> outHumidity = outsideHum, percent >>>>>> windSpeed = windSpeed, km_per_hour >>>>>> windDir = windDirection, degree_compass >>>>>> windGust = hiWindSpeed, km_per_hour >>>>>> windGustDir = hiWindDirection, degree_compass >>>>>> rainRate = hiRainRate, mm_per_hour >>>>>> rain = rain, mm >>>>>> dewpoint = >>>>>> windchill = >>>>>> heatindex = >>>>>> ET = >>>>>> radiation = >>>>>> UV = >>>>>> >>>>>> What I expected was that the following fields would have been >>>>>> calculated, starting from the other available fields: >>>>>> altimeter >>>>>> dewpoint >>>>>> heatindex >>>>>> pressure >>>>>> windchill >>>>>> But in the `archive` table, those columns are empty. >>>>>> >>>>>> Did I do something wrong, or is the behavior correct? So I cannot >>>>>> have such fields calculated...? >>>>>> >>>>> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/05885673-668c-4d8b-9ee3-b4b85413c282n%40googlegroups.com.
