Thanks for confirming what I understood from weeimport.py and weathercatimport.py.
I am using "derived' for the import because I originally configured WeatherCat with Adaptive interval set true so I have data logged at varying rates from 1 minute all the way up to 30+ minute intervals. I think in this case, weathercatimport.py needs to use timezone aware date-time objects during its construction of dateTime else data will become misplaced and DST changes will fail as noted. Once this is accomplished, then something needs to get added to deal with the apparent bug in WeatherCat where transitioning from standard time to daylight saving time in spring causes it to skip an extra hour forward sometimes. It looks to me like this can all be isolated in weathercatimport.py's getRawData function. I'm curious to know if anyone else has run into this problem, or if it is something specific to my WeatherCat data. Or perhaps WeatherCat import has been used in anger. On Thursday, August 25, 2022 at 11:59:34 AM UTC+1 gjr80 wrote: > If you are using 'derive' to derive the archive interval from the imported > data then the imported records must be in ascending timestamp/date-time > order. This is because the derived archive interval is calculated by taking > the difference between the timestamp of the current record and the > timestamp of the previous record. If the calculated interval is negative it > indicates the records being imported are not in timestamp order and the > calculated archive interval cannot be trusted. The import will abort. > > This should not present a problem at the start of a daylight saving period > where clocks jump forward (though the archive interval would be incorrectly > calculated for the first record after daylight saving begins). It would > certainly cause a problem at the end of a period of daylight saving where > clocks jump back in time. > > So what to do? WeeWX date-time objects are timezone naive > <https://docs.python.org/3/library/datetime.html#aware-and-naive-objects> > so managing daylight saving is near impossible. I would suggest removing > the data to be imported with the duplicate date-times (which set of > duplicates I leave up to you) and re-trying the import. You might also want > to consider using a fixed interval (if your data does indeed have a fixed > interval) rather than deriving it from the data. FWIW, the vantage driver > has a similar issue in that each year when daylight saving ends the driver > receives archive records with duplicate timestamps, in this case the > situation is handled slightly more gracefully in that WeeWX ignores the > duplicate timestamped records with a 'PRIMARY KEY must be unique' error. > wee_import is somewhat less forgiving. > > Gary > On Thursday, 25 August 2022 at 00:45:13 UTC+10 kurand wrote: > >> Has anyone successfully imported data from WeatherCat for months where >> Daylight Savings Time starts or ends? >> >> When I have tried this, I get the following type of error (example for >> Oct-2017): >> >> Cannot derive 'interval' for record timestamp: 2017-10-29 01:00:00 BST >> (1509235200). >> >> **** Raw data is not in ascending date time order. >> >> Looking at the 7 years of data I have logged by WeatherCat, it appears >> that WeatherCat does not really handle DST transitions very well. Data is >> generally logging in local time. >> >> For spring DST changes (in March for my UK based timezone), it seems >> usually to jump forward by two hours at 0100 going to 0300 rather than just >> one hour as you'd expect. It then tack on an hours worth of data for day 01 >> at the end of the month. This gives rise to the above error when the end of >> the month's data is processed. However, for some reason, in my data for >> 2014, it does only jump forward one hour as expected and there was no extra >> data logged at the end of the month. >> >> For autumn DST changes (in October in UK timezone), it seems to jump back >> an hour at 0200, so it repeats log entries for hour 0100 again. This gives >> rise to the above error. >> >> It's all a bit of a mess really; I'm not sure if its just my WeatherCat >> data, or the way I configured it with Adaptive interval set true or >> something else. >> >> Does anyone have a fix for this, or do I need to work something out. >> > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/369e8880-dcbb-4028-9d4f-dbf775a8fa5bn%40googlegroups.com.
