Fortunately, I think use of aware datetime objects in this case can be isolated to weathercatimport.py because the current implementation converts each entry's datatime to a unix epoch timestamp prior inserting it into the row for later processing. For my purposes, I think Python 3.9 and later provides all the necessary datetime and zoneinfo functions to handle the various cases.
As you pointed out, with the current implementation in weathercatimport.py, you can silently(ish) loose the entries during the autumn transition hour from DST to standard time because the later entries with duplicate timestamp in standard time (fold=1) will interact with the entries using the same timestamp in DST (fold=0). However, for WeatherCat data with Adaptive interval set, some of the later standard time entries' timestamps may not co-inside with the earlier DST entries' timestamps, so you could end up with a mix of standard time and DST time entries across that hour. Furthermore, with the current implementation, approximately half of a year's worth of entries will be entered into the database using the wrong time (and even date in some edge cases) because processing for the entire year is done using the local time DST setting when the actual import takes place. There will also be an hour long hole (or maybe two) in the data for each year when transitioning from standard time to DST in the spring. Of course, you may not consider these issues to be particularly problematic for historical data... The import problems seems to be easily divisible into two parts which need separate solutions: 1) transition fro standard time to DST in spring; and 2) transition from DST to standard time in the fall. I have been working on changes to weathercatimport.py to address the transition from DST to standard time (2). It is the easier of the two transitions. I thought I had competed the necessary changes, but when tested it with all of my data (7 years worth), I came across yet another anomaly in my WeatherCat data. I have at least a pair of rows in Oct-2017 sometime after the transition to standard time that have identical timestamps. I presume that WeatherCat has some kind of rounding error, or does not account for common unix timer variations/clock jitter/resolution or something. I'll work on fixing this issue, and then continue with 1). Once complete, I'll mail you the changes (along with sample data) and you can decide what to do with them: either incorporate them into the codebase; modify them to meet Python backward compatibility rules; or dump them in the nearest bin. There clearly aren't too many folks with this kind of historic WeatherCat database to import. Nick On Sunday, August 28, 2022 at 12:22:21 AM UTC+1 gjr80 wrote: > On Thursday, 25 August 2022 at 23:54:11 UTC+10 kurand wrote: > >> Thanks for confirming what I understood from weeimport.py and >> weathercatimport.py. >> >> I am using "derived' for the import because I originally configured >> WeatherCat with Adaptive interval set true so I have data logged at >> varying rates from 1 minute all the way up to 30+ minute intervals. >> >> I think in this case, weathercatimport.py needs to use timezone aware >> date-time objects during its construction of dateTime else data will become >> misplaced and DST changes will fail as noted. Once this is accomplished, >> then something needs to get added to deal with the apparent bug in >> WeatherCat where transitioning from standard time to daylight saving time >> in spring causes it to skip an extra hour forward sometimes. It looks to me >> like this can all be isolated in weathercatimport.py's getRawData function. >> >> I'm curious to know if anyone else has run into this problem, or if it is >> something specific to my WeatherCat data. Or perhaps WeatherCat import has >> been used in anger. >> > > Yes 'aware' datetime objects would be preferred; however, their use > raises a number of other issues. Programatically, there is the potential > for conflict with the rest of the WeeWX code base which uses naive datetime > objects only. Also, we try to avoid external dependencies unless absolutely > necessary so hence we need to work within the standard libraries shipped > with python. Then there is the issue of daylight saving cutover times often > being politically driven, so who knows what impact that may have - we may > be solving one issue only to create others. > > That being said there are two issues at play here. Firstly, an out of > date-time order record caused the entire import to abort. The reasoning for > this behaviour was the assumption that a randomly ordered batch of records > was provided for import, this is probably too onerous and we could (should > be able to) back this off to simply ignore the miscreant record. This > should be a simple change. Secondly, the issue of daylight saving. I am > happy to have a look at wee_import to see what can be done to support > daylight saving using the existing included python libraries. To this end > some relevant WeatherCat data files would help, would you be able to > provide example WeatherCat files covering both the start and end of > daylight saving? If you want you can reply privately to this post and > include them. thanks. > > Gary > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/8e0a63ad-b022-4013-aa54-5739f497ed05n%40googlegroups.com.
