Rod, I've spent quite a bit of time working on a "cleaner" workaround for the WU bugs. But just when I thought I had landed on the "reasonable" approach, I hit a snag.
As a reminder, with the latest approach, all is fine for stations west of the UTC date line. What I am working on now is a workaround for stations east of the UTC date line, where the data returned is a day after the date requested. The wall that I just hit is that the WU behavior around this latest bug changes, when looking historically. :-( >From today's date going back to 2019-05-28, the query returns obsTimeLocal >data from the day after the date requested. This is the bug we are trying to >workaround. To compensate, I can ask for the prior date to get the date we really want. HOWEVER, starting with 2019-05-26 (and prior), the query returns obsTimeLocal data for the date that was actually requested. This leads to the inability to get data for 2019-05-27, because the actual query for that date returns 2019-05-28 datum, and the compensation date of 2019-05-26 returns actual data for 2019-05-26. I think I am done trying to work around this bug. IBM needs to own it. See here -- scroll to second to last example where 2019-05-27 is requested, but when I tried 2019-05-26, I got back data for the actual 2019-05-26, so compensation did not work. :-( pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-06-02 --station ISYDNEY155 --test --verbose | (head -n 9 ; tail -n 1) Using configuration file /usr/share/weewx4/weewx.conf. Weather Underground Station: ISYDNEY155 Date to check: 2019-06-02 WU API obsTimeLocal: 2019-06-03 WU API WRONG DATE !!!!!!!!!!!!!!!!!!!!!!! WU API COMPENSATION DATE: 2019-06-01 WU API obsTimeLocal: 2019-06-02 epoch: 1559397600 date_epoch_local: 2019-06-01 10:00:00 tz: Australia/Sydney obsTimeUtc: 2019-06-01T14:00:00Z obsTimeLocal: 2019-06-02 00:00:00 epoch: 1559397900 date_epoch_local: 2019-06-01 10:05:00 tz: Australia/Sydney obsTimeUtc: 2019-06-01T14:05:00Z obsTimeLocal: 2019-06-02 00:05:00 Number of WU records: 288 pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-06-01 --station ISYDNEY155 --test --verbose | (head -n 9 ; tail -n 1) Using configuration file /usr/share/weewx4/weewx.conf. Weather Underground Station: ISYDNEY155 Date to check: 2019-06-01 WU API obsTimeLocal: 2019-06-02 WU API WRONG DATE !!!!!!!!!!!!!!!!!!!!!!! WU API COMPENSATION DATE: 2019-05-31 WU API obsTimeLocal: 2019-06-01 epoch: 1559311200 date_epoch_local: 2019-05-31 10:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-31T14:00:00Z obsTimeLocal: 2019-06-01 00:00:00 epoch: 1559311500 date_epoch_local: 2019-05-31 10:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-31T14:05:00Z obsTimeLocal: 2019-06-01 00:05:00 Number of WU records: 288 pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-05-31 --station ISYDNEY155 --test --verbose | (head -n 9 ; tail -n 1) Using configuration file /usr/share/weewx4/weewx.conf. Weather Underground Station: ISYDNEY155 Date to check: 2019-05-31 WU API obsTimeLocal: 2019-06-01 WU API WRONG DATE !!!!!!!!!!!!!!!!!!!!!!! WU API COMPENSATION DATE: 2019-05-30 WU API obsTimeLocal: 2019-05-31 epoch: 1559224800 date_epoch_local: 2019-05-30 10:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T14:00:00Z obsTimeLocal: 2019-05-31 00:00:00 epoch: 1559225100 date_epoch_local: 2019-05-30 10:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T14:05:00Z obsTimeLocal: 2019-05-31 00:05:00 Number of WU records: 288 pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-05-30 --station ISYDNEY155 --test --verbose | (head -n 9 ; tail -n 1) Using configuration file /usr/share/weewx4/weewx.conf. Weather Underground Station: ISYDNEY155 Date to check: 2019-05-30 WU API obsTimeLocal: 2019-05-31 WU API WRONG DATE !!!!!!!!!!!!!!!!!!!!!!! WU API COMPENSATION DATE: 2019-05-29 WU API obsTimeLocal: 2019-05-30 epoch: 1559138400 date_epoch_local: 2019-05-29 10:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-29T14:00:00Z obsTimeLocal: 2019-05-30 00:00:00 epoch: 1559138700 date_epoch_local: 2019-05-29 10:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-29T14:05:00Z obsTimeLocal: 2019-05-30 00:05:00 Number of WU records: 288 pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-05-29 --station ISYDNEY155 --test --verbose | (head -n 9 ; tail -n 1) Using configuration file /usr/share/weewx4/weewx.conf. Weather Underground Station: ISYDNEY155 Date to check: 2019-05-29 WU API obsTimeLocal: 2019-05-30 WU API WRONG DATE !!!!!!!!!!!!!!!!!!!!!!! WU API COMPENSATION DATE: 2019-05-28 WU API obsTimeLocal: 2019-05-29 epoch: 1559052000 date_epoch_local: 2019-05-28 10:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-28T14:00:00Z obsTimeLocal: 2019-05-29 00:00:00 epoch: 1559052300 date_epoch_local: 2019-05-28 10:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-28T14:05:00Z obsTimeLocal: 2019-05-29 00:05:00 Number of WU records: 288 pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-05-28 --station ISYDNEY155 --test --verbose | (head -n 9 ; tail -n 1) Using configuration file /usr/share/weewx4/weewx.conf. Weather Underground Station: ISYDNEY155 Date to check: 2019-05-28 WU API obsTimeLocal: 2019-05-29 WU API WRONG DATE !!!!!!!!!!!!!!!!!!!!!!! WU API COMPENSATION DATE: 2019-05-27 WU API obsTimeLocal: 2019-05-28 epoch: 1558965600 date_epoch_local: 2019-05-27 10:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-27T14:00:00Z obsTimeLocal: 2019-05-28 00:00:00 epoch: 1558965900 date_epoch_local: 2019-05-27 10:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-27T14:05:00Z obsTimeLocal: 2019-05-28 00:05:00 Number of WU records: 288 /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-05-27 --station ISYDNEY155 --test --verbose | (head -n 9 ; tail -n 1) Using configuration file /usr/share/weewx4/weewx.conf. Weather Underground Station: ISYDNEY155 Date to check: 2019-05-27 WU API obsTimeLocal: 2019-05-28 WU API WRONG DATE !!!!!!!!!!!!!!!!!!!!!!! WU API COMPENSATION DATE: 2019-05-26 WU API obsTimeLocal: 2019-05-26 WU API COMPSENSATION FAILURE! ABORTING!!! Number of WU records: 0 No results returned from Weather Underground (perhaps a bad station name??). pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-05-26 --station ISYDNEY155 --test --verbose | (head -n 9 ; tail -n 1) Using configuration file /usr/share/weewx4/weewx.conf. Weather Underground Station: ISYDNEY155 Date to check: 2019-05-26 WU API obsTimeLocal: 2019-05-26 epoch: 1558792800 date_epoch_local: 2019-05-25 10:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-25T14:00:00Z obsTimeLocal: 2019-05-26 00:00:00 epoch: 1558793100 date_epoch_local: 2019-05-25 10:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-25T14:05:00Z obsTimeLocal: 2019-05-26 00:05:00 epoch: 1558793400 date_epoch_local: 2019-05-25 10:10:00 tz: Australia/Sydney obsTimeUtc: 2019-05-25T14:10:00Z obsTimeLocal: 2019-05-26 00:10:00 epoch: 1558793700 date_epoch_local: 2019-05-25 10:15:00 tz: Australia/Sydney obsTimeUtc: 2019-05-25T14:15:00Z obsTimeLocal: 2019-05-26 00:15:00 epoch: 1558794000 date_epoch_local: 2019-05-25 10:20:00 tz: Australia/Sydney obsTimeUtc: 2019-05-25T14:20:00Z obsTimeLocal: 2019-05-26 00:20:00 Number of WU records: 288 Regards, \Leon -- Leon Shaner :: Dearborn, Michigan (iPad Pro) > On Jun 2, 2019, at 10:20 AM, Leon Shaner <[email protected]> wrote: > > Rod, > > It's interesting that the "workaround" that I've already done, is actually > working 100% for me. I got good results when combining the 'today' API when > asking for the current date, and only using the 'historical' result when > asking for dates other than the current date -- relative to station local > time. > > Here is my KMIDEARB5 station's transition from 7:59 p.m. to 8:00 p.m, where > time moves to within the UTC-4 offset: > > $ diff 59:19:1:6_wu.txt 0:20:1:6_wu.txt > 241c241,242 > < Number of WU records: 239 > --- > > epoch: 1559433599 date_epoch_local: 2019-06-01 19:59:59 utcepoch: > > 1559433999 date_epoch_utc: 2019-06-01 23:59:59 tz: America/New_York > > obsTimeUtc: 2019-06-01T23:59:59Z obsTimeLocal: 2019-06-01 19:59:59 > > Number of WU records: 240 > > The above is exactly correct -- the second query is one minute later and is > after/on the 5-minute record "boundary" hence one additional record. > > And then as my station transition from 11:59 p.m. to midnight next day. > Before midnight the 'today' API is used, but after midnight the 'historical' > API will be used, which was always working for me after midnight (station > local time): > > $ tail -n 4 59:23:1:6_wu.txt 0:0:2:6_wu.txt > ==> 59:23:1:6_wu.txt <== > epoch: 1559447399 date_epoch_local: 2019-06-01 23:49:59 utcepoch: 1559447799 > date_epoch_utc: 2019-06-02 03:49:59 tz: America/New_York obsTimeUtc: > 2019-06-02T03:49:59Z obsTimeLocal: 2019-06-01 23:49:59 > epoch: 1559447699 date_epoch_local: 2019-06-01 23:54:59 utcepoch: 1559448099 > date_epoch_utc: 2019-06-02 03:54:59 tz: America/New_York obsTimeUtc: > 2019-06-02T03:54:59Z obsTimeLocal: 2019-06-01 23:54:59 > epoch: 1559447701 date_epoch_local: 2019-06-01 23:55:01 utcepoch: 1559448101 > date_epoch_utc: 2019-06-02 03:55:01 tz: America/New_York obsTimeUtc: > 2019-06-02T03:55:01Z obsTimeLocal: 2019-06-01 23:55:01 > Number of WU records: 288 > > ==> 0:0:2:6_wu.txt <== > epoch: 1559447399 date_epoch_local: 2019-06-01 23:49:59 utcepoch: 1559447799 > date_epoch_utc: 2019-06-02 03:49:59 tz: America/New_York obsTimeUtc: > 2019-06-02T03:49:59Z obsTimeLocal: 2019-06-01 23:49:59 > epoch: 1559447699 date_epoch_local: 2019-06-01 23:54:59 utcepoch: 1559448099 > date_epoch_utc: 2019-06-02 03:54:59 tz: America/New_York obsTimeUtc: > 2019-06-02T03:54:59Z obsTimeLocal: 2019-06-01 23:54:59 > epoch: 1559447999 date_epoch_local: 2019-06-01 23:59:59 utcepoch: 1559448399 > date_epoch_utc: 2019-06-02 03:59:59 tz: America/New_York obsTimeUtc: > 2019-06-02T03:59:59Z obsTimeLocal: 2019-06-01 23:59:59 > Number of WU records: 288 > > There again, the second query is a minute later, so one more result, but with > explanation... ;-) > > There is a slight difference between the 'today' and 'historical' result. > The first result, using the 'today' API ran at 1 minute before midnight, > which was on the cusp of posting the 1-minute record nearest actual midnight. > The 'today' result shows two time-stamps very close to 11:55 p.m. > Where-as the 'historical' result replaces the later ~11:55 p.m. result with > the ~midnight result. > > Those differences are "in the noise" and inconsequential. Even the default > wunderfixer --epsilon=120 will filter out such a duplicate. > > I am happy that this new approach with the combined 'today' and 'historical' > queries is at least effective for stations west of the UTC dateline. > It means I have easily worked around half of the WU API bug. > > Now to further press IBM to fix the other half of the bug... =D > (Even while I continue to think about ways to *cleanly* workaround the bug > for stations east of UTC dateline). > > Regards, > \Leon > -- > Leon Shaner :: Dearborn, Michigan (iPad Pro) > >> On Jun 1, 2019, at 11:08 PM, Rod Yager <[email protected]> wrote: >> >> Dear Leon, >> >> If WU doesn’t fix their bug, the solution is really quite simple. >> >> We need to make 3 requests to WU for the data: one request for the date we >> are interested in, one request for the day before and one request for the >> day after. >> >> eg. If we are intending to upload missing data for 2019-01-01 we need to ask >> WU for the records it has for 2018-12-31, 2019-01-01 and 2019-01-02. >> >> One of these will have the information pertaining to the day we are >> interested in. >> >> We then need to filter the three responses from WU, only keeping the >> responses that pertain to the day we are interested in. >> >> At this point, we have exactly what we want from WU and so we compare that >> with the local database, and upload the data that is missing in the WU >> record, just as we always did. >> >> >> in your timezone (UTC-4) if you ask for data for your station the correct >> information will be in >> the same day request: if the request is made between 00:00 and 19:59 (at >> those times, the UTC date is the same as the date where you are ) >> the request for the next day: if the request is made between 20:00 and 23:59 >> (at those times, the UTC date is one day ahead the date where you are) >> >> If you request information for my station (I’ve sent you the details in a >> PM), the correct information will be in >> the same day request: if the request is made between 10:00 and 19:59 your >> time (at those times, the UTC date is the same as the date where my >> station is located) >> the request for the previous day: if the request is made between 0:00 and >> 09:59 or between 20:00 and 23:59 your time (at those times, the UTC date is >> one day behind the date where my station is located) >> >> Rod >> >> >> >>> On 2 Jun 2019, at 9:27 am, Leon Shaner <[email protected]> wrote: >>> >>> Rod, >>> >>> Thanks for testing. Well, WU's approach is flawed, not mine, really. :S >>> >>> It means that both API's, the one I just added for 'today" and the one from >>> before that was for 'historical' (including today), both have the same bug. >>> >>> For now, the best workaround I can think of is to only ever run the new >>> wunderfixer only ever for the previous day, only ever after UTC has rolled >>> over midnight. :-( >>> E.g. when it is 2019-06-02 in UTC, you can get the right results with >>> wunderfixer if you ask for 2019-06-01 only. In other words, always delay >>> your wunderfixer fix ups by a day, operating always on yesterday's date >>> (relative to UTC) after UTC has rolled over to the next day. >>> >>> Scroll to see the output of me running wunderdates on my local machine in >>> UTC-4 against your station in UTC+10. >>> >>> Because I am asking for 2019-06-01, which is also my local date, the logic >>> will use the URL that queries for 'today' and instead of returning >>> obsTimeLocal records, it is returning obsTimeUtc records for the requested >>> date. We can see that it's returning 2019-06-02 relative to your station, >>> even though I asked for 2019-06-01. >>> >>> Sure, I could go a bit further and use a related API query to check the >>> timezone of your station and use date conversions to check if the date >>> requested is 'today' relative to the station requested, but really this is >>> intended to be run on the same machine as weewx, so the localtime date >>> should match that of the station. It means there isn't any real reason for >>> me to workaround their bug. In fact even if I did, then the logic would >>> just fall through to the historical data URL, which we already know is >>> buggy. >>> >>> For now, I'll just report this to IBM a bug in the 'today' API, similar to >>> the bug in the 'historical' API. >>> >>> $ ./wunderdates --epsilon=125 --date 2019-06-01 --station ISYDNEY155 --test >>> --verbose | more >>> >>> Using configuration file /usr/share/weewx4/weewx.conf. >>> Weather Underground Station: ISYDNEY155 >>> Date to check: 2019-06-01 >>> epoch: 1559397600 date_epoch_local: 2019-06-01 10:00:00 utcepoch: >>> 1559398000 date_epoch_utc: 2019-06-01 14:00:00 tz: Australia/Sydney >>> obsTimeUtc: 2019-06-01T14:00:00Z obsTimeLocal: 2019-06-02 00:00:00 >>> epoch: 1559397900 date_epoch_local: 2019-06-01 10:05:00 utcepoch: >>> 1559398300 date_epoch_utc: 2019-06-01 14:05:00 tz: Australia/Sydney >>> obsTimeUtc: 2019-06-01T14:05:00Z obsTimeLocal: 2019-06-02 00:05:00 >>> ... >>> >>> >>> Regards, >>> \Leon >>> -- >>> Leon Shaner :: Dearborn, Michigan (iPad Pro) >>> >>> On Jun 1, 2019, at 6:03 PM, Rod Yager <[email protected]> wrote: >>> >>>> Dear Leon, >>>> >>>> Your approach is flawed. It won’t work for historical data. >>>> >>>> If you run >>>> >>>> wunderdates —date=2019-01-01 at 9pm your time you will receive the >>>> data for 2018-12-31 (because your station is a day behind UTC at the time >>>> of the request, so it gives you the data for the previous day). >>>> >>>> On the other hand, if you run >>>> >>>> wunderdates —date=2019-01-01 at 9am your time you will receive the >>>> data for 2019-01-01 (because your station is in the same day as UTC at >>>> the time of the request). >>>> >>>> If I do the same thing (I’m in UTC+10) >>>> >>>> wunderdates —date=2019-01-01 at 9pm my time produces the data for >>>> 2019-01-01 (because my station is in the same day as UTC at the time of >>>> the request) but >>>> wunderdates —date=2019-01-01 at 9am my time produces the data for >>>> 2019-01-02 (because my station is a day ahead of UTC at the time of the >>>> request) >>>> >>>> Rod >>>> >>>>> On 2 Jun 2019, at 3:17 am, Leon Shaner <[email protected]> wrote: >>>>> >>>>> All, >>>>> >>>>> I'm testing a new approach. Below you will find links to an updated >>>>> wunderdates utility that can be used to validate whether I am on the >>>>> right track. >>>>> The wunderdates utility simply dumps out timestamp related records from >>>>> what WU returns from the query. >>>>> >>>>> We're looking mainly at the obsTimeUtc and obsTimeLocal values, which >>>>> were demonstrated under the prior approach to be returning the wrong >>>>> dates when within the stations localtime +/- offset from UTC. >>>>> >>>>> The new approach is to detect if the requested date is 'today' and if so, >>>>> use a different API URL that already assumes 'today' and will hopefully >>>>> not be subject to the UTC offset bugs we've been chasing with the >>>>> historical data API URL. >>>>> >>>>> I have my crontab set to do another test approaching my UTC offset, just >>>>> after coming within the offset, and then again just before and after >>>>> midnight localtime. >>>>> (Same test I did before, but now with the new approach in place). >>>>> >>>>> Here is the utility for anyone else that wants to check out the behavior: >>>>> >>>>> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates3 >>>>> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates4 >>>>> >>>>> Which version you pull will depend on which base weewx you are running. >>>>> Pull the one that matches your weewx version and place it in bin, next to >>>>> wunderfixer, and it will take the same arguments as wunderfixer. >>>>> >>>>> You can just try wunderfixer as you normally would (with --apikey) and >>>>> then run wunderdates(3 or 4) with exact same arguments to be able to see >>>>> what actual timestamps WU is sending back for the date queried. >>>>> Parameters like --epsilon don't have any effect in the case of >>>>> wunderdates, but I left it there so you don't have to change options when >>>>> running the util to get the debug output. >>>>> >>>>> Regards, >>>>> \Leon >>>>> -- >>>>> Leon Shaner :: Dearborn, Michigan (iPad Pro) >>>>> >>>>> On May 30, 2019, at 12:40 AM, Leon Shaner <[email protected]> wrote: >>>>> >>>>>> Rod, >>>>>> Thanks again for this. >>>>>> >>>>>> Since the in-progress version of wunderfixer doesn't really show you the >>>>>> dates that come back from WU, I have written a tool just to debug the >>>>>> dates. >>>>>> >>>>>> The command-line input and basing of defaults on weewx.conf works the >>>>>> same as wunderfixer, except it doesn't look at your DB at all. It only >>>>>> prints out datestamps in various incarnations coming back from WU and as >>>>>> compared to your system's localtime. >>>>>> >>>>>> It would be helpful to see the "wunderdates" output at times like you've >>>>>> shown below, a la before and after your localtime rolls around to UTC >>>>>> midnight. >>>>>> >>>>>> Since you are at UTC + 10, another interesting time would be 11+ hours >>>>>> on either side of UTC midnight, in addition to within 9 or less hours. >>>>>> This is just to make sure we're covering all the corner cases. >>>>>> >>>>>> Gary reported a difference for stations that are east vs. west of GMT, >>>>>> and I expect we're really chasing the same bug in that there is some bad >>>>>> math WU is doing based on UTC offset, but since an offset can be +/-, >>>>>> the effects go in opposite directions date-wise, depending on which side >>>>>> of the UTC dateline your station is located. >>>>>> >>>>>> At least that is what I surmise may be happening, but the wunderdates >>>>>> utility should shed light one way or the other. =D >>>>>> >>>>>> The wunderdates utility is available at the links below. >>>>>> >>>>>> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates3 >>>>>> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates4 >>>>>> >>>>>> Which version you pull will depend on which base weewx you are running. >>>>>> Pull the one that matches your weewx version and place it in bin, next >>>>>> to wunderfixer, and it will take the same arguments as wunderfixer. >>>>>> >>>>>> You can just try wunderfixer as you normally would (with --apikey) and >>>>>> then run wunderdates(3 or 4) with exact same arguments to be able to see >>>>>> what actual timestamps WU is sending back for the date queried. >>>>>> Parameters like --epsilon don't have any effect in the case of >>>>>> wunderdates, but I left it there so you don't have to change options >>>>>> when running the util to get the debug output. >>>>>> >>>>>> What I've been doing is saving the output to files for use with sdiff >>>>>> (side-by-side) diff. >>>>>> Or you can just compare the head and tail of each file individually. >>>>>> >>>>>> Example for my system before and after my local time on 5-28 once it was >>>>>> at/after 8 p.m. here, which is within 4 hours of UTC midnight (I am at >>>>>> UTC-4). >>>>>> >>>>>> This output is optimized for screen widths 203 or wider. Sorry. :S >>>>>> Mainly the last two data fields in the output tell what we need to know. >>>>>> >>>>>> >>>>>> pi@nixie:/var/tmp $ head -n 3 59:19:28:5_wu.txt 0:20:28:5_wu.txt >>>>>> ==> 59:19:28:5_wu.txt <== >>>>>> Using configuration file /usr/share/weewx4/weewx.conf. >>>>>> epoch: 1559016299 date_epoch_local: 2019-05-28 00:04:59 utcepoch: >>>>>> 1559016699 date_epoch_utc: 2019-05-28 04:04:59 tz: America/New_York >>>>>> obsTimeUtc: 2019-05-28T04:04:59Z obsTimeLocal: 2019-05-28 00:04:59 >>>>>> epoch: 1559016599 date_epoch_local: 2019-05-28 00:09:59 utcepoch: >>>>>> 1559016999 date_epoch_utc: 2019-05-28 04:09:59 tz: America/New_York >>>>>> obsTimeUtc: 2019-05-28T04:09:59Z obsTimeLocal: 2019-05-28 00:09:59 >>>>>> >>>>>> ==> 0:20:28:5_wu.txt <== >>>>>> Using configuration file /usr/share/weewx4/weewx.conf. >>>>>> epoch: 1558929899 date_epoch_local: 2019-05-27 00:04:59 utcepoch: >>>>>> 1558930299 date_epoch_utc: 2019-05-27 04:04:59 tz: America/New_York >>>>>> obsTimeUtc: 2019-05-27T04:04:59Z obsTimeLocal: 2019-05-27 00:04:59 >>>>>> epoch: 1558930199 date_epoch_local: 2019-05-27 00:09:59 utcepoch: >>>>>> 1558930599 date_epoch_utc: 2019-05-27 04:09:59 tz: America/New_York >>>>>> obsTimeUtc: 2019-05-27T04:09:59Z obsTimeLocal: 2019-05-27 00:09:59 >>>>>> >>>>>> >>>>>> pi@nixie:/var/tmp $ tail -n 3 59:19:28:5_wu.txt 0:20:28:5_wu.txt >>>>>> ==> 59:19:28:5_wu.txt <== >>>>>> epoch: 1559087699 date_epoch_local: 2019-05-28 19:54:59 utcepoch: >>>>>> 1559088099 date_epoch_utc: 2019-05-28 23:54:59 tz: America/New_York >>>>>> obsTimeUtc: 2019-05-28T23:54:59Z obsTimeLocal: 2019-05-28 19:54:59 >>>>>> epoch: 1559087702 date_epoch_local: 2019-05-28 19:55:02 utcepoch: >>>>>> 1559088102 date_epoch_utc: 2019-05-28 23:55:02 tz: America/New_York >>>>>> obsTimeUtc: 2019-05-28T23:55:02Z obsTimeLocal: 2019-05-28 19:55:02 >>>>>> Number of WU records: 240 >>>>>> >>>>>> ==> 0:20:28:5_wu.txt <== >>>>>> epoch: 1559015699 date_epoch_local: 2019-05-27 23:54:59 utcepoch: >>>>>> 1559016099 date_epoch_utc: 2019-05-28 03:54:59 tz: America/New_York >>>>>> obsTimeUtc: 2019-05-28T03:54:59Z obsTimeLocal: 2019-05-27 23:54:59 >>>>>> epoch: 1559015999 date_epoch_local: 2019-05-27 23:59:59 utcepoch: >>>>>> 1559016399 date_epoch_utc: 2019-05-28 03:59:59 tz: America/New_York >>>>>> obsTimeUtc: 2019-05-28T03:59:59Z obsTimeLocal: 2019-05-27 23:59:59 >>>>>> Number of WU records: 288 >>>>>> >>>>>> >>>>>> But after my system rolled over midnight localtime, results returned to >>>>>> the correct dates when asking for 5-28: >>>>>> >>>>>> pi@nixie:/var/tmp $ head -3 0:0:29:5_wu.txt >>>>>> Using configuration file /usr/share/weewx4/weewx.conf. >>>>>> epoch: 1559016299 date_epoch_local: 2019-05-28 00:04:59 utcepoch: >>>>>> 1559016699 date_epoch_utc: 2019-05-28 04:04:59 tz: America/New_York >>>>>> obsTimeUtc: 2019-05-28T04:04:59Z obsTimeLocal: 2019-05-28 00:04:59 >>>>>> epoch: 1559016599 date_epoch_local: 2019-05-28 00:09:59 utcepoch: >>>>>> 1559016999 date_epoch_utc: 2019-05-28 04:09:59 tz: America/New_York >>>>>> obsTimeUtc: 2019-05-28T04:09:59Z obsTimeLocal: 2019-05-28 00:09:59 >>>>>> >>>>>> pi@nixie:/var/tmp $ tail -3 0:0:29:5_wu.txt >>>>>> epoch: 1559102099 date_epoch_local: 2019-05-28 23:54:59 utcepoch: >>>>>> 1559102499 date_epoch_utc: 2019-05-29 03:54:59 tz: America/New_York >>>>>> obsTimeUtc: 2019-05-29T03:54:59Z obsTimeLocal: 2019-05-28 23:54:59 >>>>>> epoch: 1559102399 date_epoch_local: 2019-05-28 23:59:59 utcepoch: >>>>>> 1559102799 date_epoch_utc: 2019-05-29 03:59:59 tz: America/New_York >>>>>> obsTimeUtc: 2019-05-29T03:59:59Z obsTimeLocal: 2019-05-28 23:59:59 >>>>>> Number of WU records: 288 >>>>>> >>>>>> >>>>>> Regards, >>>>>> \Leon >>>>>> -- >>>>>> Leon Shaner :: Dearborn, Michigan (iPad Pro) >>>>>> >>>>>> On May 29, 2019, at 8:53 PM, Rod Yager <[email protected]> wrote: >>>>>> >>>>>>> Further to this, it has now rolled past 10am here, so the local date is >>>>>>> now the same as the UTC date. (ie. local time May 30 2019 10:40 AM = >>>>>>> May 30 2019 00:40 AM UTC). >>>>>>> >>>>>>> Now I get: >>>>>>> >>>>>>> ./wunderfixer --verbose --date=2019-05-29 --epsilon=125 >>>>>>> >>>>>>> Using configuration file /home/weewx/weewx.conf. >>>>>>> >>>>>>> Using database binding 'wx_binding', which is bound to database >>>>>>> 'archive_mysql' >>>>>>> >>>>>>> Weather Underground Station: xxxxx >>>>>>> >>>>>>> Date to check: 2019-05-29 >>>>>>> >>>>>>> Number of archive records: 288 >>>>>>> >>>>>>> Number of WU records: 288 >>>>>>> >>>>>>> Number of missing records: 0 >>>>>>> >>>>>>> [root@moses bin]# ./wunderfixer --verbose --date=2019-05-30 >>>>>>> --epsilon=125 >>>>>>> >>>>>>> Using configuration file /home/weewx/weewx.conf. >>>>>>> >>>>>>> Using database binding 'wx_binding', which is bound to database >>>>>>> 'archive_mysql' >>>>>>> >>>>>>> Weather Underground Station: xxxxxx >>>>>>> >>>>>>> Date to check: 2019-05-30 >>>>>>> >>>>>>> Number of archive records: 128 >>>>>>> >>>>>>> Number of WU records: 127 >>>>>>> >>>>>>> Number of missing records: 1 >>>>>>> >>>>>>> >>>>>>> This means that WU is now actually providing the records for the date >>>>>>> requested, rather than the day after the requested date. >>>>>>> So it seems that what is happening is that WU is determining whether or >>>>>>> not the current date at the station location is the same as the UTC >>>>>>> date. >>>>>>> If it is, it returns the data for the date as in the request. But if >>>>>>> the local date is different, it makes an (unwanted) adjustment for the >>>>>>> date difference. >>>>>>> >>>>>>> Rod >>>>>>> >>>>>>> >>>>>>>> On Thursday, May 30, 2019 at 9:29:16 AM UTC+10, Leon Shaner wrote: >>>>>>>> Hi, Rod, >>>>>>>> >>>>>>>> Yes and thanks for adding yet another confirmation of the issue. =D >>>>>>>> >>>>>>>> I can show that if I do the query within X hours of my offset of UTC, >>>>>>>> what actually happens is they report 288 records from the day PRIOR to >>>>>>>> the one I am asking about. >>>>>>>> For example, I ask for 20190528 and they give me records for 20190527, >>>>>>>> so *that* is why wunderfixer "thinks" it needs to re-upload everything. >>>>>>>> >>>>>>>> I am in contact with IBM about it and have shown them irrefutable >>>>>>>> proof of the issue. >>>>>>>> They didn't respond back yet, which I expect is because the proof was >>>>>>>> irrefutable. Ha! ;-) >>>>>>>> >>>>>>>> I expect that they're investigating and would rather respond from a >>>>>>>> position of understanding, or with any luck maybe even a quick fix. =D >>>>>>>> >>>>>>>> I meant to follow-up with IBM again this morning, but got waylaid, so >>>>>>>> I'll do that now. >>>>>>>> Thanks again, and for the reminder. =D >>>>>>>> >>>>>>>> Regards, >>>>>>>> \Leon >>>>>>>> -- >>>>>>>> Leon Shaner :: Dearborn, Michigan (iPad Pro) >>>>>>>> >>>>>>>> On May 29, 2019, at 6:14 PM, Rod Yager <[email protected]> wrote: >>>>>>>> >>>>>>>>> There is definitely a time zone issue. I am in the Sydney Australia >>>>>>>>> timezone (UTC +10 hours). >>>>>>>>> >>>>>>>>> It is currently 8am local time on May 30, 2019. (10pm May 29, 2019 >>>>>>>>> UTC) >>>>>>>>> >>>>>>>>> If I execute >>>>>>>>> >>>>>>>>> ./wunderfixer --verbose --date=2019-05-29 --epsilon=125 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> I get >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Using configuration file /home/weewx/weewx.conf. >>>>>>>>> >>>>>>>>> Using database binding 'wx_binding', which is bound to database >>>>>>>>> 'archive_mysql' >>>>>>>>> >>>>>>>>> Weather Underground Station: xxxxxxx >>>>>>>>> >>>>>>>>> Date to check: 2019-05-29 >>>>>>>>> >>>>>>>>> Number of archive records: 288 >>>>>>>>> >>>>>>>>> Number of WU records: 97 >>>>>>>>> >>>>>>>>> >>>>>>>>> Number of missing records: 288 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Now WU actually has 288 records for 2019-05-29. >>>>>>>>> >>>>>>>>> But it only has 97 records for 2019-05-30. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> So it is clear that wunderfixer is downloading the record data for >>>>>>>>> 2019-05-30 from WeatherUnderground and trying to match them with the >>>>>>>>> local records for 2019-30-29. >>>>>>>>> >>>>>>>>> Of course, they all mismatch, and so wunderfixer concludes that it >>>>>>>>> must upload all the data for 2019-05-29. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Hope this narrows down the search for a solution. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Rod >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Monday, May 27, 2019 at 9:35:25 PM UTC+10, Leon Shaner wrote: >>>>>>>>>> >>>>>>>>>> On May 27, 2019, at 12:12 AM, gjr80 <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>>>> On Monday, 27 May 2019 13:16:53 UTC+10, Leon Shaner wrote: >>>>>>>>>>>> [snip] >>>>>>>>>>>> If you can see any shorter paths to a more reliable outcome than I >>>>>>>>>>>> have achieved so far, then you know know know I will be very >>>>>>>>>>>> grateful. =D >>>>>>>>>>> >>>>>>>>>>> I am not sure what local/UTC issue you refer to. When I do a >>>>>>>>>>> api.weather.com/v2/pws/history query on a station to the east of >>>>>>>>>>> Greenwich I am returned all records for the date specified (eg >>>>>>>>>>> 20190525 gives me all records for 25 May 2019), each record >>>>>>>>>>> contains an epoch timestamp which is correct and consistent with 25 >>>>>>>>>>> May 2019. Everything is as I would expect. However, when I perform >>>>>>>>>>> the same query on a station to the west of Greenwich I am returned >>>>>>>>>>> records for the day before the date specified (ie 20190525 gives me >>>>>>>>>>> all records for 24 May 2019 not 25 May 2019), again each record >>>>>>>>>>> contains an epoch timestamp but the timestamp is for the previous >>>>>>>>>>> day Ie 24 May 2019. I have checked a number of data records in the >>>>>>>>>>> stations history table and WU is definitely returning the midnight >>>>>>>>>>> to midnight data for the day before. I have confirmed this >>>>>>>>>>> behaviour with a number of stations both east and west of Greenwich. >>>>>>>>>>> >>>>>>>>>>> I don't think there is a local/UTC time issue, I think WU is having >>>>>>>>>>> some implementation issues and for stations west of Greenwich they >>>>>>>>>>> are returning the wrong day of data. >>>>>>>>>> >>>>>>>>>> Thanks, Gary! This was all very helpful. >>>>>>>>>> In addition to what you've described across the east vs west of GMT, >>>>>>>>>> I get similar behavior if I am within X hours of my local UTC offset >>>>>>>>>> when querying my own station. >>>>>>>>>> Last night as soon as localtime rolled over midnight, the queries >>>>>>>>>> for the previous day were correct. >>>>>>>>>> >>>>>>>>>> --Leon >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Gary >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "weewx-user" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/weewx-user/683a28af-35e4-474e-95a0-f684b9926af0%40googlegroups.com. >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "weewx-user" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>>> an email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/weewx-user/3570f548-caf3-43e8-858d-1b4d8c87d84a%40googlegroups.com. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "weewx-user" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>> an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/weewx-user/388426A6-12D0-45FC-AFF2-E6EB532B61DC%40isylum.org. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "weewx-user" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/weewx-user/184A035F-2BFD-4C43-9584-B84EB083F11F%40isylum.org. >>>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "weewx-user" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/weewx-user/D30A437D-57EF-46B0-A40F-98A7C1D2D793%40yager.net.au. >>>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "weewx-user" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/weewx-user/CB045A39-A0D2-415D-B8C2-B741AF0D263B%40isylum.org. >>> For more options, visit https://groups.google.com/d/optout. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "weewx-user" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/weewx-user/BD67E5D8-2F65-477A-B7DB-AD0C5B984148%40yager.net.au. >> For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-user/AB1F8DAE-A13C-4E1A-BA85-EFE4C6E19FF4%40isylum.org. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/8DC2EED8-5147-4D0F-9FDB-A9964E59DCDC%40isylum.org. For more options, visit https://groups.google.com/d/optout.
