Rod, Thanks! I checked your station via 'today' API just after I sent that note and it looks fine to me, also.
I've set a timer for just before the top of the hour to capture your station's results, and again just after. Should be the same as you doing the test from wunderdates. LOL =D I'mma bug IBM tomorrow as much as my schedule allows. I want them to schedule a real-time meeting so we can go over the outstanding issues and chip away at fixing them in priority order. Clearly e-mail to one person doesn't cut it as a "support" model, even if that one person were, "all that and a bag of chips." :-/ I may have mentioned that I am a persistent SOB. LOL ;-) We'll get there. =D Regards, \Leon -- Leon Shaner :: Dearborn, Michigan (iPad Pro) > On Jun 2, 2019, at 7:28 PM, Rod Yager <[email protected]> wrote: > > The “today" result works as desired when I am ahead of the UTC date (as I am > at the moment and will be for the next 33 minutes) > > [root@moses bin]# date > Mon Jun 3 09:25:10 AEST 2019 > [root@moses bin]# /home/weewx/bin/wunderdates3 --epsilon=125 --station > ISYDNEY155 --test --verbose | (head -n 9 ; tail -n 1) > Using configuration file /home/weewx/weewx.conf. > Weather Underground Station: ISYDNEY155 > Date to check: 2019-06-03 > epoch: 1559484000 date_epoch_local: 2019-06-03 00:00:00 utcepoch: 1559483000 > date_epoch_utc: 2019-06-02 14:00:00 tz: Australia/Sydney obsTimeUtc: > 2019-06-02T14:00:00Z obsTimeLocal: 2019-06-03 00:00:00 > epoch: 1559484300 date_epoch_local: 2019-06-03 00:05:00 utcepoch: 1559483300 > date_epoch_utc: 2019-06-02 14:05:00 tz: Australia/Sydney obsTimeUtc: > 2019-06-02T14:05:00Z obsTimeLocal: 2019-06-03 00:05:00 > epoch: 1559484600 date_epoch_local: 2019-06-03 00:10:00 utcepoch: 1559483600 > date_epoch_utc: 2019-06-02 14:10:00 tz: Australia/Sydney obsTimeUtc: > 2019-06-02T14:10:00Z obsTimeLocal: 2019-06-03 00:10:00 > epoch: 1559484900 date_epoch_local: 2019-06-03 00:15:00 utcepoch: 1559483900 > date_epoch_utc: 2019-06-02 14:15:00 tz: Australia/Sydney obsTimeUtc: > 2019-06-02T14:15:00Z obsTimeLocal: 2019-06-03 00:15:00 > epoch: 1559485200 date_epoch_local: 2019-06-03 00:20:00 utcepoch: 1559484200 > date_epoch_utc: 2019-06-02 14:20:00 tz: Australia/Sydney obsTimeUtc: > 2019-06-02T14:20:00Z obsTimeLocal: 2019-06-03 00:20:00 > epoch: 1559485500 date_epoch_local: 2019-06-03 00:25:00 utcepoch: 1559484500 > date_epoch_utc: 2019-06-02 14:25:00 tz: Australia/Sydney obsTimeUtc: > 2019-06-02T14:25:00Z obsTimeLocal: 2019-06-03 00:25:00 > Number of WU records: 112 > > > So the problem only exists for 6 days [ yesterday and the preceeding days] > > Rod > >> On 3 Jun 2019, at 9:12 am, Leon Shaner <[email protected]> wrote: >> >> Rod, >> >> Thanks again for the added validation of what I saw earlier. >> Never minding for now the issues with last week's data, what I am now most >> concerned to know is whether there is any point throughout the PRESENT day >> (your local time) that the latest wunderdates is showing the wrong results? >> >> Recall that for the present day I am using a 'today' API which is for the >> 'today' results. >> That method is working for my station at all times during the PRESENT day >> (even within my UTC offset of local midnight). >> >> I thought I understood you to say that those changes were no good for you at >> certain times of the PRESENT day, such as before 10 a.m. your local time, >> was it? >> >> Regards, >> \Leon >> -- >> Leon Shaner :: Dearborn, Michigan (iPad Pro) >> >> On Jun 2, 2019, at 5:50 PM, Rod Yager <[email protected]> wrote: >> >>> Well, it turns out that the East of the UTC date line bug only occurs for >>> the latest week. (I’d assumed that testing involving requests for the last >>> few days would show the behaviour, but that turned out to be wrong.) >>> >>> At the moment, I’m a day ahead of UTC. Here is the output for requests for >>> 2019-05-27 (within the last week and giving data for the 2019-05-28) and >>> for 2019-05-26 (not within the last week and giving the correct data for >>> 2019-05-06) [I repeated the 2019-05-27 responses to show that they hadn’t >>> happened coincidentally at a time where WU fixed the problem.] All dates >>> I’ve tested prior to 2019-05-27 also work as we would like. All dates >>> within the last week give data for the day following the day in the request. >>> >>> [rodyager@moses ~]$ date >>> Mon Jun 3 07:32:55 AEST 2019 >>> [rodyager@moses ~]$ /home/weewx/bin/wunderdates --epsilon=125 --date >>> 2019-05-27 --station ISYDNEY155 --test --verbose | (head -n 8 ; tail -n 1) >>> Using configuration file /home/weewx/weewx.conf. >>> Weather Underground Station: ISYDNEY155 >>> Date to check: 2019-05-27 >>> epoch: 1558965600 date_epoch_local: 2019-05-28 00:00:00 utcepoch: >>> 1558964600 date_epoch_utc: 2019-05-27 14:00:00 tz: Australia/Sydney >>> obsTimeUtc: 2019-05-27T14:00:00Z obsTimeLocal: 2019-05-28 00:00:00 >>> epoch: 1558965900 date_epoch_local: 2019-05-28 00:05:00 utcepoch: >>> 1558964900 date_epoch_utc: 2019-05-27 14:05:00 tz: Australia/Sydney >>> obsTimeUtc: 2019-05-27T14:05:00Z obsTimeLocal: 2019-05-28 00:05:00 >>> epoch: 1558966200 date_epoch_local: 2019-05-28 00:10:00 utcepoch: >>> 1558965200 date_epoch_utc: 2019-05-27 14:10:00 tz: Australia/Sydney >>> obsTimeUtc: 2019-05-27T14:10:00Z obsTimeLocal: 2019-05-28 00:10:00 >>> epoch: 1558966500 date_epoch_local: 2019-05-28 00:15:00 utcepoch: >>> 1558965500 date_epoch_utc: 2019-05-27 14:15:00 tz: Australia/Sydney >>> obsTimeUtc: 2019-05-27T14:15:00Z obsTimeLocal: 2019-05-28 00:15:00 >>> epoch: 1558966800 date_epoch_local: 2019-05-28 00:20:00 utcepoch: >>> 1558965800 date_epoch_utc: 2019-05-27 14:20:00 tz: Australia/Sydney >>> obsTimeUtc: 2019-05-27T14:20:00Z obsTimeLocal: 2019-05-28 00:20:00 >>> Number of WU records: 288 >>> [rodyager@moses ~]$ /home/weewx/bin/wunderdates --epsilon=125 --date >>> 2019-05-26 --station ISYDNEY155 --test --verbose | (head -n 8 ; tail -n 1) >>> Using configuration file /home/weewx/weewx.conf. >>> Weather Underground Station: ISYDNEY155 >>> Date to check: 2019-05-26 >>> epoch: 1558792800 date_epoch_local: 2019-05-26 00:00:00 utcepoch: >>> 1558791800 date_epoch_utc: 2019-05-25 14:00:00 tz: Australia/Sydney >>> obsTimeUtc: 2019-05-25T14:00:00Z obsTimeLocal: 2019-05-26 00:00:00 >>> epoch: 1558793100 date_epoch_local: 2019-05-26 00:05:00 utcepoch: >>> 1558792100 date_epoch_utc: 2019-05-25 14:05:00 tz: Australia/Sydney >>> obsTimeUtc: 2019-05-25T14:05:00Z obsTimeLocal: 2019-05-26 00:05:00 >>> epoch: 1558793400 date_epoch_local: 2019-05-26 00:10:00 utcepoch: >>> 1558792400 date_epoch_utc: 2019-05-25 14:10:00 tz: Australia/Sydney >>> obsTimeUtc: 2019-05-25T14:10:00Z obsTimeLocal: 2019-05-26 00:10:00 >>> epoch: 1558793700 date_epoch_local: 2019-05-26 00:15:00 utcepoch: >>> 1558792700 date_epoch_utc: 2019-05-25 14:15:00 tz: Australia/Sydney >>> obsTimeUtc: 2019-05-25T14:15:00Z obsTimeLocal: 2019-05-26 00:15:00 >>> epoch: 1558794000 date_epoch_local: 2019-05-26 00:20:00 utcepoch: >>> 1558793000 date_epoch_utc: 2019-05-25 14:20:00 tz: Australia/Sydney >>> obsTimeUtc: 2019-05-25T14:20:00Z obsTimeLocal: 2019-05-26 00:20:00 >>> Number of WU records: 288 >>> [rodyager@moses ~]$ /home/weewx/bin/wunderdates --epsilon=125 --date >>> 2019-05-27 --station ISYDNEY155 --test --verbose | (head -n 8 ; tail -n 1) >>> Using configuration file /home/weewx/weewx.conf. >>> Weather Underground Station: ISYDNEY155 >>> Date to check: 2019-05-27 >>> epoch: 1558965600 date_epoch_local: 2019-05-28 00:00:00 utcepoch: >>> 1558964600 date_epoch_utc: 2019-05-27 14:00:00 tz: Australia/Sydney >>> obsTimeUtc: 2019-05-27T14:00:00Z obsTimeLocal: 2019-05-28 00:00:00 >>> epoch: 1558965900 date_epoch_local: 2019-05-28 00:05:00 utcepoch: >>> 1558964900 date_epoch_utc: 2019-05-27 14:05:00 tz: Australia/Sydney >>> obsTimeUtc: 2019-05-27T14:05:00Z obsTimeLocal: 2019-05-28 00:05:00 >>> epoch: 1558966200 date_epoch_local: 2019-05-28 00:10:00 utcepoch: >>> 1558965200 date_epoch_utc: 2019-05-27 14:10:00 tz: Australia/Sydney >>> obsTimeUtc: 2019-05-27T14:10:00Z obsTimeLocal: 2019-05-28 00:10:00 >>> epoch: 1558966500 date_epoch_local: 2019-05-28 00:15:00 utcepoch: >>> 1558965500 date_epoch_utc: 2019-05-27 14:15:00 tz: Australia/Sydney >>> obsTimeUtc: 2019-05-27T14:15:00Z obsTimeLocal: 2019-05-28 00:15:00 >>> epoch: 1558966800 date_epoch_local: 2019-05-28 00:20:00 utcepoch: >>> 1558965800 date_epoch_utc: 2019-05-27 14:20:00 tz: Australia/Sydney >>> obsTimeUtc: 2019-05-27T14:20:00Z obsTimeLocal: 2019-05-28 00:20:00 >>> Number of WU records: 288 >>> >>> >>> Rod >>> >>> >>>> On 3 Jun 2019, at 5:47 am, Leon Shaner <[email protected]> wrote: >>>> >>>> Rod, >>>> >>>> I've spent quite a bit of time working on a "cleaner" workaround for the >>>> WU bugs. >>>> But just when I thought I had landed on the "reasonable" approach, I hit a >>>> snag. >>>> >>>> As a reminder, with the latest approach, all is fine for stations west of >>>> the UTC date line. What I am working on now is a workaround for stations >>>> east of the UTC date line, where the data returned is a day after the date >>>> requested. >>>> >>>> The wall that I just hit is that the WU behavior around this latest bug >>>> changes, when looking historically. :-( >>>> >>>> From today's date going back to 2019-05-28, the query returns obsTimeLocal >>>> data from the day after the date requested. This is the bug we are trying >>>> to workaround. >>>> To compensate, I can ask for the prior date to get the date we really want. >>>> >>>> HOWEVER, starting with 2019-05-26 (and prior), the query returns >>>> obsTimeLocal data for the date that was actually requested. >>>> >>>> This leads to the inability to get data for 2019-05-27, because the actual >>>> query for that date returns 2019-05-28 datum, and the compensation date of >>>> 2019-05-26 returns actual data for 2019-05-26. >>>> >>>> I think I am done trying to work around this bug. IBM needs to own it. >>>> >>>> See here -- scroll to second to last example where 2019-05-27 is >>>> requested, but when I tried 2019-05-26, I got back data for the actual >>>> 2019-05-26, so compensation did not work. :-( >>>> >>>> pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates >>>> --epsilon=125 --date 2019-06-02 --station ISYDNEY155 --test --verbose | >>>> (head -n 9 ; tail -n 1) >>>> Using configuration file /usr/share/weewx4/weewx.conf. >>>> Weather Underground Station: ISYDNEY155 >>>> Date to check: 2019-06-02 >>>> WU API obsTimeLocal: 2019-06-03 >>>> WU API WRONG DATE !!!!!!!!!!!!!!!!!!!!!!! >>>> WU API COMPENSATION DATE: 2019-06-01 >>>> WU API obsTimeLocal: 2019-06-02 >>>> epoch: 1559397600 date_epoch_local: 2019-06-01 10:00:00 tz: >>>> Australia/Sydney obsTimeUtc: 2019-06-01T14:00:00Z obsTimeLocal: 2019-06-02 >>>> 00:00:00 >>>> epoch: 1559397900 date_epoch_local: 2019-06-01 10:05:00 tz: >>>> Australia/Sydney obsTimeUtc: 2019-06-01T14:05:00Z obsTimeLocal: 2019-06-02 >>>> 00:05:00 >>>> Number of WU records: 288 >>>> >>>> pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates >>>> --epsilon=125 --date 2019-06-01 --station ISYDNEY155 --test --verbose | >>>> (head -n 9 ; tail -n 1) >>>> Using configuration file /usr/share/weewx4/weewx.conf. >>>> Weather Underground Station: ISYDNEY155 >>>> Date to check: 2019-06-01 >>>> WU API obsTimeLocal: 2019-06-02 >>>> WU API WRONG DATE !!!!!!!!!!!!!!!!!!!!!!! >>>> WU API COMPENSATION DATE: 2019-05-31 >>>> WU API obsTimeLocal: 2019-06-01 >>>> epoch: 1559311200 date_epoch_local: 2019-05-31 10:00:00 tz: >>>> Australia/Sydney obsTimeUtc: 2019-05-31T14:00:00Z obsTimeLocal: 2019-06-01 >>>> 00:00:00 >>>> epoch: 1559311500 date_epoch_local: 2019-05-31 10:05:00 tz: >>>> Australia/Sydney obsTimeUtc: 2019-05-31T14:05:00Z obsTimeLocal: 2019-06-01 >>>> 00:05:00 >>>> Number of WU records: 288 >>>> >>>> pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates >>>> --epsilon=125 --date 2019-05-31 --station ISYDNEY155 --test --verbose | >>>> (head -n 9 ; tail -n 1) >>>> Using configuration file /usr/share/weewx4/weewx.conf. >>>> Weather Underground Station: ISYDNEY155 >>>> Date to check: 2019-05-31 >>>> WU API obsTimeLocal: 2019-06-01 >>>> WU API WRONG DATE !!!!!!!!!!!!!!!!!!!!!!! >>>> WU API COMPENSATION DATE: 2019-05-30 >>>> WU API obsTimeLocal: 2019-05-31 >>>> epoch: 1559224800 date_epoch_local: 2019-05-30 10:00:00 tz: >>>> Australia/Sydney obsTimeUtc: 2019-05-30T14:00:00Z obsTimeLocal: 2019-05-31 >>>> 00:00:00 >>>> epoch: 1559225100 date_epoch_local: 2019-05-30 10:05:00 tz: >>>> Australia/Sydney obsTimeUtc: 2019-05-30T14:05:00Z obsTimeLocal: 2019-05-31 >>>> 00:05:00 >>>> Number of WU records: 288 >>>> >>>> pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates >>>> --epsilon=125 --date 2019-05-30 --station ISYDNEY155 --test --verbose | >>>> (head -n 9 ; tail -n 1) >>>> Using configuration file /usr/share/weewx4/weewx.conf. >>>> Weather Underground Station: ISYDNEY155 >>>> Date to check: 2019-05-30 >>>> WU API obsTimeLocal: 2019-05-31 >>>> WU API WRONG DATE !!!!!!!!!!!!!!!!!!!!!!! >>>> WU API COMPENSATION DATE: 2019-05-29 >>>> WU API obsTimeLocal: 2019-05-30 >>>> epoch: 1559138400 date_epoch_local: 2019-05-29 10:00:00 tz: >>>> Australia/Sydney obsTimeUtc: 2019-05-29T14:00:00Z obsTimeLocal: 2019-05-30 >>>> 00:00:00 >>>> epoch: 1559138700 date_epoch_local: 2019-05-29 10:05:00 tz: >>>> Australia/Sydney obsTimeUtc: 2019-05-29T14:05:00Z obsTimeLocal: 2019-05-30 >>>> 00:05:00 >>>> Number of WU records: 288 >>>> >>>> pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates >>>> --epsilon=125 --date 2019-05-29 --station ISYDNEY155 --test --verbose | >>>> (head -n 9 ; tail -n 1) >>>> Using configuration file /usr/share/weewx4/weewx.conf. >>>> Weather Underground Station: ISYDNEY155 >>>> Date to check: 2019-05-29 >>>> WU API obsTimeLocal: 2019-05-30 >>>> WU API WRONG DATE !!!!!!!!!!!!!!!!!!!!!!! >>>> WU API COMPENSATION DATE: 2019-05-28 >>>> WU API obsTimeLocal: 2019-05-29 >>>> epoch: 1559052000 date_epoch_local: 2019-05-28 10:00:00 tz: >>>> Australia/Sydney obsTimeUtc: 2019-05-28T14:00:00Z obsTimeLocal: 2019-05-29 >>>> 00:00:00 >>>> epoch: 1559052300 date_epoch_local: 2019-05-28 10:05:00 tz: >>>> Australia/Sydney obsTimeUtc: 2019-05-28T14:05:00Z obsTimeLocal: 2019-05-29 >>>> 00:05:00 >>>> Number of WU records: 288 >>>> >>>> pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates >>>> --epsilon=125 --date 2019-05-28 --station ISYDNEY155 --test --verbose | >>>> (head -n 9 ; tail -n 1) >>>> Using configuration file /usr/share/weewx4/weewx.conf. >>>> Weather Underground Station: ISYDNEY155 >>>> Date to check: 2019-05-28 >>>> WU API obsTimeLocal: 2019-05-29 >>>> WU API WRONG DATE !!!!!!!!!!!!!!!!!!!!!!! >>>> WU API COMPENSATION DATE: 2019-05-27 >>>> WU API obsTimeLocal: 2019-05-28 >>>> epoch: 1558965600 date_epoch_local: 2019-05-27 10:00:00 tz: >>>> Australia/Sydney obsTimeUtc: 2019-05-27T14:00:00Z obsTimeLocal: 2019-05-28 >>>> 00:00:00 >>>> epoch: 1558965900 date_epoch_local: 2019-05-27 10:05:00 tz: >>>> Australia/Sydney obsTimeUtc: 2019-05-27T14:05:00Z obsTimeLocal: 2019-05-28 >>>> 00:05:00 >>>> Number of WU records: 288 >>>> >>>> /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-05-27 >>>> --station ISYDNEY155 --test --verbose | (head -n 9 ; tail -n 1) >>>> Using configuration file /usr/share/weewx4/weewx.conf. >>>> Weather Underground Station: ISYDNEY155 >>>> Date to check: 2019-05-27 >>>> WU API obsTimeLocal: 2019-05-28 >>>> WU API WRONG DATE !!!!!!!!!!!!!!!!!!!!!!! >>>> WU API COMPENSATION DATE: 2019-05-26 >>>> WU API obsTimeLocal: 2019-05-26 >>>> WU API COMPSENSATION FAILURE! ABORTING!!! >>>> Number of WU records: 0 >>>> >>>> No results returned from Weather Underground (perhaps a bad station >>>> name??). >>>> >>>> >>>> pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates >>>> --epsilon=125 --date 2019-05-26 --station ISYDNEY155 --test --verbose | >>>> (head -n 9 ; tail -n 1) >>>> Using configuration file /usr/share/weewx4/weewx.conf. >>>> Weather Underground Station: ISYDNEY155 >>>> Date to check: 2019-05-26 >>>> WU API obsTimeLocal: 2019-05-26 >>>> epoch: 1558792800 date_epoch_local: 2019-05-25 10:00:00 tz: >>>> Australia/Sydney obsTimeUtc: 2019-05-25T14:00:00Z obsTimeLocal: 2019-05-26 >>>> 00:00:00 >>>> epoch: 1558793100 date_epoch_local: 2019-05-25 10:05:00 tz: >>>> Australia/Sydney obsTimeUtc: 2019-05-25T14:05:00Z obsTimeLocal: 2019-05-26 >>>> 00:05:00 >>>> epoch: 1558793400 date_epoch_local: 2019-05-25 10:10:00 tz: >>>> Australia/Sydney obsTimeUtc: 2019-05-25T14:10:00Z obsTimeLocal: 2019-05-26 >>>> 00:10:00 >>>> epoch: 1558793700 date_epoch_local: 2019-05-25 10:15:00 tz: >>>> Australia/Sydney obsTimeUtc: 2019-05-25T14:15:00Z obsTimeLocal: 2019-05-26 >>>> 00:15:00 >>>> epoch: 1558794000 date_epoch_local: 2019-05-25 10:20:00 tz: >>>> Australia/Sydney obsTimeUtc: 2019-05-25T14:20:00Z obsTimeLocal: 2019-05-26 >>>> 00:20:00 >>>> Number of WU records: 288 >>>> >>>> Regards, >>>> \Leon >>>> -- >>>> Leon Shaner :: Dearborn, Michigan (iPad Pro) >>>> >>>> On Jun 2, 2019, at 10:20 AM, Leon Shaner <[email protected]> wrote: >>>> >>>>> Rod, >>>>> >>>>> It's interesting that the "workaround" that I've already done, is >>>>> actually working 100% for me. I got good results when combining the >>>>> 'today' API when asking for the current date, and only using the >>>>> 'historical' result when asking for dates other than the current date -- >>>>> relative to station local time. >>>>> >>>>> Here is my KMIDEARB5 station's transition from 7:59 p.m. to 8:00 p.m, >>>>> where time moves to within the UTC-4 offset: >>>>> >>>>> $ diff 59:19:1:6_wu.txt 0:20:1:6_wu.txt >>>>> 241c241,242 >>>>> < Number of WU records: 239 >>>>> --- >>>>> > epoch: 1559433599 date_epoch_local: 2019-06-01 19:59:59 utcepoch: >>>>> > 1559433999 date_epoch_utc: 2019-06-01 23:59:59 tz: America/New_York >>>>> > obsTimeUtc: 2019-06-01T23:59:59Z obsTimeLocal: 2019-06-01 19:59:59 >>>>> > Number of WU records: 240 >>>>> >>>>> The above is exactly correct -- the second query is one minute later and >>>>> is after/on the 5-minute record "boundary" hence one additional record. >>>>> >>>>> And then as my station transition from 11:59 p.m. to midnight next day. >>>>> Before midnight the 'today' API is used, but after midnight the >>>>> 'historical' API will be used, which was always working for me after >>>>> midnight (station local time): >>>>> >>>>> $ tail -n 4 59:23:1:6_wu.txt 0:0:2:6_wu.txt >>>>> ==> 59:23:1:6_wu.txt <== >>>>> epoch: 1559447399 date_epoch_local: 2019-06-01 23:49:59 utcepoch: >>>>> 1559447799 date_epoch_utc: 2019-06-02 03:49:59 tz: America/New_York >>>>> obsTimeUtc: 2019-06-02T03:49:59Z obsTimeLocal: 2019-06-01 23:49:59 >>>>> epoch: 1559447699 date_epoch_local: 2019-06-01 23:54:59 utcepoch: >>>>> 1559448099 date_epoch_utc: 2019-06-02 03:54:59 tz: America/New_York >>>>> obsTimeUtc: 2019-06-02T03:54:59Z obsTimeLocal: 2019-06-01 23:54:59 >>>>> epoch: 1559447701 date_epoch_local: 2019-06-01 23:55:01 utcepoch: >>>>> 1559448101 date_epoch_utc: 2019-06-02 03:55:01 tz: America/New_York >>>>> obsTimeUtc: 2019-06-02T03:55:01Z obsTimeLocal: 2019-06-01 23:55:01 >>>>> Number of WU records: 288 >>>>> >>>>> ==> 0:0:2:6_wu.txt <== >>>>> epoch: 1559447399 date_epoch_local: 2019-06-01 23:49:59 utcepoch: >>>>> 1559447799 date_epoch_utc: 2019-06-02 03:49:59 tz: America/New_York >>>>> obsTimeUtc: 2019-06-02T03:49:59Z obsTimeLocal: 2019-06-01 23:49:59 >>>>> epoch: 1559447699 date_epoch_local: 2019-06-01 23:54:59 utcepoch: >>>>> 1559448099 date_epoch_utc: 2019-06-02 03:54:59 tz: America/New_York >>>>> obsTimeUtc: 2019-06-02T03:54:59Z obsTimeLocal: 2019-06-01 23:54:59 >>>>> epoch: 1559447999 date_epoch_local: 2019-06-01 23:59:59 utcepoch: >>>>> 1559448399 date_epoch_utc: 2019-06-02 03:59:59 tz: America/New_York >>>>> obsTimeUtc: 2019-06-02T03:59:59Z obsTimeLocal: 2019-06-01 23:59:59 >>>>> Number of WU records: 288 >>>>> >>>>> There again, the second query is a minute later, so one more result, but >>>>> with explanation... ;-) >>>>> >>>>> There is a slight difference between the 'today' and 'historical' result. >>>>> The first result, using the 'today' API ran at 1 minute before midnight, >>>>> which was on the cusp of posting the 1-minute record nearest actual >>>>> midnight. >>>>> The 'today' result shows two time-stamps very close to 11:55 p.m. >>>>> Where-as the 'historical' result replaces the later ~11:55 p.m. result >>>>> with the ~midnight result. >>>>> >>>>> Those differences are "in the noise" and inconsequential. Even the >>>>> default wunderfixer --epsilon=120 will filter out such a duplicate. >>>>> >>>>> I am happy that this new approach with the combined 'today' and >>>>> 'historical' queries is at least effective for stations west of the UTC >>>>> dateline. >>>>> It means I have easily worked around half of the WU API bug. >>>>> >>>>> Now to further press IBM to fix the other half of the bug... =D >>>>> (Even while I continue to think about ways to *cleanly* workaround the >>>>> bug for stations east of UTC dateline). >>>>> >>>>> Regards, >>>>> \Leon >>>>> -- >>>>> Leon Shaner :: Dearborn, Michigan (iPad Pro) >>>>> >>>>> On Jun 1, 2019, at 11:08 PM, Rod Yager <[email protected]> wrote: >>>>> >>>>>> Dear Leon, >>>>>> >>>>>> If WU doesn’t fix their bug, the solution is really quite simple. >>>>>> >>>>>> We need to make 3 requests to WU for the data: one request for the date >>>>>> we are interested in, one request for the day before and one request for >>>>>> the day after. >>>>>> >>>>>> eg. If we are intending to upload missing data for 2019-01-01 we need to >>>>>> ask WU for the records it has for 2018-12-31, 2019-01-01 and 2019-01-02. >>>>>> >>>>>> One of these will have the information pertaining to the day we are >>>>>> interested in. >>>>>> >>>>>> We then need to filter the three responses from WU, only keeping the >>>>>> responses that pertain to the day we are interested in. >>>>>> >>>>>> At this point, we have exactly what we want from WU and so we compare >>>>>> that with the local database, and upload the data that is missing in the >>>>>> WU record, just as we always did. >>>>>> >>>>>> >>>>>> in your timezone (UTC-4) if you ask for data for your station the >>>>>> correct information will be in >>>>>> the same day request: if the request is made between 00:00 and 19:59 >>>>>> (at those times, the UTC date is the same as the date where you are ) >>>>>> the request for the next day: if the request is made between 20:00 and >>>>>> 23:59 (at those times, the UTC date is one day ahead the date where you >>>>>> are) >>>>>> >>>>>> If you request information for my station (I’ve sent you the details in >>>>>> a PM), the correct information will be in >>>>>> the same day request: if the request is made between 10:00 and 19:59 >>>>>> your time (at those times, the UTC date is the same as the date where >>>>>> my station is located) >>>>>> the request for the previous day: if the request is made between 0:00 >>>>>> and 09:59 or between 20:00 and 23:59 your time (at those times, the UTC >>>>>> date is one day behind the date where my station is located) >>>>>> >>>>>> Rod >>>>>> >>>>>> >>>>>> >>>>>>> On 2 Jun 2019, at 9:27 am, Leon Shaner <[email protected]> wrote: >>>>>>> >>>>>>> Rod, >>>>>>> >>>>>>> Thanks for testing. Well, WU's approach is flawed, not mine, really. >>>>>>> :S >>>>>>> >>>>>>> It means that both API's, the one I just added for 'today" and the one >>>>>>> from before that was for 'historical' (including today), both have the >>>>>>> same bug. >>>>>>> >>>>>>> For now, the best workaround I can think of is to only ever run the new >>>>>>> wunderfixer only ever for the previous day, only ever after UTC has >>>>>>> rolled over midnight. :-( >>>>>>> E.g. when it is 2019-06-02 in UTC, you can get the right results with >>>>>>> wunderfixer if you ask for 2019-06-01 only. In other words, always >>>>>>> delay your wunderfixer fix ups by a day, operating always on >>>>>>> yesterday's date (relative to UTC) after UTC has rolled over to the >>>>>>> next day. >>>>>>> >>>>>>> Scroll to see the output of me running wunderdates on my local machine >>>>>>> in UTC-4 against your station in UTC+10. >>>>>>> >>>>>>> Because I am asking for 2019-06-01, which is also my local date, the >>>>>>> logic will use the URL that queries for 'today' and instead of >>>>>>> returning obsTimeLocal records, it is returning obsTimeUtc records for >>>>>>> the requested date. We can see that it's returning 2019-06-02 relative >>>>>>> to your station, even though I asked for 2019-06-01. >>>>>>> >>>>>>> Sure, I could go a bit further and use a related API query to check the >>>>>>> timezone of your station and use date conversions to check if the date >>>>>>> requested is 'today' relative to the station requested, but really this >>>>>>> is intended to be run on the same machine as weewx, so the localtime >>>>>>> date should match that of the station. It means there isn't any real >>>>>>> reason for me to workaround their bug. In fact even if I did, then the >>>>>>> logic would just fall through to the historical data URL, which we >>>>>>> already know is buggy. >>>>>>> >>>>>>> For now, I'll just report this to IBM a bug in the 'today' API, similar >>>>>>> to the bug in the 'historical' API. >>>>>>> >>>>>>> $ ./wunderdates --epsilon=125 --date 2019-06-01 --station ISYDNEY155 >>>>>>> --test --verbose | more >>>>>>> >>>>>>> Using configuration file /usr/share/weewx4/weewx.conf. >>>>>>> Weather Underground Station: ISYDNEY155 >>>>>>> Date to check: 2019-06-01 >>>>>>> epoch: 1559397600 date_epoch_local: 2019-06-01 10:00:00 utcepoch: >>>>>>> 1559398000 date_epoch_utc: 2019-06-01 14:00:00 tz: Australia/Sydney >>>>>>> obsTimeUtc: 2019-06-01T14:00:00Z obsTimeLocal: 2019-06-02 00:00:00 >>>>>>> epoch: 1559397900 date_epoch_local: 2019-06-01 10:05:00 utcepoch: >>>>>>> 1559398300 date_epoch_utc: 2019-06-01 14:05:00 tz: Australia/Sydney >>>>>>> obsTimeUtc: 2019-06-01T14:05:00Z obsTimeLocal: 2019-06-02 00:05:00 >>>>>>> ... >>>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> \Leon >>>>>>> -- >>>>>>> Leon Shaner :: Dearborn, Michigan (iPad Pro) >>>>>>> >>>>>>> On Jun 1, 2019, at 6:03 PM, Rod Yager <[email protected]> wrote: >>>>>>> >>>>>>>> Dear Leon, >>>>>>>> >>>>>>>> Your approach is flawed. It won’t work for historical data. >>>>>>>> >>>>>>>> If you run >>>>>>>> >>>>>>>> wunderdates —date=2019-01-01 at 9pm your time you will receive the >>>>>>>> data for 2018-12-31 (because your station is a day behind UTC at the >>>>>>>> time of the request, so it gives you the data for the previous day). >>>>>>>> >>>>>>>> On the other hand, if you run >>>>>>>> >>>>>>>> wunderdates —date=2019-01-01 at 9am your time you will receive the >>>>>>>> data for 2019-01-01 (because your station is in the same day as UTC >>>>>>>> at the time of the request). >>>>>>>> >>>>>>>> If I do the same thing (I’m in UTC+10) >>>>>>>> >>>>>>>> wunderdates —date=2019-01-01 at 9pm my time produces the data for >>>>>>>> 2019-01-01 (because my station is in the same day as UTC at the time >>>>>>>> of the request) but >>>>>>>> wunderdates —date=2019-01-01 at 9am my time produces the data for >>>>>>>> 2019-01-02 (because my station is a day ahead of UTC at the time of >>>>>>>> the request) >>>>>>>> >>>>>>>> Rod >>>>>>>> >>>>>>>>> On 2 Jun 2019, at 3:17 am, Leon Shaner <[email protected]> wrote: >>>>>>>>> >>>>>>>>> All, >>>>>>>>> >>>>>>>>> I'm testing a new approach. Below you will find links to an updated >>>>>>>>> wunderdates utility that can be used to validate whether I am on the >>>>>>>>> right track. >>>>>>>>> The wunderdates utility simply dumps out timestamp related records >>>>>>>>> from what WU returns from the query. >>>>>>>>> >>>>>>>>> We're looking mainly at the obsTimeUtc and obsTimeLocal values, which >>>>>>>>> were demonstrated under the prior approach to be returning the wrong >>>>>>>>> dates when within the stations localtime +/- offset from UTC. >>>>>>>>> >>>>>>>>> The new approach is to detect if the requested date is 'today' and if >>>>>>>>> so, use a different API URL that already assumes 'today' and will >>>>>>>>> hopefully not be subject to the UTC offset bugs we've been chasing >>>>>>>>> with the historical data API URL. >>>>>>>>> >>>>>>>>> I have my crontab set to do another test approaching my UTC offset, >>>>>>>>> just after coming within the offset, and then again just before and >>>>>>>>> after midnight localtime. >>>>>>>>> (Same test I did before, but now with the new approach in place). >>>>>>>>> >>>>>>>>> Here is the utility for anyone else that wants to check out the >>>>>>>>> behavior: >>>>>>>>> >>>>>>>>> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates3 >>>>>>>>> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates4 >>>>>>>>> >>>>>>>>> Which version you pull will depend on which base weewx you are >>>>>>>>> running. >>>>>>>>> Pull the one that matches your weewx version and place it in bin, >>>>>>>>> next to wunderfixer, and it will take the same arguments as >>>>>>>>> wunderfixer. >>>>>>>>> >>>>>>>>> You can just try wunderfixer as you normally would (with --apikey) >>>>>>>>> and then run wunderdates(3 or 4) with exact same arguments to be able >>>>>>>>> to see what actual timestamps WU is sending back for the date queried. >>>>>>>>> Parameters like --epsilon don't have any effect in the case of >>>>>>>>> wunderdates, but I left it there so you don't have to change options >>>>>>>>> when running the util to get the debug output. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> \Leon >>>>>>>>> -- >>>>>>>>> Leon Shaner :: Dearborn, Michigan (iPad Pro) >>>>>>>>> >>>>>>>>> On May 30, 2019, at 12:40 AM, Leon Shaner <[email protected]> wrote: >>>>>>>>> >>>>>>>>>> Rod, >>>>>>>>>> Thanks again for this. >>>>>>>>>> >>>>>>>>>> Since the in-progress version of wunderfixer doesn't really show you >>>>>>>>>> the dates that come back from WU, I have written a tool just to >>>>>>>>>> debug the dates. >>>>>>>>>> >>>>>>>>>> The command-line input and basing of defaults on weewx.conf works >>>>>>>>>> the same as wunderfixer, except it doesn't look at your DB at all. >>>>>>>>>> It only prints out datestamps in various incarnations coming back >>>>>>>>>> from WU and as compared to your system's localtime. >>>>>>>>>> >>>>>>>>>> It would be helpful to see the "wunderdates" output at times like >>>>>>>>>> you've shown below, a la before and after your localtime rolls >>>>>>>>>> around to UTC midnight. >>>>>>>>>> >>>>>>>>>> Since you are at UTC + 10, another interesting time would be 11+ >>>>>>>>>> hours on either side of UTC midnight, in addition to within 9 or >>>>>>>>>> less hours. This is just to make sure we're covering all the corner >>>>>>>>>> cases. >>>>>>>>>> >>>>>>>>>> Gary reported a difference for stations that are east vs. west of >>>>>>>>>> GMT, and I expect we're really chasing the same bug in that there is >>>>>>>>>> some bad math WU is doing based on UTC offset, but since an offset >>>>>>>>>> can be +/-, the effects go in opposite directions date-wise, >>>>>>>>>> depending on which side of the UTC dateline your station is located. >>>>>>>>>> >>>>>>>>>> At least that is what I surmise may be happening, but the >>>>>>>>>> wunderdates utility should shed light one way or the other. =D >>>>>>>>>> >>>>>>>>>> The wunderdates utility is available at the links below. >>>>>>>>>> >>>>>>>>>> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates3 >>>>>>>>>> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates4 >>>>>>>>>> >>>>>>>>>> Which version you pull will depend on which base weewx you are >>>>>>>>>> running. >>>>>>>>>> Pull the one that matches your weewx version and place it in bin, >>>>>>>>>> next to wunderfixer, and it will take the same arguments as >>>>>>>>>> wunderfixer. >>>>>>>>>> >>>>>>>>>> You can just try wunderfixer as you normally would (with --apikey) >>>>>>>>>> and then run wunderdates(3 or 4) with exact same arguments to be >>>>>>>>>> able to see what actual timestamps WU is sending back for the date >>>>>>>>>> queried. >>>>>>>>>> Parameters like --epsilon don't have any effect in the case of >>>>>>>>>> wunderdates, but I left it there so you don't have to change options >>>>>>>>>> when running the util to get the debug output. >>>>>>>>>> >>>>>>>>>> What I've been doing is saving the output to files for use with >>>>>>>>>> sdiff (side-by-side) diff. >>>>>>>>>> Or you can just compare the head and tail of each file individually. >>>>>>>>>> >>>>>>>>>> Example for my system before and after my local time on 5-28 once it >>>>>>>>>> was at/after 8 p.m. here, which is within 4 hours of UTC midnight (I >>>>>>>>>> am at UTC-4). >>>>>>>>>> >>>>>>>>>> This output is optimized for screen widths 203 or wider. Sorry. :S >>>>>>>>>> Mainly the last two data fields in the output tell what we need to >>>>>>>>>> know. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> pi@nixie:/var/tmp $ head -n 3 59:19:28:5_wu.txt 0:20:28:5_wu.txt >>>>>>>>>> ==> 59:19:28:5_wu.txt <== >>>>>>>>>> Using configuration file /usr/share/weewx4/weewx.conf. >>>>>>>>>> epoch: 1559016299 date_epoch_local: 2019-05-28 00:04:59 utcepoch: >>>>>>>>>> 1559016699 date_epoch_utc: 2019-05-28 04:04:59 tz: America/New_York >>>>>>>>>> obsTimeUtc: 2019-05-28T04:04:59Z obsTimeLocal: 2019-05-28 00:04:59 >>>>>>>>>> epoch: 1559016599 date_epoch_local: 2019-05-28 00:09:59 utcepoch: >>>>>>>>>> 1559016999 date_epoch_utc: 2019-05-28 04:09:59 tz: America/New_York >>>>>>>>>> obsTimeUtc: 2019-05-28T04:09:59Z obsTimeLocal: 2019-05-28 00:09:59 >>>>>>>>>> >>>>>>>>>> ==> 0:20:28:5_wu.txt <== >>>>>>>>>> Using configuration file /usr/share/weewx4/weewx.conf. >>>>>>>>>> epoch: 1558929899 date_epoch_local: 2019-05-27 00:04:59 utcepoch: >>>>>>>>>> 1558930299 date_epoch_utc: 2019-05-27 04:04:59 tz: America/New_York >>>>>>>>>> obsTimeUtc: 2019-05-27T04:04:59Z obsTimeLocal: 2019-05-27 00:04:59 >>>>>>>>>> epoch: 1558930199 date_epoch_local: 2019-05-27 00:09:59 utcepoch: >>>>>>>>>> 1558930599 date_epoch_utc: 2019-05-27 04:09:59 tz: America/New_York >>>>>>>>>> obsTimeUtc: 2019-05-27T04:09:59Z obsTimeLocal: 2019-05-27 00:09:59 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> pi@nixie:/var/tmp $ tail -n 3 59:19:28:5_wu.txt 0:20:28:5_wu.txt >>>>>>>>>> ==> 59:19:28:5_wu.txt <== >>>>>>>>>> epoch: 1559087699 date_epoch_local: 2019-05-28 19:54:59 utcepoch: >>>>>>>>>> 1559088099 date_epoch_utc: 2019-05-28 23:54:59 tz: America/New_York >>>>>>>>>> obsTimeUtc: 2019-05-28T23:54:59Z obsTimeLocal: 2019-05-28 19:54:59 >>>>>>>>>> epoch: 1559087702 date_epoch_local: 2019-05-28 19:55:02 utcepoch: >>>>>>>>>> 1559088102 date_epoch_utc: 2019-05-28 23:55:02 tz: America/New_York >>>>>>>>>> obsTimeUtc: 2019-05-28T23:55:02Z obsTimeLocal: 2019-05-28 19:55:02 >>>>>>>>>> Number of WU records: 240 >>>>>>>>>> >>>>>>>>>> ==> 0:20:28:5_wu.txt <== >>>>>>>>>> epoch: 1559015699 date_epoch_local: 2019-05-27 23:54:59 utcepoch: >>>>>>>>>> 1559016099 date_epoch_utc: 2019-05-28 03:54:59 tz: America/New_York >>>>>>>>>> obsTimeUtc: 2019-05-28T03:54:59Z obsTimeLocal: 2019-05-27 23:54:59 >>>>>>>>>> epoch: 1559015999 date_epoch_local: 2019-05-27 23:59:59 utcepoch: >>>>>>>>>> 1559016399 date_epoch_utc: 2019-05-28 03:59:59 tz: America/New_York >>>>>>>>>> obsTimeUtc: 2019-05-28T03:59:59Z obsTimeLocal: 2019-05-27 23:59:59 >>>>>>>>>> Number of WU records: 288 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> But after my system rolled over midnight localtime, results returned >>>>>>>>>> to the correct dates when asking for 5-28: >>>>>>>>>> >>>>>>>>>> pi@nixie:/var/tmp $ head -3 0:0:29:5_wu.txt >>>>>>>>>> Using configuration file /usr/share/weewx4/weewx.conf. >>>>>>>>>> epoch: 1559016299 date_epoch_local: 2019-05-28 00:04:59 utcepoch: >>>>>>>>>> 1559016699 date_epoch_utc: 2019-05-28 04:04:59 tz: America/New_York >>>>>>>>>> obsTimeUtc: 2019-05-28T04:04:59Z obsTimeLocal: 2019-05-28 00:04:59 >>>>>>>>>> epoch: 1559016599 date_epoch_local: 2019-05-28 00:09:59 utcepoch: >>>>>>>>>> 1559016999 date_epoch_utc: 2019-05-28 04:09:59 tz: America/New_York >>>>>>>>>> obsTimeUtc: 2019-05-28T04:09:59Z obsTimeLocal: 2019-05-28 00:09:59 >>>>>>>>>> >>>>>>>>>> pi@nixie:/var/tmp $ tail -3 0:0:29:5_wu.txt >>>>>>>>>> epoch: 1559102099 date_epoch_local: 2019-05-28 23:54:59 utcepoch: >>>>>>>>>> 1559102499 date_epoch_utc: 2019-05-29 03:54:59 tz: America/New_York >>>>>>>>>> obsTimeUtc: 2019-05-29T03:54:59Z obsTimeLocal: 2019-05-28 23:54:59 >>>>>>>>>> epoch: 1559102399 date_epoch_local: 2019-05-28 23:59:59 utcepoch: >>>>>>>>>> 1559102799 date_epoch_utc: 2019-05-29 03:59:59 tz: America/New_York >>>>>>>>>> obsTimeUtc: 2019-05-29T03:59:59Z obsTimeLocal: 2019-05-28 23:59:59 >>>>>>>>>> Number of WU records: 288 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> \Leon >>>>>>>>>> -- >>>>>>>>>> Leon Shaner :: Dearborn, Michigan (iPad Pro) >>>>>>>>>> >>>>>>>>>> On May 29, 2019, at 8:53 PM, Rod Yager <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Further to this, it has now rolled past 10am here, so the local >>>>>>>>>>> date is now the same as the UTC date. (ie. local time May 30 2019 >>>>>>>>>>> 10:40 AM = May 30 2019 00:40 AM UTC). >>>>>>>>>>> >>>>>>>>>>> Now I get: >>>>>>>>>>> >>>>>>>>>>> ./wunderfixer --verbose --date=2019-05-29 --epsilon=125 >>>>>>>>>>> >>>>>>>>>>> Using configuration file /home/weewx/weewx.conf. >>>>>>>>>>> >>>>>>>>>>> Using database binding 'wx_binding', which is bound to database >>>>>>>>>>> 'archive_mysql' >>>>>>>>>>> >>>>>>>>>>> Weather Underground Station: xxxxx >>>>>>>>>>> >>>>>>>>>>> Date to check: 2019-05-29 >>>>>>>>>>> >>>>>>>>>>> Number of archive records: 288 >>>>>>>>>>> >>>>>>>>>>> Number of WU records: 288 >>>>>>>>>>> >>>>>>>>>>> Number of missing records: 0 >>>>>>>>>>> >>>>>>>>>>> [root@moses bin]# ./wunderfixer --verbose --date=2019-05-30 >>>>>>>>>>> --epsilon=125 >>>>>>>>>>> >>>>>>>>>>> Using configuration file /home/weewx/weewx.conf. >>>>>>>>>>> >>>>>>>>>>> Using database binding 'wx_binding', which is bound to database >>>>>>>>>>> 'archive_mysql' >>>>>>>>>>> >>>>>>>>>>> Weather Underground Station: xxxxxx >>>>>>>>>>> >>>>>>>>>>> Date to check: 2019-05-30 >>>>>>>>>>> >>>>>>>>>>> Number of archive records: 128 >>>>>>>>>>> >>>>>>>>>>> Number of WU records: 127 >>>>>>>>>>> >>>>>>>>>>> Number of missing records: 1 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> This means that WU is now actually providing the records for the >>>>>>>>>>> date requested, rather than the day after the requested date. >>>>>>>>>>> So it seems that what is happening is that WU is determining >>>>>>>>>>> whether or not the current date at the station location is the same >>>>>>>>>>> as the UTC date. >>>>>>>>>>> If it is, it returns the data for the date as in the request. But >>>>>>>>>>> if the local date is different, it makes an (unwanted) adjustment >>>>>>>>>>> for the date difference. >>>>>>>>>>> >>>>>>>>>>> Rod >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On Thursday, May 30, 2019 at 9:29:16 AM UTC+10, Leon Shaner wrote: >>>>>>>>>>>> Hi, Rod, >>>>>>>>>>>> >>>>>>>>>>>> Yes and thanks for adding yet another confirmation of the issue. >>>>>>>>>>>> =D >>>>>>>>>>>> >>>>>>>>>>>> I can show that if I do the query within X hours of my offset of >>>>>>>>>>>> UTC, what actually happens is they report 288 records from the day >>>>>>>>>>>> PRIOR to the one I am asking about. >>>>>>>>>>>> For example, I ask for 20190528 and they give me records for >>>>>>>>>>>> 20190527, so *that* is why wunderfixer "thinks" it needs to >>>>>>>>>>>> re-upload everything. >>>>>>>>>>>> >>>>>>>>>>>> I am in contact with IBM about it and have shown them irrefutable >>>>>>>>>>>> proof of the issue. >>>>>>>>>>>> They didn't respond back yet, which I expect is because the proof >>>>>>>>>>>> was irrefutable. Ha! ;-) >>>>>>>>>>>> >>>>>>>>>>>> I expect that they're investigating and would rather respond from >>>>>>>>>>>> a position of understanding, or with any luck maybe even a quick >>>>>>>>>>>> fix. =D >>>>>>>>>>>> >>>>>>>>>>>> I meant to follow-up with IBM again this morning, but got waylaid, >>>>>>>>>>>> so I'll do that now. >>>>>>>>>>>> Thanks again, and for the reminder. =D >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> \Leon >>>>>>>>>>>> -- >>>>>>>>>>>> Leon Shaner :: Dearborn, Michigan (iPad Pro) >>>>>>>>>>>> >>>>>>>>>>>> On May 29, 2019, at 6:14 PM, Rod Yager <[email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> There is definitely a time zone issue. I am in the Sydney >>>>>>>>>>>>> Australia timezone (UTC +10 hours). >>>>>>>>>>>>> >>>>>>>>>>>>> It is currently 8am local time on May 30, 2019. (10pm May 29, >>>>>>>>>>>>> 2019 UTC) >>>>>>>>>>>>> >>>>>>>>>>>>> If I execute >>>>>>>>>>>>> >>>>>>>>>>>>> ./wunderfixer --verbose --date=2019-05-29 --epsilon=125 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I get >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Using configuration file /home/weewx/weewx.conf. >>>>>>>>>>>>> >>>>>>>>>>>>> Using database binding 'wx_binding', which is bound to database >>>>>>>>>>>>> 'archive_mysql' >>>>>>>>>>>>> >>>>>>>>>>>>> Weather Underground Station: xxxxxxx >>>>>>>>>>>>> >>>>>>>>>>>>> Date to check: 2019-05-29 >>>>>>>>>>>>> >>>>>>>>>>>>> Number of archive records: 288 >>>>>>>>>>>>> >>>>>>>>>>>>> Number of WU records: 97 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Number of missing records: 288 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Now WU actually has 288 records for 2019-05-29. >>>>>>>>>>>>> >>>>>>>>>>>>> But it only has 97 records for 2019-05-30. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> So it is clear that wunderfixer is downloading the record data >>>>>>>>>>>>> for 2019-05-30 from WeatherUnderground and trying to match them >>>>>>>>>>>>> with the local records for 2019-30-29. >>>>>>>>>>>>> >>>>>>>>>>>>> Of course, they all mismatch, and so wunderfixer concludes that >>>>>>>>>>>>> it must upload all the data for 2019-05-29. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Hope this narrows down the search for a solution. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Rod >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> On Monday, May 27, 2019 at 9:35:25 PM UTC+10, Leon Shaner wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> On May 27, 2019, at 12:12 AM, gjr80 <[email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Monday, 27 May 2019 13:16:53 UTC+10, Leon Shaner wrote: >>>>>>>>>>>>>>>> [snip] >>>>>>>>>>>>>>>> If you can see any shorter paths to a more reliable outcome >>>>>>>>>>>>>>>> than I have achieved so far, then you know know know I will be >>>>>>>>>>>>>>>> very grateful. =D >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I am not sure what local/UTC issue you refer to. When I do a >>>>>>>>>>>>>>> api.weather.com/v2/pws/history query on a station to the east >>>>>>>>>>>>>>> of Greenwich I am returned all records for the date specified >>>>>>>>>>>>>>> (eg 20190525 gives me all records for 25 May 2019), each record >>>>>>>>>>>>>>> contains an epoch timestamp which is correct and consistent >>>>>>>>>>>>>>> with 25 May 2019. Everything is as I would expect. However, >>>>>>>>>>>>>>> when I perform the same query on a station to the west of >>>>>>>>>>>>>>> Greenwich I am returned records for the day before the date >>>>>>>>>>>>>>> specified (ie 20190525 gives me all records for 24 May 2019 not >>>>>>>>>>>>>>> 25 May 2019), again each record contains an epoch timestamp but >>>>>>>>>>>>>>> the timestamp is for the previous day Ie 24 May 2019. I have >>>>>>>>>>>>>>> checked a number of data records in the stations history table >>>>>>>>>>>>>>> and WU is definitely returning the midnight to midnight data >>>>>>>>>>>>>>> for the day before. I have confirmed this behaviour with a >>>>>>>>>>>>>>> number of stations both east and west of Greenwich. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I don't think there is a local/UTC time issue, I think WU is >>>>>>>>>>>>>>> having some implementation issues and for stations west of >>>>>>>>>>>>>>> Greenwich they are returning the wrong day of data. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Gary! This was all very helpful. >>>>>>>>>>>>>> In addition to what you've described across the east vs west of >>>>>>>>>>>>>> GMT, I get similar behavior if I am within X hours of my local >>>>>>>>>>>>>> UTC offset when querying my own station. >>>>>>>>>>>>>> Last night as soon as localtime rolled over midnight, the >>>>>>>>>>>>>> queries for the previous day were correct. >>>>>>>>>>>>>> >>>>>>>>>>>>>> --Leon >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Gary >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>> Google Groups "weewx-user" group. >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>>>>> send an email to [email protected]. >>>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>>> https://groups.google.com/d/msgid/weewx-user/683a28af-35e4-474e-95a0-f684b9926af0%40googlegroups.com. >>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>>>> Groups "weewx-user" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>>> send an email to [email protected]. >>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>> https://groups.google.com/d/msgid/weewx-user/3570f548-caf3-43e8-858d-1b4d8c87d84a%40googlegroups.com. >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>>> Groups "weewx-user" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to [email protected]. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/weewx-user/388426A6-12D0-45FC-AFF2-E6EB532B61DC%40isylum.org. >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "weewx-user" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/weewx-user/184A035F-2BFD-4C43-9584-B84EB083F11F%40isylum.org. >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "weewx-user" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>>>> an email to [email protected]. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/weewx-user/D30A437D-57EF-46B0-A40F-98A7C1D2D793%40yager.net.au. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "weewx-user" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>>> an email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/weewx-user/CB045A39-A0D2-415D-B8C2-B741AF0D263B%40isylum.org. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "weewx-user" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>> an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/weewx-user/BD67E5D8-2F65-477A-B7DB-AD0C5B984148%40yager.net.au. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "weewx-user" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/weewx-user/AB1F8DAE-A13C-4E1A-BA85-EFE4C6E19FF4%40isylum.org. >>>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "weewx-user" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/weewx-user/8DC2EED8-5147-4D0F-9FDB-A9964E59DCDC%40isylum.org. >>>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "weewx-user" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/weewx-user/F2BA4631-596B-413C-A1FF-59918A37A339%40yager.net.au. >>> For more options, visit https://groups.google.com/d/optout. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "weewx-user" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/weewx-user/9825A07F-F1AD-44EF-9C90-0987BD82751F%40isylum.org. >> For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-user/D6C1F955-E49C-4AE3-9D91-EFC319B23E56%40yager.net.au. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/8162E364-9453-4503-BDCA-FDCB384A72B7%40isylum.org. For more options, visit https://groups.google.com/d/optout.
