Rod, Thanks for testing. Well, WU's approach is flawed, not mine, really. :S
It means that both API's, the one I just added for 'today" and the one from before that was for 'historical' (including today), both have the same bug. For now, the best workaround I can think of is to only ever run the new wunderfixer only ever for the previous day, only ever after UTC has rolled over midnight. :-( E.g. when it is 2019-06-02 in UTC, you can get the right results with wunderfixer if you ask for 2019-06-01 only. In other words, always delay your wunderfixer fix ups by a day, operating always on yesterday's date (relative to UTC) after UTC has rolled over to the next day. Scroll to see the output of me running wunderdates on my local machine in UTC-4 against your station in UTC+10. Because I am asking for 2019-06-01, which is also my local date, the logic will use the URL that queries for 'today' and instead of returning obsTimeLocal records, it is returning obsTimeUtc records for the requested date. We can see that it's returning 2019-06-02 relative to your station, even though I asked for 2019-06-01. Sure, I could go a bit further and use a related API query to check the timezone of your station and use date conversions to check if the date requested is 'today' relative to the station requested, but really this is intended to be run on the same machine as weewx, so the localtime date should match that of the station. It means there isn't any real reason for me to workaround their bug. In fact even if I did, then the logic would just fall through to the historical data URL, which we already know is buggy. For now, I'll just report this to IBM a bug in the 'today' API, similar to the bug in the 'historical' API. $ ./wunderdates --epsilon=125 --date 2019-06-01 --station ISYDNEY155 --test --verbose | more Using configuration file /usr/share/weewx4/weewx.conf. Weather Underground Station: ISYDNEY155 Date to check: 2019-06-01 epoch: 1559397600 date_epoch_local: 2019-06-01 10:00:00 utcepoch: 1559398000 date_epoch_utc: 2019-06-01 14:00:00 tz: Australia/Sydney obsTimeUtc: 2019-06-01T14:00:00Z obsTimeLocal: 2019-06-02 00:00:00 epoch: 1559397900 date_epoch_local: 2019-06-01 10:05:00 utcepoch: 1559398300 date_epoch_utc: 2019-06-01 14:05:00 tz: Australia/Sydney obsTimeUtc: 2019-06-01T14:05:00Z obsTimeLocal: 2019-06-02 00:05:00 ... Regards, \Leon -- Leon Shaner :: Dearborn, Michigan (iPad Pro) > On Jun 1, 2019, at 6:03 PM, Rod Yager <r...@yager.net.au> wrote: > > Dear Leon, > > Your approach is flawed. It won’t work for historical data. > > If you run > > wunderdates —date=2019-01-01 at 9pm your time you will receive the data > for 2018-12-31 (because your station is a day behind UTC at the time of the > request, so it gives you the data for the previous day). > > On the other hand, if you run > > wunderdates —date=2019-01-01 at 9am your time you will receive the data > for 2019-01-01 (because your station is in the same day as UTC at the time > of the request). > > If I do the same thing (I’m in UTC+10) > > wunderdates —date=2019-01-01 at 9pm my time produces the data for > 2019-01-01 (because my station is in the same day as UTC at the time of the > request) but > wunderdates —date=2019-01-01 at 9am my time produces the data for > 2019-01-02 (because my station is a day ahead of UTC at the time of the > request) > > Rod > >> On 2 Jun 2019, at 3:17 am, Leon Shaner <l...@isylum.org> wrote: >> >> All, >> >> I'm testing a new approach. Below you will find links to an updated >> wunderdates utility that can be used to validate whether I am on the right >> track. >> The wunderdates utility simply dumps out timestamp related records from what >> WU returns from the query. >> >> We're looking mainly at the obsTimeUtc and obsTimeLocal values, which were >> demonstrated under the prior approach to be returning the wrong dates when >> within the stations localtime +/- offset from UTC. >> >> The new approach is to detect if the requested date is 'today' and if so, >> use a different API URL that already assumes 'today' and will hopefully not >> be subject to the UTC offset bugs we've been chasing with the historical >> data API URL. >> >> I have my crontab set to do another test approaching my UTC offset, just >> after coming within the offset, and then again just before and after >> midnight localtime. >> (Same test I did before, but now with the new approach in place). >> >> Here is the utility for anyone else that wants to check out the behavior: >> >> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates3 >> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates4 >> >> Which version you pull will depend on which base weewx you are running. >> Pull the one that matches your weewx version and place it in bin, next to >> wunderfixer, and it will take the same arguments as wunderfixer. >> >> You can just try wunderfixer as you normally would (with --apikey) and then >> run wunderdates(3 or 4) with exact same arguments to be able to see what >> actual timestamps WU is sending back for the date queried. >> Parameters like --epsilon don't have any effect in the case of wunderdates, >> but I left it there so you don't have to change options when running the >> util to get the debug output. >> >> Regards, >> \Leon >> -- >> Leon Shaner :: Dearborn, Michigan (iPad Pro) >> >> On May 30, 2019, at 12:40 AM, Leon Shaner <l...@isylum.org> wrote: >> >>> Rod, >>> Thanks again for this. >>> >>> Since the in-progress version of wunderfixer doesn't really show you the >>> dates that come back from WU, I have written a tool just to debug the dates. >>> >>> The command-line input and basing of defaults on weewx.conf works the same >>> as wunderfixer, except it doesn't look at your DB at all. It only prints >>> out datestamps in various incarnations coming back from WU and as compared >>> to your system's localtime. >>> >>> It would be helpful to see the "wunderdates" output at times like you've >>> shown below, a la before and after your localtime rolls around to UTC >>> midnight. >>> >>> Since you are at UTC + 10, another interesting time would be 11+ hours on >>> either side of UTC midnight, in addition to within 9 or less hours. This >>> is just to make sure we're covering all the corner cases. >>> >>> Gary reported a difference for stations that are east vs. west of GMT, and >>> I expect we're really chasing the same bug in that there is some bad math >>> WU is doing based on UTC offset, but since an offset can be +/-, the >>> effects go in opposite directions date-wise, depending on which side of the >>> UTC dateline your station is located. >>> >>> At least that is what I surmise may be happening, but the wunderdates >>> utility should shed light one way or the other. =D >>> >>> The wunderdates utility is available at the links below. >>> >>> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates3 >>> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates4 >>> >>> Which version you pull will depend on which base weewx you are running. >>> Pull the one that matches your weewx version and place it in bin, next to >>> wunderfixer, and it will take the same arguments as wunderfixer. >>> >>> You can just try wunderfixer as you normally would (with --apikey) and then >>> run wunderdates(3 or 4) with exact same arguments to be able to see what >>> actual timestamps WU is sending back for the date queried. >>> Parameters like --epsilon don't have any effect in the case of wunderdates, >>> but I left it there so you don't have to change options when running the >>> util to get the debug output. >>> >>> What I've been doing is saving the output to files for use with sdiff >>> (side-by-side) diff. >>> Or you can just compare the head and tail of each file individually. >>> >>> Example for my system before and after my local time on 5-28 once it was >>> at/after 8 p.m. here, which is within 4 hours of UTC midnight (I am at >>> UTC-4). >>> >>> This output is optimized for screen widths 203 or wider. Sorry. :S >>> Mainly the last two data fields in the output tell what we need to know. >>> >>> >>> pi@nixie:/var/tmp $ head -n 3 59:19:28:5_wu.txt 0:20:28:5_wu.txt >>> ==> 59:19:28:5_wu.txt <== >>> Using configuration file /usr/share/weewx4/weewx.conf. >>> epoch: 1559016299 date_epoch_local: 2019-05-28 00:04:59 utcepoch: >>> 1559016699 date_epoch_utc: 2019-05-28 04:04:59 tz: America/New_York >>> obsTimeUtc: 2019-05-28T04:04:59Z obsTimeLocal: 2019-05-28 00:04:59 >>> epoch: 1559016599 date_epoch_local: 2019-05-28 00:09:59 utcepoch: >>> 1559016999 date_epoch_utc: 2019-05-28 04:09:59 tz: America/New_York >>> obsTimeUtc: 2019-05-28T04:09:59Z obsTimeLocal: 2019-05-28 00:09:59 >>> >>> ==> 0:20:28:5_wu.txt <== >>> Using configuration file /usr/share/weewx4/weewx.conf. >>> epoch: 1558929899 date_epoch_local: 2019-05-27 00:04:59 utcepoch: >>> 1558930299 date_epoch_utc: 2019-05-27 04:04:59 tz: America/New_York >>> obsTimeUtc: 2019-05-27T04:04:59Z obsTimeLocal: 2019-05-27 00:04:59 >>> epoch: 1558930199 date_epoch_local: 2019-05-27 00:09:59 utcepoch: >>> 1558930599 date_epoch_utc: 2019-05-27 04:09:59 tz: America/New_York >>> obsTimeUtc: 2019-05-27T04:09:59Z obsTimeLocal: 2019-05-27 00:09:59 >>> >>> >>> pi@nixie:/var/tmp $ tail -n 3 59:19:28:5_wu.txt 0:20:28:5_wu.txt >>> ==> 59:19:28:5_wu.txt <== >>> epoch: 1559087699 date_epoch_local: 2019-05-28 19:54:59 utcepoch: >>> 1559088099 date_epoch_utc: 2019-05-28 23:54:59 tz: America/New_York >>> obsTimeUtc: 2019-05-28T23:54:59Z obsTimeLocal: 2019-05-28 19:54:59 >>> epoch: 1559087702 date_epoch_local: 2019-05-28 19:55:02 utcepoch: >>> 1559088102 date_epoch_utc: 2019-05-28 23:55:02 tz: America/New_York >>> obsTimeUtc: 2019-05-28T23:55:02Z obsTimeLocal: 2019-05-28 19:55:02 >>> Number of WU records: 240 >>> >>> ==> 0:20:28:5_wu.txt <== >>> epoch: 1559015699 date_epoch_local: 2019-05-27 23:54:59 utcepoch: >>> 1559016099 date_epoch_utc: 2019-05-28 03:54:59 tz: America/New_York >>> obsTimeUtc: 2019-05-28T03:54:59Z obsTimeLocal: 2019-05-27 23:54:59 >>> epoch: 1559015999 date_epoch_local: 2019-05-27 23:59:59 utcepoch: >>> 1559016399 date_epoch_utc: 2019-05-28 03:59:59 tz: America/New_York >>> obsTimeUtc: 2019-05-28T03:59:59Z obsTimeLocal: 2019-05-27 23:59:59 >>> Number of WU records: 288 >>> >>> >>> But after my system rolled over midnight localtime, results returned to the >>> correct dates when asking for 5-28: >>> >>> pi@nixie:/var/tmp $ head -3 0:0:29:5_wu.txt >>> Using configuration file /usr/share/weewx4/weewx.conf. >>> epoch: 1559016299 date_epoch_local: 2019-05-28 00:04:59 utcepoch: >>> 1559016699 date_epoch_utc: 2019-05-28 04:04:59 tz: America/New_York >>> obsTimeUtc: 2019-05-28T04:04:59Z obsTimeLocal: 2019-05-28 00:04:59 >>> epoch: 1559016599 date_epoch_local: 2019-05-28 00:09:59 utcepoch: >>> 1559016999 date_epoch_utc: 2019-05-28 04:09:59 tz: America/New_York >>> obsTimeUtc: 2019-05-28T04:09:59Z obsTimeLocal: 2019-05-28 00:09:59 >>> >>> pi@nixie:/var/tmp $ tail -3 0:0:29:5_wu.txt >>> epoch: 1559102099 date_epoch_local: 2019-05-28 23:54:59 utcepoch: >>> 1559102499 date_epoch_utc: 2019-05-29 03:54:59 tz: America/New_York >>> obsTimeUtc: 2019-05-29T03:54:59Z obsTimeLocal: 2019-05-28 23:54:59 >>> epoch: 1559102399 date_epoch_local: 2019-05-28 23:59:59 utcepoch: >>> 1559102799 date_epoch_utc: 2019-05-29 03:59:59 tz: America/New_York >>> obsTimeUtc: 2019-05-29T03:59:59Z obsTimeLocal: 2019-05-28 23:59:59 >>> Number of WU records: 288 >>> >>> >>> Regards, >>> \Leon >>> -- >>> Leon Shaner :: Dearborn, Michigan (iPad Pro) >>> >>> On May 29, 2019, at 8:53 PM, Rod Yager <r...@yager.net.au> wrote: >>> >>>> Further to this, it has now rolled past 10am here, so the local date is >>>> now the same as the UTC date. (ie. local time May 30 2019 10:40 AM = May >>>> 30 2019 00:40 AM UTC). >>>> >>>> Now I get: >>>> >>>> ./wunderfixer --verbose --date=2019-05-29 --epsilon=125 >>>> >>>> Using configuration file /home/weewx/weewx.conf. >>>> >>>> Using database binding 'wx_binding', which is bound to database >>>> 'archive_mysql' >>>> >>>> Weather Underground Station: xxxxx >>>> >>>> Date to check: 2019-05-29 >>>> >>>> Number of archive records: 288 >>>> >>>> Number of WU records: 288 >>>> >>>> Number of missing records: 0 >>>> >>>> [root@moses bin]# ./wunderfixer --verbose --date=2019-05-30 --epsilon=125 >>>> >>>> Using configuration file /home/weewx/weewx.conf. >>>> >>>> Using database binding 'wx_binding', which is bound to database >>>> 'archive_mysql' >>>> >>>> Weather Underground Station: xxxxxx >>>> >>>> Date to check: 2019-05-30 >>>> >>>> Number of archive records: 128 >>>> >>>> Number of WU records: 127 >>>> >>>> Number of missing records: 1 >>>> >>>> >>>> This means that WU is now actually providing the records for the date >>>> requested, rather than the day after the requested date. >>>> So it seems that what is happening is that WU is determining whether or >>>> not the current date at the station location is the same as the UTC date. >>>> If it is, it returns the data for the date as in the request. But if the >>>> local date is different, it makes an (unwanted) adjustment for the date >>>> difference. >>>> >>>> Rod >>>> >>>> >>>>> On Thursday, May 30, 2019 at 9:29:16 AM UTC+10, Leon Shaner wrote: >>>>> Hi, Rod, >>>>> >>>>> Yes and thanks for adding yet another confirmation of the issue. =D >>>>> >>>>> I can show that if I do the query within X hours of my offset of UTC, >>>>> what actually happens is they report 288 records from the day PRIOR to >>>>> the one I am asking about. >>>>> For example, I ask for 20190528 and they give me records for 20190527, so >>>>> *that* is why wunderfixer "thinks" it needs to re-upload everything. >>>>> >>>>> I am in contact with IBM about it and have shown them irrefutable proof >>>>> of the issue. >>>>> They didn't respond back yet, which I expect is because the proof was >>>>> irrefutable. Ha! ;-) >>>>> >>>>> I expect that they're investigating and would rather respond from a >>>>> position of understanding, or with any luck maybe even a quick fix. =D >>>>> >>>>> I meant to follow-up with IBM again this morning, but got waylaid, so >>>>> I'll do that now. >>>>> Thanks again, and for the reminder. =D >>>>> >>>>> Regards, >>>>> \Leon >>>>> -- >>>>> Leon Shaner :: Dearborn, Michigan (iPad Pro) >>>>> >>>>> On May 29, 2019, at 6:14 PM, Rod Yager <r...@yager.net.au> wrote: >>>>> >>>>>> There is definitely a time zone issue. I am in the Sydney Australia >>>>>> timezone (UTC +10 hours). >>>>>> >>>>>> It is currently 8am local time on May 30, 2019. (10pm May 29, 2019 UTC) >>>>>> >>>>>> If I execute >>>>>> >>>>>> ./wunderfixer --verbose --date=2019-05-29 --epsilon=125 >>>>>> >>>>>> >>>>>> >>>>>> I get >>>>>> >>>>>> >>>>>> >>>>>> Using configuration file /home/weewx/weewx.conf. >>>>>> >>>>>> Using database binding 'wx_binding', which is bound to database >>>>>> 'archive_mysql' >>>>>> >>>>>> Weather Underground Station: xxxxxxx >>>>>> >>>>>> Date to check: 2019-05-29 >>>>>> >>>>>> Number of archive records: 288 >>>>>> >>>>>> Number of WU records: 97 >>>>>> >>>>>> >>>>>> Number of missing records: 288 >>>>>> >>>>>> >>>>>> >>>>>> Now WU actually has 288 records for 2019-05-29. >>>>>> >>>>>> But it only has 97 records for 2019-05-30. >>>>>> >>>>>> >>>>>> >>>>>> So it is clear that wunderfixer is downloading the record data for >>>>>> 2019-05-30 from WeatherUnderground and trying to match them with the >>>>>> local records for 2019-30-29. >>>>>> >>>>>> Of course, they all mismatch, and so wunderfixer concludes that it must >>>>>> upload all the data for 2019-05-29. >>>>>> >>>>>> >>>>>> >>>>>> Hope this narrows down the search for a solution. >>>>>> >>>>>> >>>>>> >>>>>> Rod >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> On Monday, May 27, 2019 at 9:35:25 PM UTC+10, Leon Shaner wrote: >>>>>>> >>>>>>> On May 27, 2019, at 12:12 AM, gjr80 <gjrod...@gmail.com> wrote: >>>>>>> >>>>>>>>> On Monday, 27 May 2019 13:16:53 UTC+10, Leon Shaner wrote: >>>>>>>>> [snip] >>>>>>>>> If you can see any shorter paths to a more reliable outcome than I >>>>>>>>> have achieved so far, then you know know know I will be very >>>>>>>>> grateful. =D >>>>>>>> >>>>>>>> I am not sure what local/UTC issue you refer to. When I do a >>>>>>>> api.weather.com/v2/pws/history query on a station to the east of >>>>>>>> Greenwich I am returned all records for the date specified (eg >>>>>>>> 20190525 gives me all records for 25 May 2019), each record contains >>>>>>>> an epoch timestamp which is correct and consistent with 25 May 2019. >>>>>>>> Everything is as I would expect. However, when I perform the same >>>>>>>> query on a station to the west of Greenwich I am returned records for >>>>>>>> the day before the date specified (ie 20190525 gives me all records >>>>>>>> for 24 May 2019 not 25 May 2019), again each record contains an epoch >>>>>>>> timestamp but the timestamp is for the previous day Ie 24 May 2019. I >>>>>>>> have checked a number of data records in the stations history table >>>>>>>> and WU is definitely returning the midnight to midnight data for the >>>>>>>> day before. I have confirmed this behaviour with a number of stations >>>>>>>> both east and west of Greenwich. >>>>>>>> >>>>>>>> I don't think there is a local/UTC time issue, I think WU is having >>>>>>>> some implementation issues and for stations west of Greenwich they are >>>>>>>> returning the wrong day of data. >>>>>>> >>>>>>> Thanks, Gary! This was all very helpful. >>>>>>> In addition to what you've described across the east vs west of GMT, I >>>>>>> get similar behavior if I am within X hours of my local UTC offset when >>>>>>> querying my own station. >>>>>>> Last night as soon as localtime rolled over midnight, the queries for >>>>>>> the previous day were correct. >>>>>>> >>>>>>> --Leon >>>>>>> >>>>>>>> >>>>>>>> Gary >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "weewx-user" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>> an email to weewx...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/weewx-user/683a28af-35e4-474e-95a0-f684b9926af0%40googlegroups.com. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "weewx-user" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to weewx-user+unsubscr...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/weewx-user/3570f548-caf3-43e8-858d-1b4d8c87d84a%40googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "weewx-user" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to weewx-user+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/weewx-user/388426A6-12D0-45FC-AFF2-E6EB532B61DC%40isylum.org. >>> For more options, visit https://groups.google.com/d/optout. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "weewx-user" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to weewx-user+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/weewx-user/184A035F-2BFD-4C43-9584-B84EB083F11F%40isylum.org. >> For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to weewx-user+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-user/D30A437D-57EF-46B0-A40F-98A7C1D2D793%40yager.net.au. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CB045A39-A0D2-415D-B8C2-B741AF0D263B%40isylum.org. For more options, visit https://groups.google.com/d/optout.
