Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Following an extended local internet outage yesterday, I found myself with several hours' worth of data missing from WU. I recalled wunderfix, tried it, encountered the 403 problem, and eventually discovered this thread. Having read most of it, I tried looking for the modified 3.9.1 version on

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

I didn't touch how wunderfixer posts (uploads) the fixes, only how it downloads data. -tk On Thursday, July 18, 2019 at 12:21:58 PM UTC-7, Leon Shaner wrote: > > Tom, > > There's nothing wrong with the old methods of reposting old records to WU. > > IBM is pretty clueless about the old

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Tom, There's nothing wrong with the old methods of reposting old records to WU. IBM is pretty clueless about the old interfaces we use from the pre-IBM WU days. When IBM says they don't yet support re-uploading old records via API, they are most surely talking about the new API's. I can

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

I've ported wunderfixer to the new WU API. Commit 32c35ce on the development branch. Unfortunately, it looks like the WU no longer allows re-posting old records, so the utility may no longer be useful. I'd be

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

I’ve already migrated the today logic into my local version of wunderfixer, which runs as a cron job at 05:58, 11:58, 17:58 and 23:58. This works as expected here. The only circumstance I can see in which it would break is if I make a request for today’s data at 23:59:59.95 on my machine. The

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Thanks, Rod! =D Those are precisely the same tests I ran and exact same results that noted, before before publishing. =D Plus a ton of checks against my own station, KMIDEARB5, west of UTC. As before, even more important than these 'historical' API tests is to query against the current day

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Dear Leon, Thanks for all your work on this. This works as well as we are going to be able to manage unless and until WU fixes the bug. The remaining issue affects just one date, the “change-over” date when WU transitions to returning the correct data - and will only bite for stations east

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

All, There are updates to the wunderdates utilities, to implement the following: Add +/- 1-day compensation logic to workaround WU 'historical' API bug, depending on whether station is east/west of UTC, and whether it is within X hours of UTC midnight, relative to the local station time. These

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Rod, I've spent quite a bit of time working on a "cleaner" workaround for the WU bugs. But just when I thought I had landed on the "reasonable" approach, I hit a snag. As a reminder, with the latest approach, all is fine for stations west of the UTC date line. What I am working on now is a

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Dear Leon, If WU doesn’t fix their bug, the solution is really quite simple. We need to make 3 requests to WU for the data: one request for the date we are interested in, one request for the day before and one request for the day after. eg. If we are intending to upload missing data for

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Rod, Thanks for testing. Well, WU's approach is flawed, not mine, really. :S It means that both API's, the one I just added for 'today" and the one from before that was for 'historical' (including today), both have the same bug. For now, the best workaround I can think of is to only ever run

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Dear Leon, Your approach is flawed. It won’t work for historical data. If you run wunderdates —date=2019-01-01 at 9pm your time you will receive the data for 2018-12-31 (because your station is a day behind UTC at the time of the request, so it gives you the data for the previous day).

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

All, I'm testing a new approach. Below you will find links to an updated wunderdates utility that can be used to validate whether I am on the right track. The wunderdates utility simply dumps out timestamp related records from what WU returns from the query. We're looking mainly at the

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Andrew, We can't use this as a workaround, but what you're suggesting is essentially what WU should be doing with the date that we provide in the query URL. WU should be doing the conversion to station localtime and returning the values for that date, relative to the station. But they seem to

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

No. You always get all available data for a complete day (00:00:00 to 23:59:59 local time). So in UTC+10, the first entry in the returned data is 00:00:00 UTC+10 (14:00:00 UTC the previous day) and the last entry in the returned data is 23:59:59 UTC+10 (13:59:59 UTC same day) and in

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

. so does this just simplify to the fact that in the api the date is taken as being the UTC date? ie if one requests data for 20190529, and timezone is +3 you get back data for UTC00:00 29 may to 23:59 29 may which is 03:00 29 may local to 23:59 29 may local AND 00:00 30 may local - 02:59

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Have downloaded the wunderdates. I am in the UTC+10 timezone. So far, my experience is that when the current time is between 00:00+10 and 09:59+10 (14:00UTC to 23:59 UTC) (so the local date is one day in advance of the UTC date), the API fetches data for the local-time day immediately

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Further to this, it has now rolled past 10am here, so the local date is now the same as the UTC date. (ie. local time May 30 2019 10:40 AM = May 30 2019 00:40 AM UTC). Now I get: ./wunderfixer --verbose --date=2019-05-29 --epsilon=125 Using configuration file /home/weewx/weewx.conf. Using

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Hi, Rod, Yes and thanks for adding yet another confirmation of the issue. =D I can show that if I do the query within X hours of my offset of UTC, what actually happens is they report 288 records from the day PRIOR to the one I am asking about. For example, I ask for 20190528 and they give me

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

There is definitely a time zone issue. I am in the Sydney Australia timezone (UTC +10 hours). It is currently 8am local time on May 30, 2019. (10pm May 29, 2019 UTC) If I execute ./wunderfixer --verbose --date=2019-05-29 --epsilon=125 I get Using configuration file

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

> On May 27, 2019, at 12:12 AM, gjr80 wrote: > >> On Monday, 27 May 2019 13:16:53 UTC+10, Leon Shaner wrote: >> [snip] >> If you can see any shorter paths to a more reliable outcome than I have >> achieved so far, then you know know know I will be very grateful. =D > > I am not sure what

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Gary, It's late, so I'll respond to the rest later, but... The problem here is that if we compare our local every 1-minute records to WU's query that only shows every 5-minute records, then we'll keep re-uploading the the "missing" 1-minute records every time wunderfixer is called. This

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

On Monday, 27 May 2019 13:16:53 UTC+10, Leon Shaner wrote: > > Gary, > > In practice, WU seems to discard data that is not close to their > APPARENTly preferred 5-minute "normalization buckets." > > I upload via rapidfire *and* regular loop on 1-minute intervals, and > irregardless of same, the

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Gary, In practice, WU seems to discard data that is not close to their APPARENTly preferred 5-minute "normalization buckets." I upload via rapidfire *and* regular loop on 1-minute intervals, and irregardless of same, the queries only ever show the records most closely aligned to their

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

On Saturday, 25 May 2019 22:54:25 UTC+10, Leon Shaner wrote: > > > There is no point re-uploading 00:00:00, then 00:01:00, then 00:02:00, > because WU is only going to keep 00:00:00, and then later it will keep > whatever is closest to 00:05:00. That's been the behavior of wunderfixer > vs.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Tested on 4.x (python 2 and 3) and 3.9.x. (python 2). Ultimately went with apikey instead of apiKey in weewx.conf, which was to be more consistent with other parameter names. If you are willing to pass "--apikey=0123456789012345678901" on wunderfixer command-line, then you only need

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

So this is where I am... These all seem reasonable (but then scroll for the problem which starts with yesterday's date): pi@nixie:/usr/share/weewx4/bin $ ./wunderfixer --epsilon=125 --date=2019-05-21 --test --verbose | more Using configuration file /usr/share/weewx4/weewx.conf. Using database

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Hey, Gary, I figured this out last night with a cue from Tom. But it requires modifying two different classes in restx.py, similar to the way you mentioned and I am a little uncomfortable with it, since I don't know all the different touch points / ramifications for other code. Also,

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

No, all get_site_dict does is check that the listed config options exist and are not the install defaults. The real issue is that _ambient_dict is used to pass keyword arguments to class AmbientThread init, apiKey is not a parameter in the init signature so hence the error. One approach will be

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Hey, Gary, Below is the error on startup, simply because I added the "apiKey = blah" right after the "password = oog" in the [[Wunderground]] section. This led me to believe that I need to somehow "extend" the ambient_dict, but for the life of me I am not seeing how to do that. I looked every

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

What does the error trace show? WeeWX shouldn't be concerned about extra config options. We don't need to re-invent wheels, forecast extension (among others) already uses a WU API key, not appropriate to use that config option here but using a consistent config option naming convention makes it

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Well shoot. Found an issue that surfaced only upon weewx restart. Don't add the apiKey to weewx.conf. Found out engine.py is picky about properties it isn't expecting to find in the weewx.conf file. Thought it would just ignore them. So I guess I need to extend the dictionary of expected

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Hi, WeeWX'ers! =D For those on WeeWX 4.0 / development, ONLY... I am done with changing wunderfixer to use the new wunderground API KEY way of doing the WU query to see what records they have for comparison to those in the archive DB. Yay! =D Please see the comments in the pull request

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

FYI, all, I am currently hot on the trail to converting the wunderfixer query to use the new API instead. The new API is going to be much easier than the old query, because the output is neatly formatted as JSON, and crucially for each record they even include the "epoch" i.e. Unix timestamp,

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

I am in contact with IBM. This whole intersection is entirely for THEIR benefit. I am persistent SOB. It'll be fixed. LOL Regards, \Leon -- Leon Shaner :: Dearborn, Michigan (iPad Pro) > On May 23, 2019, at 9:22 PM, Jarom Hatch wrote: > > Even after that none of the backfilled data made it

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Even after that none of the backfilled data made it in. If they are no longer accepting backfilled data then fixing wunderfixer might not be possible. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Yesterday when I was able to download the csv I saved a copy of it. I put it up on a site hosted by me and changed the URL in wunderfixer to point there instead. Now it parses it and starts the backfill. Still isn't showing up on their side though. I'm going to run it about 10 times over

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

I haven't seen a 403 since you changed the user agent. On Thursday, May 23, 2019 at 10:10:28 AM UTC-6, Leon Shaner wrote: > > Jarom, > > So you got 503's with my updated code, but not 403's? > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

I haven't seen a 403 since you changed the user agent. On Thursday, May 23, 2019 at 10:10:28 AM UTC-6, Leon Shaner wrote: > > Jarom, > > So you got 503's with my updated code, but not 403's? > > Yeah, I have found that it is normal for WU to lazily process re-uploaded > records, and it seems as

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Jarom, So you got 503's with my updated code, but not 403's? Yeah, I have found that it is normal for WU to lazily process re-uploaded records, and it seems as if they drop them...as if some backend process is dying. I found through trial and error that after a weewx "outage" I need to

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Those 503's are coming from Akamai. When I have used Akamai in the past those errors are because the origin is not responding correctly. Related sorta, when I took out the code to pull the timestamps and just force-ran all my records as a big backfill I got success messages back however none

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Sad, now I'm getting the 503. I'll keep trying. On Thursday, May 23, 2019 at 8:27:37 AM UTC-6, Leon Shaner wrote: > > Jarom, > > Thanks so much! I see what I did wrong and I was able to make a stubbed > down version for basic testing to prove it's at least trying to connect. > > Same location

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Jarom, Thanks so much! I see what I did wrong and I was able to make a stubbed down version for basic testing to prove it's at least trying to connect. Same location (for WeeWX 3.9.1): https://raw.githubusercontent.com/UberEclectic/weewx/master/bin/wunderfixer The thing is, I'm still

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

I tried the 3.9.1 version and I get Could not get Weather Underground data. Exiting. Curl still works, even for yesterday's data. Tracing the script it doesn't appear to be actually attempting the download. On Wednesday, May 22, 2019 at 7:03:49 PM UTC-6, Leon Shaner wrote: > > Say, we need

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Hey WeeWX'ers!!! =D I have a fix in the hopper. There's nothing that can be done for the occasional HTTP 404, or even 503's I am now seeing, but the HTTP 403 was due to a change on WU's part where they are rejecting certain HTTP User-Agent strings. The fact that they are putting Akamai in

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Hi Leon, Agreed, I realize my post is a bit off topic but if WU continues to be less than reliable then perhaps other solutions might be viable? Thanks for the reply, perhaps a new thread needs to be started by someone smarter than me about historical data search. Good luck with your research

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Hey, Doug, Creating a front end to search your own local data would certainly be doable. For sure, directing end-users to our own sites and letting them search our historical data would be useful. But that wouldn't fix the problem at hand. :-/ The wunderfixer utility is there to query what data

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

I'm still working on this.CURL is telling me they are not only using https, but also TLSv1.2.Here is a transcript, in case one of y'all beats me to the fix.  =D -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Jarom, CURL is pretty sophisticated in its ability to emulate browser state in pretty much any way but JavaScript. When it worked this morning, I saw some cookies were involved. It may well be that the python way isn't handling that part. I don't know enough about how python fetches pages to

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Interesting, using curl sometimes I can it fine, but wunderfixer is always getting a 403 Forbidden, as if it is actively being blocked... When it doesn't work in curl I get `HTTP/1.1 404 Not Found` and when it does work I get `HTTP/1.1 200 OK`. Curl never gets a 403 error. On Wednesday, May

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

I was able to get it to work twice in my web browser, but as you said, it is sporadic. I don't ever recall them using Akamai before so that may very well be a contributing factor. I wonder if we can find out the origin address and see what happens if we can bypass Akamai... On Wednesday, May

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

For one thing, the URL of this form:http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=SOMESTATION=5=22=2019=1Is now redirecting to one using HTTPS:https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=SOMESTATION=5=22=2019=1Also, the redirect itself takes an

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Hmm, I notice that WXDailyHistory is working as usual now (well for me anyway) but wunderfixer still gives a 403 error. Gary On Wednesday, 22 May 2019 13:02:24 UTC+10, gjr80 wrote: > > Wunderfixer (as it stands now) relies on WUs WXDailyHistory.asp page. If > WXDailyHistory doesn't work then

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Wunderfixer (as it stands now) relies on WUs WXDailyHistory.asp page. If WXDailyHistory doesn't work then wunderfixer will not work. Period. I guess depending on where you sit on the conspiracy theory scale will determine whether you believe the current issues with WXDailyHistory are short

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

FWIW, I had the odd 404 the other day, but then hours later it worked. They may be just moving infrastructure around and not getting all the load balancer rules correct on the first go. Hope that's all it is... Could it be that we need to request an API key and use that as a password, instead?

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

Just tried posting a question to WU using their email form. Got this error: Validation failed: weather_underground is invalid. Looks like they're a mess right now. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

It looks like the WU completely changed their API without warning. For example, to access my station, it used to be https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KORHOODR3 but now it's https://www.wunderground.com/dashboard/pws/KORHOODR3 You used to be able to download a