Jarom, CURL is pretty sophisticated in its ability to emulate browser state in pretty much any way but JavaScript. When it worked this morning, I saw some cookies were involved. It may well be that the python way isn't handling that part. I don't know enough about how python fetches pages to work that out, but I am very familiar with CURL, so if I can find a path that works consistently, then I'll go back to the python to see about how to implement same.
I was getting 404's in the browser even, when I looked at it earlier. I'll keep working on it, but not too hard, so as to not get on their radar in any unwanted sort of way. ;-) Regards, \Leon -- Leon Shaner :: Dearborn, Michigan (iPad Pro) > On May 22, 2019, at 2:04 PM, Jarom Hatch <[email protected]> wrote: > > Interesting, using curl sometimes I can it fine, but wunderfixer is always > getting a 403 Forbidden, as if it is actively being blocked... When it > doesn't work in curl I get `HTTP/1.1 404 Not Found` and when it does work I > get `HTTP/1.1 200 OK`. Curl never gets a 403 error. > >> On Wednesday, May 22, 2019 at 11:48:08 AM UTC-6, Jarom Hatch wrote: >> I was able to get it to work twice in my web browser, but as you said, it is >> sporadic. I don't ever recall them using Akamai before so that may very >> well be a contributing factor. >> >> I wonder if we can find out the origin address and see what happens if we >> can bypass Akamai... >> >>> On Wednesday, May 22, 2019 at 7:35:18 AM UTC-6, Leon Shaner wrote: >>> For one thing, the URL of this form: >>> >>> http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=SOMESTATION&month=5&day=22&year=2019&format=1 >>> >>> Is now redirecting to one using HTTPS: >>> >>> https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=SOMESTATION&month=5&day=22&year=2019&format=1 >>> >>> Also, the redirect itself takes an excruciatingly long time. >>> So I just changed the URL to https directly... >>> >>> The first time I tried any of the above using CURL this morning it worked, >>> but then after that I started getting: >>> >>> An error occurred while processing your request. >>> Reference #30.6f451160.1558531514.16ced4f6 >>> >>> It looks as if they've put some kind of Akamai proxy in the middle, which >>> is fine for static content, but not so fine for a query of this nature. >>> Strange that it worked for me the very first time. It's almost as if the >>> Akamai "farm" has lost some "state" information and not all nodes have the >>> same content, so if you get stuck going through a bad node you get a bogus >>> response. >>> >>> Attached is a transcript of a failed attempt. I put SOMESTATION there only >>> after the fact. The actual query was for my actual station, which used to >>> work. >>> > > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-user/07ac6f86-ae4d-4854-8398-ce4ab8d846c1%40googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/3260F8E7-37D4-4737-999B-97522E324370%40isylum.org. For more options, visit https://groups.google.com/d/optout.
