thanks for the replies! i should have emphasized that this is for www.instagram.com, not the API. API requests are working fine.
you're right that IP blocking wouldn't usually be the first culprit in general, especially for 429s. i tried from a few different apps, though, including shell-hrd (log in my first post), which pretty much never uses urlfetch otherwise based on its quota numbers, so i doubt it's User-Agent blocking. i tried an entirely new www.instagram.com URL and still got a 429, so it's probably not specific URLs, at least due to my own traffic. and i can fetch the same URL fine from my local machine. hence my IP suspicion. i've already worked around this, so it's not urgent. just figured you all might want to know. thanks again! On Monday, May 2, 2016 at 10:52:11 AM UTC-7, Nick (Cloud Platform Support) wrote: > > Hey Ryan, > > I'm unsure that this indicates that App Engine specifically is being > rate-limited. It's likely that the 429 response is directly related to the > frequency with which you're making requests, regardless of the origin of > those requests. While not impossible, I suppose, it would be surprising if > they were keeping track of App Engine IP ranges and applying a different > rate-limit, and would require some thorough A/B testing to prove. So, I > recommend just checking their documentation > <https://www.instagram.com/developer/limits/> or, if the rate-limit is > undocumented, benchmarking to attempt to determine it, and try to fly under > it. Generally, exponential-backoff > <https://en.wikipedia.org/wiki/Exponential_backoff> is a good tactic when > dealing with rate-limiting. > > Sincerely, > > Nick > Cloud Platform Community Support > > On Monday, May 2, 2016 at 11:57:15 AM UTC-4, Nickolas Daskalou wrote: >> >> Hi Ryan, >> >> It seems to be working fine for us (SocialPage.me >> <https://socialpage.me/>). >> >> Are you accessing their API using separate access tokens for each user? >> >> Nick >> >> >> On 2 May 2016 at 14:30, Ryan Barrett <[email protected] <javascript:>> >> wrote: >> >>> hi all! just FYI, it looks like Instagram is blocking/rate limiting App >>> Engine's IPs from fetching www.instagram.com, both urlfetch and >>> sockets, across apps. e.g. this session from >>> https://shell-hrd.appspot.com/ : >>> >>> >>> urllib2.urlopen('https://www.instagram.com/snarfed/') >>> Traceback (most recent call last): >>> ... >>> File >>> "/base/data/home/runtimes/python/python_dist/lib/python2.5/urllib2.py", >>> line 506, in http_error_default >>> raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) >>> HTTPError: HTTP Error 429: Unknown >>> >>> it's not 100% consistent - i occasionally see requests make it through - >>> but the majority get 429ed. >>> >>> not holding my breath, but i figured you all might want to know, >>> especially in case cloud support people have lines of communication open >>> with instagram/facebook for this kind of thing. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected] <javascript:>. >>> To post to this group, send email to [email protected] >>> <javascript:>. >>> Visit this group at https://groups.google.com/group/google-appengine. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/google-appengine/be7f6ead-fe34-45c4-9ee0-00956b5f89de%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/google-appengine/be7f6ead-fe34-45c4-9ee0-00956b5f89de%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/df130e09-f87b-40a7-bd4c-9ab0dc6fad0e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
