[twitter-dev] Re: Search API Rate limiting - App Engine (again)
Chad, Sorry for not being clear. I was thinking about Abraham William's suggestion above where Twitter Search API works with authenticated sessions+rate limiting, instead of IP based rate filtering. Just so you know, AppEngine has 30 second timeout on request to all AppEngine urls, and 10 second timeout on each individual HTTP request made within an AppEngine request. In case you are making multiple HTTP requests to Twitter within each individual AppEngine request, all the communication microseconds, from AppEngine to Proxy and Proxy to Twitter and then Twitter to Proxy and Proxy to AppEngine, quickly addup leading to timeouts. Personally i have tried quite a few scenarios to catch all the data i can, but from my experience, i can catch only 30%(sometimes better, sometimes almost nothing) of what i want, and rest just ends up with 503 and eventually since_id/max_id getting too old to get response from the Twitter Search API. So, right now Twitter is putting it's resources to offer a very robust Search API, but we as developers cannot use it effectively just 'cause of the way the hits are counted. Not to mention we are also investing funds to keep our apps running. Hope you understand our position. Thanks On Oct 18, 3:12 pm, Chad Etzel wrote: > On Sun, Oct 18, 2009 at 8:09 AM, vivekpuri wrote: > > > Will someone from Twitter please respond if there is an ETA to resolve > > this issue. Work arounds can never be really as effective as the real > > deal. > > Sorry, I thought it was clear from the previous email. There is no ETA > because it's not going to be resolved. GAE does not use an IP > infrastructure that is amicable to our rate-limiting logic, so if you > want to integrate IP rate-limited calls into your web-based > applications, you will need to either use the workaround stated > earlier or use a hosting service that will let you use a static IP. > > -Chad
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
On Sun, Oct 18, 2009 at 8:09 AM, vivekpuri wrote: > > Will someone from Twitter please respond if there is an ETA to resolve > this issue. Work arounds can never be really as effective as the real > deal. Sorry, I thought it was clear from the previous email. There is no ETA because it's not going to be resolved. GAE does not use an IP infrastructure that is amicable to our rate-limiting logic, so if you want to integrate IP rate-limited calls into your web-based applications, you will need to either use the workaround stated earlier or use a hosting service that will let you use a static IP. -Chad
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
Will someone from Twitter please respond if there is an ETA to resolve this issue. Work arounds can never be really as effective as the real deal.
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
I would recommend just using a physical server and uploading a simple php proxy script. If you have existing webspace, it will save you the trouble of setting up an complete ec2 build just to run a proxy script. On Oct 9, 7:11 pm, Akshar wrote: > Thanks Abraham. > > Any pointers on how to setup a proxy on amazon ec2 for GAE? > > On Oct 8, 6:07 pm, Abraham Williams <4bra...@gmail.com> wrote: > > > Pretty much. You have limited options: > > 1) Run your Search API requests through a proxy where you will have > > exclusive access to the IP. > > 2) Wait for V2 of the Twitter API where the REST and Search APIs get > > combined so you can have authenticated search queries. > > 3) Hope Twitter slaps some duct tape on the issue and rolls out a > > whitelisting method for the Search API that includes passkeys in your user > > agent or some such thing. > > 4) Develop on non cloud base infrastructure. > > 5) Something else. > > > Abraham > > > 2009/10/8 Akshar > > > >http://apiwiki.twitter.com/Rate-limitingstatesthat "for cloud > > > platforms like Google App Engine, applications without a static IP > > > addresses cannot receive Search whitelisting." > > > > Does that mean there is no way to avoid getting HTTP 503 response > > > codes to search requests from app engine? > > > > On Oct 8, 2:09 pm, Akshar wrote: > > > > Any other solutions available for app engine folks stuck out here? > > > > Please help! > > > > > I'm noticing this exact problem as well. I'm making only a few > > > > requests per hour. I have tried setting the user-agent but it did not > > > > help. > > > > > Akshar > > > > > On Oct 6, 9:50 am, Chad Etzel wrote: > > > > > > Hi All, > > > > > > GAE sites are problematic for the Twitter/SearchAPIbecause the IPs > > > > > making outgoing requests are fluid and cannot as such be easily > > > > > allowed for access. Also, since most IPs are shared, other > > > > > applications on the same IPs making requests mean that fewer requests > > > > > per app get through. > > > > > > One work around would be to spin up a server in EC2 or Rackspace Cloud > > > > > or something and use it as a proxy for your requests. That way you > > > > > have a dedicated IP that will have its full share of resources talking > > > > > with the Twitter servers. > > > > > > HTH, > > > > > -Chad > > > > > > On Tue, Oct 6, 2009 at 12:45 PM, Martin Omander > > > wrote: > > > > > > > Same here; my app runs on Google App Engine and 40% of the requests > > > to > > > > > > the TwitterSearchAPIget the 503 error message indicating rate > > > > > > limiting. > > > > > > > Is there anything we as app authors can do on our side to alleviate > > > > > > the problem? > > > > > > > /Martin > > > > > > > On Oct 5, 1:53 pm, Paul Kinlan wrote: > > > > > >> I am pretty sure there are custom headers on the App Engine that > > > indicate > > > > > >> the application that is sending the request. > > > > > > >> 2009/10/5 elkelk > > > > > > >> > Hi all, > > > > > > >> > I am having the same issue. I have tried setting a custom > > > user-agent, > > > > > >> > but this doesn't seem to affect the fact that twitter is limiting > > > > > >> > based on I.P. address. I'm only making about 5 searches an hour > > > and > > > > > >> > 80% of them are failing on app engine due to a 503 rate limit. > > > > > >> > Twitter needs to determine a better way to let cloud clients > > > access > > > > > >> > theirsearchAPI. It seems like they have really started blocking > > > > > >> >searchrequests in the last week or so. > > > > > > >> > If anyone has any idea about how to better identify my app engine > > > app > > > > > >> > please let let me know. > > > > > > >> > On Oct 5, 2:59 am, steel wrote: > > > > > >> > > Hi. I have this problem too. > > > > > >> > > My application does two request per hour and it get "rate > > > limit". > > > > > >> > > What is wrong? I think it is twitter's problems > > > > > > >> > > On 1 окт, 01:45, Paul Kinlan wrote: > > > > > > >> > > > Hi Guys, > > > > > >> > > > I have an app on the App engine using thesearchAPIand it is > > > getting > > > > > >> > > > heavily rate limited again this past couple of days. > > > > > > >> > > > I know that we are on a shared set of IP addresses and > > > > > >> > > > someone > > > else > > > > > >> > could be > > > > > >> > > > hammering the system, but it seems to run for weeks without > > > seeing the > > > > > >> > rate > > > > > >> > > > limit being hit and then all of a sudden only about 60% of > > > > > >> > > > the > > > searches > > > > > >> > > > I perform will be rate limited. This seems to occur every > > > > > >> > > > two > > > months > > > > > >> > or so. > > > > > > >> > > > Has something changed recently? > > > > > > >> > > > Paul > > > -- > > Abraham Williams | Community Evangelist |http://web608.org > > Hacker > > |http://abrah.am|http://twitter.com/abrahamhttp://web608.org/geeks/abraham/blogs/2009/... > > This email is: [ ] blogable [x] ask first [ ] private. > >
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
Thanks Abraham. Any pointers on how to setup a proxy on amazon ec2 for GAE? On Oct 8, 6:07 pm, Abraham Williams <4bra...@gmail.com> wrote: > Pretty much. You have limited options: > 1) Run your Search API requests through a proxy where you will have > exclusive access to the IP. > 2) Wait for V2 of the Twitter API where the REST and Search APIs get > combined so you can have authenticated search queries. > 3) Hope Twitter slaps some duct tape on the issue and rolls out a > whitelisting method for the Search API that includes passkeys in your user > agent or some such thing. > 4) Develop on non cloud base infrastructure. > 5) Something else. > > Abraham > > 2009/10/8 Akshar > > > > > > >http://apiwiki.twitter.com/Rate-limitingstates that "for cloud > > platforms like Google App Engine, applications without a static IP > > addresses cannot receive Search whitelisting." > > > Does that mean there is no way to avoid getting HTTP 503 response > > codes to search requests from app engine? > > > On Oct 8, 2:09 pm, Akshar wrote: > > > Any other solutions available for app engine folks stuck out here? > > > Please help! > > > > I'm noticing this exact problem as well. I'm making only a few > > > requests per hour. I have tried setting the user-agent but it did not > > > help. > > > > Akshar > > > > On Oct 6, 9:50 am, Chad Etzel wrote: > > > > > Hi All, > > > > > GAE sites are problematic for the Twitter/SearchAPIbecause the IPs > > > > making outgoing requests are fluid and cannot as such be easily > > > > allowed for access. Also, since most IPs are shared, other > > > > applications on the same IPs making requests mean that fewer requests > > > > per app get through. > > > > > One work around would be to spin up a server in EC2 or Rackspace Cloud > > > > or something and use it as a proxy for your requests. That way you > > > > have a dedicated IP that will have its full share of resources talking > > > > with the Twitter servers. > > > > > HTH, > > > > -Chad > > > > > On Tue, Oct 6, 2009 at 12:45 PM, Martin Omander > > wrote: > > > > > > Same here; my app runs on Google App Engine and 40% of the requests > > to > > > > > the TwitterSearchAPIget the 503 error message indicating rate > > > > > limiting. > > > > > > Is there anything we as app authors can do on our side to alleviate > > > > > the problem? > > > > > > /Martin > > > > > > On Oct 5, 1:53 pm, Paul Kinlan wrote: > > > > >> I am pretty sure there are custom headers on the App Engine that > > indicate > > > > >> the application that is sending the request. > > > > > >> 2009/10/5 elkelk > > > > > >> > Hi all, > > > > > >> > I am having the same issue. I have tried setting a custom > > user-agent, > > > > >> > but this doesn't seem to affect the fact that twitter is limiting > > > > >> > based on I.P. address. I'm only making about 5 searches an hour > > and > > > > >> > 80% of them are failing on app engine due to a 503 rate limit. > > > > >> > Twitter needs to determine a better way to let cloud clients > > access > > > > >> > theirsearchAPI. It seems like they have really started blocking > > > > >> >searchrequests in the last week or so. > > > > > >> > If anyone has any idea about how to better identify my app engine > > app > > > > >> > please let let me know. > > > > > >> > On Oct 5, 2:59 am, steel wrote: > > > > >> > > Hi. I have this problem too. > > > > >> > > My application does two request per hour and it get "rate > > limit". > > > > >> > > What is wrong? I think it is twitter's problems > > > > > >> > > On 1 окт, 01:45, Paul Kinlan wrote: > > > > > >> > > > Hi Guys, > > > > >> > > > I have an app on the App engine using thesearchAPIand it is > > getting > > > > >> > > > heavily rate limited again this past couple of days. > > > > > >> > > > I know that we are on a shared set of IP addresses and someone > > else > > > > >> > could be > > > > >> > > > hammering the system, but it seems to run for weeks without > > seeing the > > > > >> > rate > > > > >> > > > limit being hit and then all of a sudden only about 60% of the > > searches > > > > >> > > > I perform will be rate limited. This seems to occur every two > > months > > > > >> > or so. > > > > > >> > > > Has something changed recently? > > > > > >> > > > Paul > > -- > Abraham Williams | Community Evangelist |http://web608.org > Hacker > |http://abrah.am|http://twitter.com/abrahamhttp://web608.org/geeks/abraham/blogs/2009/10/03/win-google-wave-invite > This email is: [ ] blogable [x] ask first [ ] private. > Sent from Madison, WI, United States
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
Pretty much. You have limited options: 1) Run your Search API requests through a proxy where you will have exclusive access to the IP. 2) Wait for V2 of the Twitter API where the REST and Search APIs get combined so you can have authenticated search queries. 3) Hope Twitter slaps some duct tape on the issue and rolls out a whitelisting method for the Search API that includes passkeys in your user agent or some such thing. 4) Develop on non cloud base infrastructure. 5) Something else. Abraham 2009/10/8 Akshar > > http://apiwiki.twitter.com/Rate-limiting states that "for cloud > platforms like Google App Engine, applications without a static IP > addresses cannot receive Search whitelisting." > > Does that mean there is no way to avoid getting HTTP 503 response > codes to search requests from app engine? > > On Oct 8, 2:09 pm, Akshar wrote: > > Any other solutions available for app engine folks stuck out here? > > Please help! > > > > I'm noticing this exact problem as well. I'm making only a few > > requests per hour. I have tried setting the user-agent but it did not > > help. > > > > Akshar > > > > On Oct 6, 9:50 am, Chad Etzel wrote: > > > > > > > > > Hi All, > > > > > GAE sites are problematic for the Twitter/SearchAPIbecause the IPs > > > making outgoing requests are fluid and cannot as such be easily > > > allowed for access. Also, since most IPs are shared, other > > > applications on the same IPs making requests mean that fewer requests > > > per app get through. > > > > > One work around would be to spin up a server in EC2 or Rackspace Cloud > > > or something and use it as a proxy for your requests. That way you > > > have a dedicated IP that will have its full share of resources talking > > > with the Twitter servers. > > > > > HTH, > > > -Chad > > > > > On Tue, Oct 6, 2009 at 12:45 PM, Martin Omander > wrote: > > > > > > Same here; my app runs on Google App Engine and 40% of the requests > to > > > > the TwitterSearchAPIget the 503 error message indicating rate > > > > limiting. > > > > > > Is there anything we as app authors can do on our side to alleviate > > > > the problem? > > > > > > /Martin > > > > > > On Oct 5, 1:53 pm, Paul Kinlan wrote: > > > >> I am pretty sure there are custom headers on the App Engine that > indicate > > > >> the application that is sending the request. > > > > > >> 2009/10/5 elkelk > > > > > >> > Hi all, > > > > > >> > I am having the same issue. I have tried setting a custom > user-agent, > > > >> > but this doesn't seem to affect the fact that twitter is limiting > > > >> > based on I.P. address. I'm only making about 5 searches an hour > and > > > >> > 80% of them are failing on app engine due to a 503 rate limit. > > > >> > Twitter needs to determine a better way to let cloud clients > access > > > >> > theirsearchAPI. It seems like they have really started blocking > > > >> >searchrequests in the last week or so. > > > > > >> > If anyone has any idea about how to better identify my app engine > app > > > >> > please let let me know. > > > > > >> > On Oct 5, 2:59 am, steel wrote: > > > >> > > Hi. I have this problem too. > > > >> > > My application does two request per hour and it get "rate > limit". > > > >> > > What is wrong? I think it is twitter's problems > > > > > >> > > On 1 окт, 01:45, Paul Kinlan wrote: > > > > > >> > > > Hi Guys, > > > >> > > > I have an app on the App engine using thesearchAPIand it is > getting > > > >> > > > heavily rate limited again this past couple of days. > > > > > >> > > > I know that we are on a shared set of IP addresses and someone > else > > > >> > could be > > > >> > > > hammering the system, but it seems to run for weeks without > seeing the > > > >> > rate > > > >> > > > limit being hit and then all of a sudden only about 60% of the > searches > > > >> > > > I perform will be rate limited. This seems to occur every two > months > > > >> > or so. > > > > > >> > > > Has something changed recently? > > > > > >> > > > Paul > -- Abraham Williams | Community Evangelist | http://web608.org Hacker | http://abrah.am | http://twitter.com/abraham http://web608.org/geeks/abraham/blogs/2009/10/03/win-google-wave-invite This email is: [ ] blogable [x] ask first [ ] private. Sent from Madison, WI, United States
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
http://apiwiki.twitter.com/Rate-limiting states that "for cloud platforms like Google App Engine, applications without a static IP addresses cannot receive Search whitelisting." Does that mean there is no way to avoid getting HTTP 503 response codes to search requests from app engine? On Oct 8, 2:09 pm, Akshar wrote: > Any other solutions available for app engine folks stuck out here? > Please help! > > I'm noticing this exact problem as well. I'm making only a few > requests per hour. I have tried setting the user-agent but it did not > help. > > Akshar > > On Oct 6, 9:50 am, Chad Etzel wrote: > > > > > Hi All, > > > GAE sites are problematic for the Twitter/SearchAPIbecause the IPs > > making outgoing requests are fluid and cannot as such be easily > > allowed for access. Also, since most IPs are shared, other > > applications on the same IPs making requests mean that fewer requests > > per app get through. > > > One work around would be to spin up a server in EC2 or Rackspace Cloud > > or something and use it as a proxy for your requests. That way you > > have a dedicated IP that will have its full share of resources talking > > with the Twitter servers. > > > HTH, > > -Chad > > > On Tue, Oct 6, 2009 at 12:45 PM, Martin Omander wrote: > > > > Same here; my app runs on Google App Engine and 40% of the requests to > > > the TwitterSearchAPIget the 503 error message indicating rate > > > limiting. > > > > Is there anything we as app authors can do on our side to alleviate > > > the problem? > > > > /Martin > > > > On Oct 5, 1:53 pm, Paul Kinlan wrote: > > >> I am pretty sure there are custom headers on the App Engine that indicate > > >> the application that is sending the request. > > > >> 2009/10/5 elkelk > > > >> > Hi all, > > > >> > I am having the same issue. I have tried setting a custom user-agent, > > >> > but this doesn't seem to affect the fact that twitter is limiting > > >> > based on I.P. address. I'm only making about 5 searches an hour and > > >> > 80% of them are failing on app engine due to a 503 rate limit. > > >> > Twitter needs to determine a better way to let cloud clients access > > >> > theirsearchAPI. It seems like they have really started blocking > > >> >searchrequests in the last week or so. > > > >> > If anyone has any idea about how to better identify my app engine app > > >> > please let let me know. > > > >> > On Oct 5, 2:59 am, steel wrote: > > >> > > Hi. I have this problem too. > > >> > > My application does two request per hour and it get "rate limit". > > >> > > What is wrong? I think it is twitter's problems > > > >> > > On 1 окт, 01:45, Paul Kinlan wrote: > > > >> > > > Hi Guys, > > >> > > > I have an app on the App engine using thesearchAPIand it is getting > > >> > > > heavily rate limited again this past couple of days. > > > >> > > > I know that we are on a shared set of IP addresses and someone else > > >> > could be > > >> > > > hammering the system, but it seems to run for weeks without seeing > > >> > > > the > > >> > rate > > >> > > > limit being hit and then all of a sudden only about 60% of the > > >> > > > searches > > >> > > > I perform will be rate limited. This seems to occur every two > > >> > > > months > > >> > or so. > > > >> > > > Has something changed recently? > > > >> > > > Paul
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
Any other solutions available for app engine folks stuck out here? Please help! I'm noticing this exact problem as well. I'm making only a few requests per hour. I have tried setting the user-agent but it did not help. Akshar On Oct 6, 9:50 am, Chad Etzel wrote: > Hi All, > > GAE sites are problematic for the Twitter/Search API because the IPs > making outgoing requests are fluid and cannot as such be easily > allowed for access. Also, since most IPs are shared, other > applications on the same IPs making requests mean that fewer requests > per app get through. > > One work around would be to spin up a server in EC2 or Rackspace Cloud > or something and use it as a proxy for your requests. That way you > have a dedicated IP that will have its full share of resources talking > with the Twitter servers. > > HTH, > -Chad > > > > On Tue, Oct 6, 2009 at 12:45 PM, Martin Omander wrote: > > > Same here; my app runs on Google App Engine and 40% of the requests to > > the Twitter Search API get the 503 error message indicating rate > > limiting. > > > Is there anything we as app authors can do on our side to alleviate > > the problem? > > > /Martin > > > On Oct 5, 1:53 pm, Paul Kinlan wrote: > >> I am pretty sure there are custom headers on the App Engine that indicate > >> the application that is sending the request. > > >> 2009/10/5 elkelk > > >> > Hi all, > > >> > I am having the same issue. I have tried setting a custom user-agent, > >> > but this doesn't seem to affect the fact that twitter is limiting > >> > based on I.P. address. I'm only making about 5 searches an hour and > >> > 80% of them are failing on app engine due to a 503 rate limit. > >> > Twitter needs to determine a better way to let cloud clients access > >> > their search API. It seems like they have really started blocking > >> > search requests in the last week or so. > > >> > If anyone has any idea about how to better identify my app engine app > >> > please let let me know. > > >> > On Oct 5, 2:59 am, steel wrote: > >> > > Hi. I have this problem too. > >> > > My application does two request per hour and it get "rate limit". > >> > > What is wrong? I think it is twitter's problems > > >> > > On 1 окт, 01:45, Paul Kinlan wrote: > > >> > > > Hi Guys, > >> > > > I have an app on the App engine using the search API and it is > >> > > > getting > >> > > > heavily rate limited again this past couple of days. > > >> > > > I know that we are on a shared set of IP addresses and someone else > >> > could be > >> > > > hammering the system, but it seems to run for weeks without seeing > >> > > > the > >> > rate > >> > > > limit being hit and then all of a sudden only about 60% of the > >> > > > searches > >> > > > I perform will be rate limited. This seems to occur every two months > >> > or so. > > >> > > > Has something changed recently? > > >> > > > Paul
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
I have solved a problem like that: While I receive an error 503 - my application continue knocking to twitter with query. Everything works ;)
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
Twitter should really in this case either white list all GAE IPs (I'm sure an email to Google could get all IPs they use) or allow charging API requests to an authenticated account rather than by IP (much like the REST API does). This way each GAE application would just set up a twitter account and each gets its own 150 request per hour. Josh
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
I am also facing this issue. I'm only making a couple of requests from GAE (about 3-4) and none of them are getting through, I keep getting the following using Twitter4J Twitter Exception while retrieving status twitter4j.TwitterException: 400:The request was invalid. An accompanying error message will explain why. This is the status code will be returned during rate limiting. /statuses/show/2.xml Rate limit exceeded. Clients may not make more than 150 requests per hour. On Oct 6, 7:13 pm, Paul Kinlan wrote: > Hi Chad, > > I am sorry but that doesn't even help in the slightest. > > You are essentially saying that we shouldn't develop on the App > Engine, since would now have to also buy a proxy. Which is completely > unfeasible and defeats the purpose of why people are using the app > engine. > > I understand that this might also be an App Engine issue - for > instance they could have reduced the number of IP addresses they pool > from to make external requests. > > This is a very noticeable change inratelimiting in the last few > weeks. For instance I could run roughly 2 searches a second, then all > of a sudden I would be lucky to run 2 every 15 seconds. User-Agent > strings were supposed to allievate this issue. There are more than > enough pieces of meta data on an App Engine request that Identify the > exact application that is making the requests - I guess it is too much > effort to take these into account. > > I am in the fortunate position that allowed me to set up a nginx proxy > quickly, but I suspect a lot of other people couldn't do that. > > I hope something can be sorted for the large number of GAE based > Twitter apps. > > Paul Kinlan > > On 6 Oct 2009, at 17:50, Chad Etzel wrote: > > > > > Hi All, > > > GAE sites are problematic for the Twitter/Search API because the IPs > > making outgoing requests are fluid and cannot as such be easily > > allowed for access. Also, since most IPs are shared, other > > applications on the same IPs making requests mean that fewer requests > > per app get through. > > > One work around would be to spin up a server in EC2 or Rackspace Cloud > > or something and use it as a proxy for your requests. That way you > > have a dedicated IP that will have its full share of resources talking > > with the Twitter servers. > > > HTH, > > -Chad > > > On Tue, Oct 6, 2009 at 12:45 PM, Martin Omander > > wrote: > > >> Same here; my app runs on Google App Engine and 40% of the requests > >> to > >> the Twitter Search API get the 503 error message indicatingrate > >> limiting. > > >> Is there anything we as app authors can do on our side to alleviate > >> the problem? > > >> /Martin > > >> On Oct 5, 1:53 pm, Paul Kinlan wrote: > >>> I am pretty sure there are custom headers on the App Engine that > >>> indicate > >>> the application that is sending the request. > > >>> 2009/10/5 elkelk > > Hi all, > > I am having the same issue. I have tried setting a custom user- > agent, > but this doesn't seem to affect the fact that twitter is limiting > based on I.P. address. I'm only making about 5 searches an hour > and > 80% of them are failing on app engine due to a 503ratelimit. > Twitter needs to determine a better way to let cloud clients access > their search API. It seems like they have really started blocking > search requests in the last week or so. > > If anyone has any idea about how to better identify my app engine > app > please let let me know. > > On Oct 5, 2:59 am, steel wrote: > > Hi. I have this problem too. > > My application does two request per hour and it get "ratelimit". > > What is wrong? I think it is twitter's problems > > > On 1 окт, 01:45, Paul Kinlan wrote: > > >> Hi Guys, > >> I have an app on the App engine using the search API and it is > >> getting > >> heavilyratelimited again this past couple of days. > > >> I know that we are on a shared set of IP addresses and someone > >> else > could be > >> hammering the system, but it seems to run for weeks without > >> seeing the > rate > >>limitbeing hit and then all of a sudden only about 60% of the > >> searches > >> I perform will beratelimited. This seems to occur every two > >> months > or so. > > >> Has something changed recently? > > >> Paul
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
Hi Chad, I am sorry but that doesn't even help in the slightest. You are essentially saying that we shouldn't develop on the App Engine, since would now have to also buy a proxy. Which is completely unfeasible and defeats the purpose of why people are using the app engine. I understand that this might also be an App Engine issue - for instance they could have reduced the number of IP addresses they pool from to make external requests. This is a very noticeable change in rate limiting in the last few weeks. For instance I could run roughly 2 searches a second, then all of a sudden I would be lucky to run 2 every 15 seconds. User-Agent strings were supposed to allievate this issue. There are more than enough pieces of meta data on an App Engine request that Identify the exact application that is making the requests - I guess it is too much effort to take these into account. I am in the fortunate position that allowed me to set up a nginx proxy quickly, but I suspect a lot of other people couldn't do that. I hope something can be sorted for the large number of GAE based Twitter apps. Paul Kinlan On 6 Oct 2009, at 17:50, Chad Etzel wrote: > > Hi All, > > GAE sites are problematic for the Twitter/Search API because the IPs > making outgoing requests are fluid and cannot as such be easily > allowed for access. Also, since most IPs are shared, other > applications on the same IPs making requests mean that fewer requests > per app get through. > > One work around would be to spin up a server in EC2 or Rackspace Cloud > or something and use it as a proxy for your requests. That way you > have a dedicated IP that will have its full share of resources talking > with the Twitter servers. > > HTH, > -Chad > > On Tue, Oct 6, 2009 at 12:45 PM, Martin Omander > wrote: >> >> >> Same here; my app runs on Google App Engine and 40% of the requests >> to >> the Twitter Search API get the 503 error message indicating rate >> limiting. >> >> Is there anything we as app authors can do on our side to alleviate >> the problem? >> >> /Martin >> >> >> On Oct 5, 1:53 pm, Paul Kinlan wrote: >>> I am pretty sure there are custom headers on the App Engine that >>> indicate >>> the application that is sending the request. >>> >>> 2009/10/5 elkelk >>> >>> >>> >>> >>> Hi all, >>> I am having the same issue. I have tried setting a custom user- agent, but this doesn't seem to affect the fact that twitter is limiting based on I.P. address. I'm only making about 5 searches an hour and 80% of them are failing on app engine due to a 503 rate limit. Twitter needs to determine a better way to let cloud clients access their search API. It seems like they have really started blocking search requests in the last week or so. >>> If anyone has any idea about how to better identify my app engine app please let let me know. >>> On Oct 5, 2:59 am, steel wrote: > Hi. I have this problem too. > My application does two request per hour and it get "rate limit". > What is wrong? I think it is twitter's problems >>> > On 1 окт, 01:45, Paul Kinlan wrote: >>> >> Hi Guys, >> I have an app on the App engine using the search API and it is >> getting >> heavily rate limited again this past couple of days. >>> >> I know that we are on a shared set of IP addresses and someone >> else could be >> hammering the system, but it seems to run for weeks without >> seeing the rate >> limit being hit and then all of a sudden only about 60% of the >> searches >> I perform will be rate limited. This seems to occur every two >> months or so. >>> >> Has something changed recently? >>> >> Paul >>
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
Hi All, GAE sites are problematic for the Twitter/Search API because the IPs making outgoing requests are fluid and cannot as such be easily allowed for access. Also, since most IPs are shared, other applications on the same IPs making requests mean that fewer requests per app get through. One work around would be to spin up a server in EC2 or Rackspace Cloud or something and use it as a proxy for your requests. That way you have a dedicated IP that will have its full share of resources talking with the Twitter servers. HTH, -Chad On Tue, Oct 6, 2009 at 12:45 PM, Martin Omander wrote: > > > Same here; my app runs on Google App Engine and 40% of the requests to > the Twitter Search API get the 503 error message indicating rate > limiting. > > Is there anything we as app authors can do on our side to alleviate > the problem? > > /Martin > > > On Oct 5, 1:53 pm, Paul Kinlan wrote: >> I am pretty sure there are custom headers on the App Engine that indicate >> the application that is sending the request. >> >> 2009/10/5 elkelk >> >> >> >> >> >> > Hi all, >> >> > I am having the same issue. I have tried setting a custom user-agent, >> > but this doesn't seem to affect the fact that twitter is limiting >> > based on I.P. address. I'm only making about 5 searches an hour and >> > 80% of them are failing on app engine due to a 503 rate limit. >> > Twitter needs to determine a better way to let cloud clients access >> > their search API. It seems like they have really started blocking >> > search requests in the last week or so. >> >> > If anyone has any idea about how to better identify my app engine app >> > please let let me know. >> >> > On Oct 5, 2:59 am, steel wrote: >> > > Hi. I have this problem too. >> > > My application does two request per hour and it get "rate limit". >> > > What is wrong? I think it is twitter's problems >> >> > > On 1 окт, 01:45, Paul Kinlan wrote: >> >> > > > Hi Guys, >> > > > I have an app on the App engine using the search API and it is getting >> > > > heavily rate limited again this past couple of days. >> >> > > > I know that we are on a shared set of IP addresses and someone else >> > could be >> > > > hammering the system, but it seems to run for weeks without seeing the >> > rate >> > > > limit being hit and then all of a sudden only about 60% of the searches >> > > > I perform will be rate limited. This seems to occur every two months >> > or so. >> >> > > > Has something changed recently? >> >> > > > Paul >
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
Same here; my app runs on Google App Engine and 40% of the requests to the Twitter Search API get the 503 error message indicating rate limiting. Is there anything we as app authors can do on our side to alleviate the problem? /Martin On Oct 5, 1:53 pm, Paul Kinlan wrote: > I am pretty sure there are custom headers on the App Engine that indicate > the application that is sending the request. > > 2009/10/5 elkelk > > > > > > > Hi all, > > > I am having the same issue. I have tried setting a custom user-agent, > > but this doesn't seem to affect the fact that twitter is limiting > > based on I.P. address. I'm only making about 5 searches an hour and > > 80% of them are failing on app engine due to a 503 rate limit. > > Twitter needs to determine a better way to let cloud clients access > > their search API. It seems like they have really started blocking > > search requests in the last week or so. > > > If anyone has any idea about how to better identify my app engine app > > please let let me know. > > > On Oct 5, 2:59 am, steel wrote: > > > Hi. I have this problem too. > > > My application does two request per hour and it get "rate limit". > > > What is wrong? I think it is twitter's problems > > > > On 1 окт, 01:45, Paul Kinlan wrote: > > > > > Hi Guys, > > > > I have an app on the App engine using the search API and it is getting > > > > heavily rate limited again this past couple of days. > > > > > I know that we are on a shared set of IP addresses and someone else > > could be > > > > hammering the system, but it seems to run for weeks without seeing the > > rate > > > > limit being hit and then all of a sudden only about 60% of the searches > > > > I perform will be rate limited. This seems to occur every two months > > or so. > > > > > Has something changed recently? > > > > > Paul
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
I am pretty sure there are custom headers on the App Engine that indicate the application that is sending the request. 2009/10/5 elkelk > > Hi all, > > I am having the same issue. I have tried setting a custom user-agent, > but this doesn't seem to affect the fact that twitter is limiting > based on I.P. address. I'm only making about 5 searches an hour and > 80% of them are failing on app engine due to a 503 rate limit. > Twitter needs to determine a better way to let cloud clients access > their search API. It seems like they have really started blocking > search requests in the last week or so. > > If anyone has any idea about how to better identify my app engine app > please let let me know. > > > > On Oct 5, 2:59 am, steel wrote: > > Hi. I have this problem too. > > My application does two request per hour and it get "rate limit". > > What is wrong? I think it is twitter's problems > > > > On 1 окт, 01:45, Paul Kinlan wrote: > > > > > Hi Guys, > > > I have an app on the App engine using the search API and it is getting > > > heavily rate limited again this past couple of days. > > > > > I know that we are on a shared set of IP addresses and someone else > could be > > > hammering the system, but it seems to run for weeks without seeing the > rate > > > limit being hit and then all of a sudden only about 60% of the searches > > > I perform will be rate limited. This seems to occur every two months > or so. > > > > > Has something changed recently? > > > > > Paul > > > > >
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
I'm noticing this problem as well. I'm making only a couple requests per hour. I have tried setting the user-agent and the HTTP_REFERER headers to a custom name, but Twitter doesn't seem to care. On Oct 5, 2:59 am, steel wrote: > Hi. I have this problem too. > My application does two request per hour and it get "rate limit". > What is wrong? I think it is twitter's problems > > On 1 окт, 01:45, Paul Kinlan wrote: > > > Hi Guys, > > I have an app on the App engine using the search API and it is getting > > heavily rate limited again this past couple of days. > > > I know that we are on a shared set of IP addresses and someone else could be > > hammering the system, but it seems to run for weeks without seeing the rate > > limit being hit and then all of a sudden only about 60% of the searches > > I perform will be rate limited. This seems to occur every two months or so. > > > Has something changed recently? > > > Paul > >
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
Hi all, I am having the same issue. I have tried setting a custom user-agent, but this doesn't seem to affect the fact that twitter is limiting based on I.P. address. I'm only making about 5 searches an hour and 80% of them are failing on app engine due to a 503 rate limit. Twitter needs to determine a better way to let cloud clients access their search API. It seems like they have really started blocking search requests in the last week or so. If anyone has any idea about how to better identify my app engine app please let let me know. On Oct 5, 2:59 am, steel wrote: > Hi. I have this problem too. > My application does two request per hour and it get "rate limit". > What is wrong? I think it is twitter's problems > > On 1 окт, 01:45, Paul Kinlan wrote: > > > Hi Guys, > > I have an app on the App engine using the search API and it is getting > > heavily rate limited again this past couple of days. > > > I know that we are on a shared set of IP addresses and someone else could be > > hammering the system, but it seems to run for weeks without seeing the rate > > limit being hit and then all of a sudden only about 60% of the searches > > I perform will be rate limited. This seems to occur every two months or so. > > > Has something changed recently? > > > Paul > >
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
Hi. I have this problem too. My application does two request per hour and it get "rate limit". What is wrong? I think it is twitter's problems On 1 окт, 01:45, Paul Kinlan wrote: > Hi Guys, > I have an app on the App engine using the search API and it is getting > heavily rate limited again this past couple of days. > > I know that we are on a shared set of IP addresses and someone else could be > hammering the system, but it seems to run for weeks without seeing the rate > limit being hit and then all of a sudden only about 60% of the searches > I perform will be rate limited. This seems to occur every two months or so. > > Has something changed recently? > > Paul
