Re: [twitter-dev] Re: Search API rate limit change?
I just wanted to add to this. The 420s have let up for the most part and I'm no longer seeing rate limiting behavior significantly different from the norm. I've noticed that many result pages are coming back with empty results but if I re-request the same page (after a couple second delay), I can often get results for that page. These are for queries with very low tweet velocity, so it's not like these are new results coming in. Is this related to http://status.twitter.com/post/3785043723/slow-searches ? Thanks. Hayes On Mon, Mar 21, 2011 at 6:13 PM, Matt Harris thematthar...@twitter.comwrote: In many cases we are forced to change the rate limits in response to a significant increase in requests, which means it isn't always possible to give advanced notice of rate limit changes. For some of you it sounds like your code that handles rate limiting didn't react appropriately. When receiving a 420 response we recommend you stop making requests and then after the retry-after, slowly build up the number of requests you make. Put another way it isn't a good idea to make requests to the API at the velocity that caused the 420 response before. As always, the rate limits are there to ensure the system is responsive and available to as many users as possible. This means it is necessary to reduce the number of queries you can make without notice. The best place to stay informed about issues like this are posted through @twitterapi and published on the Twitter status blog: http://status.twitter.com/post/3785043723/slow-searches Best, @themattharris Developer Advocate, Twitter http://twitter.com/themattharris On Mon, Mar 21, 2011 at 7:41 AM, Colin Surprenant colin.surpren...@gmail.com wrote: By adjusting the rate limits to reduce the stress on your search api without notice you have significantly increased the stress level on our end :P Seriously, advanced notice of the situation would have been welcome. In particular what created lots of confusion on our end is that even after pausing for the specified retry_after delay we would immediately get repeated 420s at which point we started to assume our IPs were banned (which also contributed to increase the stress level). Colin On Mar 21, 9:12 am, Jeffrey Greenberg jeffreygreenb...@gmail.com wrote: Taylor, Yeah this was definitely NOT good.In the past, when there is a service disruption, your api group would post something on your status page and tweet about it... Instead, I'm finding out about this from my customers... Did y'all tweet about this or present this somewhere where I could find it? Jeffrey Tweettronics.com On Sun, Mar 20, 2011 at 3:14 PM, Waldron Faulkner waldronfaulk...@gmail.com wrote: Without prior notice, I can understand (circumstances), but without any kind of subsequent announcement?? Means we have to discover issues ourselves, verify that they're Twitter related (and not internal), then search around for existing discussion on the topic. Saves us a lot of time and headaches if Twitter would just announce stuff like this. On Mar 18, 2:51 pm, Taylor Singletary taylorsinglet...@twitter.com wrote: We're working to reinstate the usual limits on the Search API; due to the impact of the Japanese earthquake and resultant query increase against the Search API, some rates were adjusted to cope better serve queries. Will give everyone an update with the various limits are adjusted. @episod http://twitter.com/episod - Taylor Singletary - Twitter Developer Advocate On Fri, Mar 18, 2011 at 11:39 AM, Hayes Davis ha...@appozite.com wrote: Hi, We're seeing this as well starting at approximately the same time as described. We've backed off on searching but are seeing no reduction in the sporadic limiting. It also appears that the amount of results returned on successful queries is severely limited. Some queries that often have 1500 tweets from the last 5 days are returning far fewer results from only the last day. Could we get an update on this? Hayes On Fri, Mar 18, 2011 at 10:13 AM, Eric e...@telvetto.com wrote: We're also seeing 400s on different boxes across different IP addresses with different queries (so it does not appear to be server or query specific). These began on all boxes at 2 a.m. UTC. We've backed off on both number and rate of queries with no effect. We've also noticed an increase in sporadic fail whales via browser based search (atom and html) from personal accounts, although we haven't attempted to quantify it. On Mar 18, 7:40 am, zaver zave...@hotmail.com wrote: Hello, After the latest performance issues with the search api i have been seeing a lot of 420 response codes.From yesterday until now i only get 420 responses on the every search i make. In particular, i search for about
Re: [twitter-dev] Re: Search API rate limit change?
Taylor, Yeah this was definitely NOT good.In the past, when there is a service disruption, your api group would post something on your status page and tweet about it... Instead, I'm finding out about this from my customers... Did y'all tweet about this or present this somewhere where I could find it? Jeffrey Tweettronics.com On Sun, Mar 20, 2011 at 3:14 PM, Waldron Faulkner waldronfaulk...@gmail.com wrote: Without prior notice, I can understand (circumstances), but without any kind of subsequent announcement?? Means we have to discover issues ourselves, verify that they're Twitter related (and not internal), then search around for existing discussion on the topic. Saves us a lot of time and headaches if Twitter would just announce stuff like this. On Mar 18, 2:51 pm, Taylor Singletary taylorsinglet...@twitter.com wrote: We're working to reinstate the usual limits on the Search API; due to the impact of the Japanese earthquake and resultant query increase against the Search API, some rates were adjusted to cope better serve queries. Will give everyone an update with the various limits are adjusted. @episod http://twitter.com/episod - Taylor Singletary - Twitter Developer Advocate On Fri, Mar 18, 2011 at 11:39 AM, Hayes Davis ha...@appozite.com wrote: Hi, We're seeing this as well starting at approximately the same time as described. We've backed off on searching but are seeing no reduction in the sporadic limiting. It also appears that the amount of results returned on successful queries is severely limited. Some queries that often have 1500 tweets from the last 5 days are returning far fewer results from only the last day. Could we get an update on this? Hayes On Fri, Mar 18, 2011 at 10:13 AM, Eric e...@telvetto.com wrote: We're also seeing 400s on different boxes across different IP addresses with different queries (so it does not appear to be server or query specific). These began on all boxes at 2 a.m. UTC. We've backed off on both number and rate of queries with no effect. We've also noticed an increase in sporadic fail whales via browser based search (atom and html) from personal accounts, although we haven't attempted to quantify it. On Mar 18, 7:40 am, zaver zave...@hotmail.com wrote: Hello, After the latest performance issues with the search api i have been seeing a lot of 420 response codes.From yesterday until now i only get 420 responses on the every search i make. In particular, i search for about 100 keywords simultaneously every 6 mins. Why is this happening? Was there any change on the Search API limit? Any help is greatly appreciated. Thanks, Zaver -- Twitter developer documentation and resources:http://dev.twitter.com/doc API updates via Twitter:http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources:http://dev.twitter.com/doc API updates via Twitter:http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Re: Search API rate limit change?
In many cases we are forced to change the rate limits in response to a significant increase in requests, which means it isn't always possible to give advanced notice of rate limit changes. For some of you it sounds like your code that handles rate limiting didn't react appropriately. When receiving a 420 response we recommend you stop making requests and then after the retry-after, slowly build up the number of requests you make. Put another way it isn't a good idea to make requests to the API at the velocity that caused the 420 response before. As always, the rate limits are there to ensure the system is responsive and available to as many users as possible. This means it is necessary to reduce the number of queries you can make without notice. The best place to stay informed about issues like this are posted through @twitterapi and published on the Twitter status blog: http://status.twitter.com/post/3785043723/slow-searches Best, @themattharris Developer Advocate, Twitter http://twitter.com/themattharris On Mon, Mar 21, 2011 at 7:41 AM, Colin Surprenant colin.surpren...@gmail.com wrote: By adjusting the rate limits to reduce the stress on your search api without notice you have significantly increased the stress level on our end :P Seriously, advanced notice of the situation would have been welcome. In particular what created lots of confusion on our end is that even after pausing for the specified retry_after delay we would immediately get repeated 420s at which point we started to assume our IPs were banned (which also contributed to increase the stress level). Colin On Mar 21, 9:12 am, Jeffrey Greenberg jeffreygreenb...@gmail.com wrote: Taylor, Yeah this was definitely NOT good.In the past, when there is a service disruption, your api group would post something on your status page and tweet about it... Instead, I'm finding out about this from my customers... Did y'all tweet about this or present this somewhere where I could find it? Jeffrey Tweettronics.com On Sun, Mar 20, 2011 at 3:14 PM, Waldron Faulkner waldronfaulk...@gmail.com wrote: Without prior notice, I can understand (circumstances), but without any kind of subsequent announcement?? Means we have to discover issues ourselves, verify that they're Twitter related (and not internal), then search around for existing discussion on the topic. Saves us a lot of time and headaches if Twitter would just announce stuff like this. On Mar 18, 2:51 pm, Taylor Singletary taylorsinglet...@twitter.com wrote: We're working to reinstate the usual limits on the Search API; due to the impact of the Japanese earthquake and resultant query increase against the Search API, some rates were adjusted to cope better serve queries. Will give everyone an update with the various limits are adjusted. @episod http://twitter.com/episod - Taylor Singletary - Twitter Developer Advocate On Fri, Mar 18, 2011 at 11:39 AM, Hayes Davis ha...@appozite.com wrote: Hi, We're seeing this as well starting at approximately the same time as described. We've backed off on searching but are seeing no reduction in the sporadic limiting. It also appears that the amount of results returned on successful queries is severely limited. Some queries that often have 1500 tweets from the last 5 days are returning far fewer results from only the last day. Could we get an update on this? Hayes On Fri, Mar 18, 2011 at 10:13 AM, Eric e...@telvetto.com wrote: We're also seeing 400s on different boxes across different IP addresses with different queries (so it does not appear to be server or query specific). These began on all boxes at 2 a.m. UTC. We've backed off on both number and rate of queries with no effect. We've also noticed an increase in sporadic fail whales via browser based search (atom and html) from personal accounts, although we haven't attempted to quantify it. On Mar 18, 7:40 am, zaver zave...@hotmail.com wrote: Hello, After the latest performance issues with the search api i have been seeing a lot of 420 response codes.From yesterday until now i only get 420 responses on the every search i make. In particular, i search for about 100 keywords simultaneously every 6 mins. Why is this happening? Was there any change on the Search API limit? Any help is greatly appreciated. Thanks, Zaver -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter:http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via
Re: [twitter-dev] Re: Search API rate limit change?
We're working to reinstate the usual limits on the Search API; due to the impact of the Japanese earthquake and resultant query increase against the Search API, some rates were adjusted to cope better serve queries. Will give everyone an update with the various limits are adjusted. @episod http://twitter.com/episod - Taylor Singletary - Twitter Developer Advocate On Fri, Mar 18, 2011 at 11:39 AM, Hayes Davis ha...@appozite.com wrote: Hi, We're seeing this as well starting at approximately the same time as described. We've backed off on searching but are seeing no reduction in the sporadic limiting. It also appears that the amount of results returned on successful queries is severely limited. Some queries that often have 1500 tweets from the last 5 days are returning far fewer results from only the last day. Could we get an update on this? Hayes On Fri, Mar 18, 2011 at 10:13 AM, Eric e...@telvetto.com wrote: We're also seeing 400s on different boxes across different IP addresses with different queries (so it does not appear to be server or query specific). These began on all boxes at 2 a.m. UTC. We've backed off on both number and rate of queries with no effect. We've also noticed an increase in sporadic fail whales via browser based search (atom and html) from personal accounts, although we haven't attempted to quantify it. On Mar 18, 7:40 am, zaver zave...@hotmail.com wrote: Hello, After the latest performance issues with the search api i have been seeing a lot of 420 response codes.From yesterday until now i only get 420 responses on the every search i make. In particular, i search for about 100 keywords simultaneously every 6 mins. Why is this happening? Was there any change on the Search API limit? Any help is greatly appreciated. Thanks, Zaver -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Re: Search API rate limit
Shan, as far as I know twitter has been reluctant to state definite numbers, so you'll have to experiment and implement a backoff mechanism in your app. Here is the relevant part of the docs: Search API Rate Limiting The Search API is rate limited by IP address. The number of search requests that originate from a given IP address are counted against the search rate limiter. The specific number of requests a client is able to make to the Search API for a given hour is not released. Note that the Search API is not limited by the same 150 requests per hour limit as the REST API. The number is quite a bit higher and we feel it is both liberal and sufficient for most applications. We do not give the exact number because we want to discourage unnecessary search usage. Search API usage requires that applications include a unique and identifying User Agent string. A HTTP Referrer is expected but is not required. Consumers using the Search API but failing to include a User Agent string will receive a lower rate limit. An application that exceeds the rate limitations of the Search API will receive HTTP 420 response codes to requests. It is a best practice to watch for this error condition and honor the Retry-After header that instructs the application when it is safe to continue. The Retry-After header's value is the number of seconds your application should wait before submitting another query (for example: Retry-After: 67). Cheers, Pascal On Jul 7, 2010, at 1:55 , Ramanean wrote: Matt, What is exact limit..Whether I can write to twitter for whitelisting of the IP? Whether whitelisting of the IP would do any good? Shan
