[twitter-dev] Re: Search API Rate Limiting and Performance
Search API will rock if it would only be reliable what we see looks to be some sort of a funky cache, a query (atom) can be missing some latest tweets and then after a while they show up, if you tweak the query you can see 'em. you ever seen this problem? also what did you do special with user agent? On Sep 2, 12:12 pm, Amitab hiamita...@gmail.com wrote: I have exchanged emails with Twitter on this and I believe they are working on it. We use search extensively atwww.Twaller.com. The errors in search that we are seing is as follows: (1) HTTP status code: 403 Message:The request is understood, but it has been refused. An accompanying error message will explain why. (2)HTTP status code: 503 Message:Service Unavailable: The Twitter servers are up, but overloaded with requests. Try again later. The search and trend methods use this to indicate when you are being rate limited. (3) HTTP status code: -1 Message:Read timed out These errors are becoming more and more frequent lately. /Amitab On Sep 1, 5:41 pm, Dewald Pretorius dpr...@gmail.com wrote: Twitter team, can you please do something about the performance and rate limiting of the Search API. It is becoming completely unworkable. I have jumped through all the hoops, with unique User Agents, sleeping my scripts in-between API calls, and yet the rate limiting is just becoming more severe, and the performance is just becoming slower and slower. Please help. A lot of us are using Search to provide services, and I am sure it is not only my service that is hurting very much under the present circumstances. Dewald
[twitter-dev] Re: Search API Rate Limiting and Performance
The Search team is working on indexing latency and throughput, along with a many other things. There have been big improvements recently and more are on the way. In the mean time, if you need closer to real-time results, consider the track parameter on the Streaming API. -John Kalucki http://twitter.com/jkalucki Services, Twitter Inc. On Sep 4, 6:58 am, pxpilot pxpi...@gmail.com wrote: Search API will rock if it would only be reliable what we see looks to be some sort of a funky cache, a query (atom) can be missing some latest tweets and then after a while they show up, if you tweak the query you can see 'em. you ever seen this problem? also what did you do special with user agent? On Sep 2, 12:12 pm, Amitab hiamita...@gmail.com wrote: I have exchanged emails with Twitter on this and I believe they are working on it. We use search extensively atwww.Twaller.com. The errors in search that we are seing is as follows: (1) HTTP status code: 403 Message:The request is understood, but it has been refused. An accompanying error message will explain why. (2)HTTP status code: 503 Message:Service Unavailable: The Twitter servers are up, but overloaded with requests. Try again later. The search and trend methods use this to indicate when you are being rate limited. (3) HTTP status code: -1 Message:Read timed out These errors are becoming more and more frequent lately. /Amitab On Sep 1, 5:41 pm, Dewald Pretorius dpr...@gmail.com wrote: Twitter team, can you please do something about the performance and rate limiting of the Search API. It is becoming completely unworkable. I have jumped through all the hoops, with unique User Agents, sleeping my scripts in-between API calls, and yet the rate limiting is just becoming more severe, and the performance is just becoming slower and slower. Please help. A lot of us are using Search to provide services, and I am sure it is not only my service that is hurting very much under the present circumstances. Dewald
[twitter-dev] Re: Search API Rate Limiting and Performance
John, the original message of this thread is about rate limit being totally erratic, as several users have noticed. here is the detail of what I'm seeing: http://groups.google.com/group/twitter-development-talk/browse_thread/thread/40c82b4dbc0536bd Here is another user reporting the problem : http://groups.google.com/group/twitter-development-talk/browse_thread/thread/51bd6667d3b0a0a6 Basically the X-RateLimit-Remaining header goes up and down and up again between adjacent requests (all authenticated requests). The result is that the limit can appear to be reached way before 150 requests were issued... Thanks On Sep 4, 6:30 pm, John Kalucki jkalu...@gmail.com wrote: The Search team is working on indexing latency and throughput, along with a many other things. There have been big improvements recently and more are on the way. In the mean time, if you need closer to real-time results, consider the track parameter on the Streaming API. -John Kaluckihttp://twitter.com/jkalucki Services, Twitter Inc. On Sep 4, 6:58 am, pxpilot pxpi...@gmail.com wrote: Search API will rock if it would only be reliable what we see looks to be some sort of a funky cache, a query (atom) can be missing some latest tweets and then after a while they show up, if you tweak the query you can see 'em. you ever seen this problem? also what did you do special with user agent? On Sep 2, 12:12 pm, Amitab hiamita...@gmail.com wrote: I have exchanged emails with Twitter on this and I believe they are working on it. We use search extensively atwww.Twaller.com. The errors in search that we are seing is as follows: (1) HTTP status code: 403 Message:The request is understood, but it has been refused. An accompanying error message will explain why. (2)HTTP status code: 503 Message:Service Unavailable: The Twitter servers are up, but overloaded with requests. Try again later. The search and trend methods use this to indicate when you are being rate limited. (3) HTTP status code: -1 Message:Read timed out These errors are becoming more and more frequent lately. /Amitab On Sep 1, 5:41 pm,Dewald Pretoriusdpr...@gmail.com wrote: Twitter team, can you please do something about the performance and rate limiting of the Search API. It is becoming completely unworkable. I have jumped through all the hoops, with unique User Agents, sleeping my scripts in-between API calls, and yet the rate limiting is just becoming more severe, and the performance is just becoming slower and slower. Please help. A lot of us are using Search to provide services, and I am sure it is not only my service that is hurting very much under the present circumstances. Dewald
[twitter-dev] Re: Search API Rate Limiting and Performance
Various APIs have their own rate limiting mechanisms. The www, search and streaming rate limits are all customized to their usage patterns and share little to no code and/or state. -John On Sep 4, 9:49 am, Reivax xavier.yo...@gmail.com wrote: John, the original message of this thread is about rate limit being totally erratic, as several users have noticed. here is the detail of what I'm seeing: http://groups.google.com/group/twitter-development-talk/browse_thread... Here is another user reporting the problem :http://groups.google.com/group/twitter-development-talk/browse_thread... Basically the X-RateLimit-Remaining header goes up and down and up again between adjacent requests (all authenticated requests). The result is that the limit can appear to be reached way before 150 requests were issued... Thanks On Sep 4, 6:30 pm, John Kalucki jkalu...@gmail.com wrote: The Search team is working on indexing latency and throughput, along with a many other things. There have been big improvements recently and more are on the way. In the mean time, if you need closer to real-time results, consider the track parameter on the Streaming API. -John Kaluckihttp://twitter.com/jkalucki Services, Twitter Inc. On Sep 4, 6:58 am, pxpilot pxpi...@gmail.com wrote: Search API will rock if it would only be reliable what we see looks to be some sort of a funky cache, a query (atom) can be missing some latest tweets and then after a while they show up, if you tweak the query you can see 'em. you ever seen this problem? also what did you do special with user agent? On Sep 2, 12:12 pm, Amitab hiamita...@gmail.com wrote: I have exchanged emails with Twitter on this and I believe they are working on it. We use search extensively atwww.Twaller.com. The errors in search that we are seing is as follows: (1) HTTP status code: 403 Message:The request is understood, but it has been refused. An accompanying error message will explain why. (2)HTTP status code: 503 Message:Service Unavailable: The Twitter servers are up, but overloaded with requests. Try again later. The search and trend methods use this to indicate when you are being rate limited. (3) HTTP status code: -1 Message:Read timed out These errors are becoming more and more frequent lately. /Amitab On Sep 1, 5:41 pm,Dewald Pretoriusdpr...@gmail.com wrote: Twitter team, can you please do something about the performance and rate limiting of the Search API. It is becoming completely unworkable. I have jumped through all the hoops, with unique User Agents, sleeping my scripts in-between API calls, and yet the rate limiting is just becoming more severe, and the performance is just becoming slower and slower. Please help. A lot of us are using Search to provide services, and I am sure it is not only my service that is hurting very much under the present circumstances. Dewald
[twitter-dev] Re: Search API Rate Limiting and Performance
Dewald, I'm not on the search team, but there are a lot of discussions over there this morning about search api rate limits and related issues. Search rate limiting issues (vs. www.twitter.com or api.twitter.com) probably boil down to one of three categories: 1) Search service interruptions - We're working on these, obviously. You may see odd behavior when search capacity is reduced due to some operational issue, and the system is running far too hot. 2) General API limit reductions - We're slightly less generous than we have been in the past due to persistent abuse of the search api and due to overall capacity issues. We can't always provide for all valid use cases, although that is certainly a high priority. 3) Bugs. There could be defects that prevent valid use of the API. Detailed data of limiting mishaps would be helpful here. Timestamps, queries and statistics around reasonable use that was denied may allow us to identify a previously unknown defect. The search team is quite interested in working on #3. -John Kalucki http://twitter.com/jkalucki Services, Twitter Inc. On Sep 1, 5:41 pm, Dewald Pretorius dpr...@gmail.com wrote: Twitter team, can you please do something about the performance and rate limiting of the Search API. It is becoming completely unworkable. I have jumped through all the hoops, with unique User Agents, sleeping my scripts in-between API calls, and yet the rate limiting is just becoming more severe, and the performance is just becoming slower and slower. Please help. A lot of us are using Search to provide services, and I am sure it is not only my service that is hurting very much under the present circumstances. Dewald