[twitter-dev] Re: Search API Rate Limiting and Performance

2009-09-04 Thread pxpilot

Search API will rock if it would only be reliable

what we see looks to be some sort of a funky cache,  a query (atom)
can be missing some latest tweets and then after a while they show up,
if you tweak the query you can see 'em.

you ever seen this problem?
also what did you do special with user agent?

On Sep 2, 12:12 pm, Amitab hiamita...@gmail.com wrote:
 I have exchanged emails with Twitter on this and I believe they are
 working on it.

 We use search extensively atwww.Twaller.com. The errors in search
 that we are seing is as follows:

 (1) HTTP status code: 403
      Message:The request is understood, but it has been refused.  An
 accompanying error message will explain why.

 (2)HTTP status code: 503
 Message:Service Unavailable: The Twitter servers are up, but
 overloaded with requests. Try again later. The search and trend
 methods use this to indicate when you are being rate limited.

 (3) HTTP status code: -1
 Message:Read timed out

 These errors are becoming more and more frequent lately.

 /Amitab

 On Sep 1, 5:41 pm, Dewald Pretorius dpr...@gmail.com wrote:

  Twitter team, can you please do something about the performance and
  rate limiting of the Search API.

  It is becoming completely unworkable. I have jumped through all the
  hoops, with unique User Agents, sleeping my scripts in-between API
  calls, and yet the rate limiting is just becoming more severe, and the
  performance is just becoming slower and slower.

  Please help. A lot of us are using Search to provide services, and I
  am sure it is not only my service that is hurting very much under the
  present circumstances.

  Dewald


[twitter-dev] Re: Search API Rate Limiting and Performance

2009-09-04 Thread John Kalucki

The Search team is working on indexing latency and throughput, along
with a many other things. There have been big improvements recently
and more are on the way.

In the mean time, if you need closer to real-time results, consider
the track parameter on the Streaming API.

-John Kalucki
http://twitter.com/jkalucki
Services, Twitter Inc.



On Sep 4, 6:58 am, pxpilot pxpi...@gmail.com wrote:
 Search API will rock if it would only be reliable

 what we see looks to be some sort of a funky cache,  a query (atom)
 can be missing some latest tweets and then after a while they show up,
 if you tweak the query you can see 'em.

 you ever seen this problem?
 also what did you do special with user agent?

 On Sep 2, 12:12 pm, Amitab hiamita...@gmail.com wrote:

  I have exchanged emails with Twitter on this and I believe they are
  working on it.

  We use search extensively atwww.Twaller.com. The errors in search
  that we are seing is as follows:

  (1) HTTP status code: 403
       Message:The request is understood, but it has been refused.  An
  accompanying error message will explain why.

  (2)HTTP status code: 503
  Message:Service Unavailable: The Twitter servers are up, but
  overloaded with requests. Try again later. The search and trend
  methods use this to indicate when you are being rate limited.

  (3) HTTP status code: -1
  Message:Read timed out

  These errors are becoming more and more frequent lately.

  /Amitab

  On Sep 1, 5:41 pm, Dewald Pretorius dpr...@gmail.com wrote:

   Twitter team, can you please do something about the performance and
   rate limiting of the Search API.

   It is becoming completely unworkable. I have jumped through all the
   hoops, with unique User Agents, sleeping my scripts in-between API
   calls, and yet the rate limiting is just becoming more severe, and the
   performance is just becoming slower and slower.

   Please help. A lot of us are using Search to provide services, and I
   am sure it is not only my service that is hurting very much under the
   present circumstances.

   Dewald


[twitter-dev] Re: Search API Rate Limiting and Performance

2009-09-04 Thread Reivax

John, the original message of this thread is about rate limit being
totally erratic, as several users have noticed. here is the detail of
what I'm seeing:

http://groups.google.com/group/twitter-development-talk/browse_thread/thread/40c82b4dbc0536bd

Here is another user reporting the problem :
http://groups.google.com/group/twitter-development-talk/browse_thread/thread/51bd6667d3b0a0a6


Basically the X-RateLimit-Remaining header goes up and down and up
again between adjacent requests (all authenticated requests).

The result is that the limit can appear to be reached way before 150
requests were issued...

Thanks

On Sep 4, 6:30 pm, John Kalucki jkalu...@gmail.com wrote:
 The Search team is working on indexing latency and throughput, along
 with a many other things. There have been big improvements recently
 and more are on the way.

 In the mean time, if you need closer to real-time results, consider
 the track parameter on the Streaming API.

 -John Kaluckihttp://twitter.com/jkalucki
 Services, Twitter Inc.

 On Sep 4, 6:58 am, pxpilot pxpi...@gmail.com wrote:

  Search API will rock if it would only be reliable

  what we see looks to be some sort of a funky cache,  a query (atom)
  can be missing some latest tweets and then after a while they show up,
  if you tweak the query you can see 'em.

  you ever seen this problem?
  also what did you do special with user agent?

  On Sep 2, 12:12 pm, Amitab hiamita...@gmail.com wrote:

   I have exchanged emails with Twitter on this and I believe they are
   working on it.

   We use search extensively atwww.Twaller.com. The errors in search
   that we are seing is as follows:

   (1) HTTP status code: 403
        Message:The request is understood, but it has been refused.  An
   accompanying error message will explain why.

   (2)HTTP status code: 503
   Message:Service Unavailable: The Twitter servers are up, but
   overloaded with requests. Try again later. The search and trend
   methods use this to indicate when you are being rate limited.

   (3) HTTP status code: -1
   Message:Read timed out

   These errors are becoming more and more frequent lately.

   /Amitab

   On Sep 1, 5:41 pm,Dewald Pretoriusdpr...@gmail.com wrote:

Twitter team, can you please do something about the performance and
rate limiting of the Search API.

It is becoming completely unworkable. I have jumped through all the
hoops, with unique User Agents, sleeping my scripts in-between API
calls, and yet the rate limiting is just becoming more severe, and the
performance is just becoming slower and slower.

Please help. A lot of us are using Search to provide services, and I
am sure it is not only my service that is hurting very much under the
present circumstances.

Dewald


[twitter-dev] Re: Search API Rate Limiting and Performance

2009-09-04 Thread John Kalucki

Various APIs have their own rate limiting mechanisms. The www, search
and streaming rate limits are all customized to their usage patterns
and share little to no code and/or state.

-John


On Sep 4, 9:49 am, Reivax xavier.yo...@gmail.com wrote:
 John, the original message of this thread is about rate limit being
 totally erratic, as several users have noticed. here is the detail of
 what I'm seeing:

 http://groups.google.com/group/twitter-development-talk/browse_thread...

 Here is another user reporting the problem 
 :http://groups.google.com/group/twitter-development-talk/browse_thread...

 Basically the X-RateLimit-Remaining header goes up and down and up
 again between adjacent requests (all authenticated requests).

 The result is that the limit can appear to be reached way before 150
 requests were issued...

 Thanks

 On Sep 4, 6:30 pm, John Kalucki jkalu...@gmail.com wrote:

  The Search team is working on indexing latency and throughput, along
  with a many other things. There have been big improvements recently
  and more are on the way.

  In the mean time, if you need closer to real-time results, consider
  the track parameter on the Streaming API.

  -John Kaluckihttp://twitter.com/jkalucki
  Services, Twitter Inc.

  On Sep 4, 6:58 am, pxpilot pxpi...@gmail.com wrote:

   Search API will rock if it would only be reliable

   what we see looks to be some sort of a funky cache,  a query (atom)
   can be missing some latest tweets and then after a while they show up,
   if you tweak the query you can see 'em.

   you ever seen this problem?
   also what did you do special with user agent?

   On Sep 2, 12:12 pm, Amitab hiamita...@gmail.com wrote:

I have exchanged emails with Twitter on this and I believe they are
working on it.

We use search extensively atwww.Twaller.com. The errors in search
that we are seing is as follows:

(1) HTTP status code: 403
     Message:The request is understood, but it has been refused.  An
accompanying error message will explain why.

(2)HTTP status code: 503
Message:Service Unavailable: The Twitter servers are up, but
overloaded with requests. Try again later. The search and trend
methods use this to indicate when you are being rate limited.

(3) HTTP status code: -1
Message:Read timed out

These errors are becoming more and more frequent lately.

/Amitab

On Sep 1, 5:41 pm,Dewald Pretoriusdpr...@gmail.com wrote:

 Twitter team, can you please do something about the performance and
 rate limiting of the Search API.

 It is becoming completely unworkable. I have jumped through all the
 hoops, with unique User Agents, sleeping my scripts in-between API
 calls, and yet the rate limiting is just becoming more severe, and the
 performance is just becoming slower and slower.

 Please help. A lot of us are using Search to provide services, and I
 am sure it is not only my service that is hurting very much under the
 present circumstances.

 Dewald


[twitter-dev] Re: Search API Rate Limiting and Performance

2009-09-04 Thread John Kalucki

Dewald,

I'm not on the search team, but there are a lot of discussions over
there this morning about search api rate limits and related issues.
Search rate limiting issues (vs. www.twitter.com or api.twitter.com)
probably boil down to one of three categories:

1) Search service interruptions - We're working on these, obviously.
You may see odd behavior when search capacity is reduced due to some
operational issue, and the system is running far too hot.

2) General API limit reductions - We're slightly less generous than we
have been in the past due to persistent abuse of the search api and
due to overall capacity issues. We can't always provide for all valid
use cases, although that is certainly a high priority.

3) Bugs. There could be defects that prevent valid use of the API.
Detailed data of limiting mishaps would be helpful here. Timestamps,
queries and statistics around reasonable use that was denied may allow
us to identify a previously unknown defect.

The search team is quite interested in working on #3.

-John Kalucki
http://twitter.com/jkalucki
Services, Twitter Inc.


On Sep 1, 5:41 pm, Dewald Pretorius dpr...@gmail.com wrote:
 Twitter team, can you please do something about the performance and
 rate limiting of the Search API.

 It is becoming completely unworkable. I have jumped through all the
 hoops, with unique User Agents, sleeping my scripts in-between API
 calls, and yet the rate limiting is just becoming more severe, and the
 performance is just becoming slower and slower.

 Please help. A lot of us are using Search to provide services, and I
 am sure it is not only my service that is hurting very much under the
 present circumstances.

 Dewald