[twitter-dev] Re: heavy throttling by search.twitter.com API from GAE application

2009-08-28 Thread BenHedrington

I agree I have been using the Twitter search APIs for more than a year
on 3 App Engine apps, this is seriously handicapping my usage of the
API.
-Ben Hedrington

On Aug 27, 1:42 am, Paul Kinlan paul.kin...@gmail.com wrote:
 Hi,
 I still think something is going on (or at least different) - I have never
 seen this level of throttling on the Google App Engine.  I am doing far less
 than 1 request a second and it is getting massively rate limited.  In the
 past I have performed searches far more frequently.

 Paul

 2009/8/26 Paul Kinlan paul.kin...@gmail.com

  Hi Chad,
  Has this limit changed recently? I used to query it far more frequently
  from the app engine.  Obviously, Google use a lot of different IP addresses
  so I presuming it can fluctuate.  But over the last couple of days I have
  noticed far more that I used to get.

  If it is by IP first what is the point of using the User-Agent (it was
  stated a little while back that we must include it now for rate limiting) -
  is it just for tracking of an application?

  Paul

  2009/8/26 Chad Etzel c...@twitter.com

  Hi Paul,

  If you are sharing your IP with any other GAE twitter apps that are
  also doing search, then you are sharing the resource at that point.
  The limiting is by IP first, then user-agent. Also, 1 search per
  second is on the borderline of the normal rate-limit anyway, so I
  would try calling less frequently if possible.

  -Chad

  On Wed, Aug 26, 2009 at 12:29 PM, Paul Kinlanpaul.kin...@gmail.com
  wrote:
   Hi,
   Just a question, I am starting to see very heavy throttling to the
  Twitter
   Search API from the Google App engine.
   I am receiving 503's enhance your calm very frequently.  I have a custom
  set
   User-Agent string and I am probably doing less than 1 search per second.
   It has been happening for a couple of days now.  Has there been a recent
   change to cause this behaviour.
   Paul.


[twitter-dev] Re: heavy throttling by search.twitter.com API from GAE application

2009-08-21 Thread BenHedrington

I agree GAE throttle on the Search API is not behaving as it has in
the past, Can someone please look into this?
-Ben Hedrington

On Aug 21, 11:48 am, Jud jvale...@gmail.com wrote:
 I've got a python app running on Google App Engine (appspot hosted)
 that querieshttp://search.twitter.com/search.atom?q=for simple
 queries (e.g. foo OR bar), and it's being severely throttled (e.g.
 can't get a successful request through (response 200 w/ data) more
 than a couple of times per _hour_).

 - I'm setting the UA string to something unique/identifiable (e.g. my
 company name)
 - I'm respecting the retry-after header coming back when I see a 503
 (average retry-after duration is ~750)
 - GAE turns the IP address behind the app over ~ every 6 hours
 - app hits tries to hit search.twitter.com every 5 minutes.

 I've successfully polled the endpoint at much higher rates (in
 completely different IP address ranges) in the past, without issue.
 Unclear what's going on. Any suggestions would be greatly appreciated.