Hi all,
The error code for search rate limiting will be changing from HTTP
503 to HTTP 401 in the very near future (today or tomorrow). For
details, continue reading.
Why the 401 change?
The search API rate limit is something that nobody should be
hitting in an ideal world. Last week we moved from the old Summize
data center and hosting to our new home in the Twitter data center.
This increased capacity should let us relax the rate limiting
somewhat. Along with this move we've seen some issues with
applications that do hit the rate limit getting the wrong response
code or worse, a timeout. After digging through this with our
operations team last week it looks like the problem is special
handling of HTTP 503 in proxies. To fix that we're changing the rate
limit response code to HTTP 401.
What *is* the rate limit anyway?
The rate limiting code in the search system was a one-hour, middle-
of=the-night-and-under-attack project and does not have very good
management logic. I'm sad to say the logic for when you get rate
limited is a bit fuzzier than I would like as well. Moving search to
the Twitter data center, Fixing other issues (see the issues list) and
OAuth (Hooray!) have all been higher priorities than re-writing the
search rate limiting code.
If you're being rate limited by search you can email me and I'll
review your traffic and we can come up with a new limit that suits
you. Search white-listing does not give any application carte blanche,
we instead review the traffic pattern, help optimize your queries or
usage and raise the limit to something above the current usage. This
prevents a run away script in one white-listed developers code from
creating an unintentional DOS attack. I know it's unfair, but after
being bitten a few times my snake handling days are over.
When will is be fixed?
As part of the long-awaited next version of the API (which I've
begun work on) we'll be folding search into the main API and start
sharing rate limiting code. This will means we'll be able to publish
the exact logic used to rate limit, and allow you to check the status.
If you have any questions feel free to reply to this thread or email
me off-list at [EMAIL PROTECTED] If you do reply or email please be
patient, as I'm sure this will bring a metric-ton of replies and not
all of them positive.
Thanks;
— Matt Sanford (@mzsanford)
