Chris ,
We implemented something like this "network status" using the
rate_limit_status call (for the IP), while some of the numbers are
sometimes wonky with this api right now we poll this every 5 minutes
and set a flag to enable or disable all twitter requests from the
server depending on what is returned. This has the effect of backing
off twitter when we get limited, as well as automatically bring
twitter service up when the blackout period subsides.. Even with the
wonky numbers, we blindly accept what the call returns and disable
when api credits say they are 0 and if the call times out you know for
sure its time to back off. ( I know these criteria are overly
aggressive, but we figure its better to aggressively back off in the
hopes it helps things recover quicker)
When the server is in blackout, simply report that twitter is still
having ongoing issues to the user with a note and a link to the
twitter status page, keeps them informed as to what is going on
without flooding your inbox or you having to respond to all the users
emails, If the message in the UI saves you from having to read through
and answer 10 emails, it is worth the work to implement. Its all about
keeping the user informed, just like we like to be informed by twitter
about what the status is.
Another tip is to make sure you set a resonable timeout for all your
twitter requests or people get stuck waiting for your page to load and
think its broken, better to timeout and again report that twitter is
still recovering for DDoS attack and direct them to the twitter status
page..
While this may seem like unnecessary work if twitter is going to get
back to normal, it makes your app more robust to issues in the future,
keeps your users better informed as to what is happening (without
contacting you), and also helps reduce number of requests to twitter
that they have to ignore while they are recovering
On that note, it seems our servers are disabling significantly less
frequently over the past few hours. Heres hoping it lasts.. Not sure
if it was the back off or if Twitter has just managed to cope better
with the traffic issues.. Doesn't really matter to me which..
Oh and thanks for the update Chad, it really is nice just to hear
something, even if its not great news.
On Aug 8, 2009, at 4:19 AM, Chris Babcock wrote:
> On Fri, 7 Aug 2009 11:05:32 -0700
> Ryan Sarver <rsar...@twitter.com> wrote:
>
>> I wanted to send everyone an update to let you know what has been
>> happening, the known issues, some suggestions on how to resolve them
>> and some idea of how to move forward.
>
> This was really appreciated. When the dust clears, maybe one more
> suggestion? An API to check on the network status? I think
> infrastucture attacks aren't going away anytime soon. We've got a
> diversity of applications here, some of which can chew up the
> bandwidth
> pretty well and some of those just don't make sense to run if other
> users can't get on-line. Instead of answering, "Is it OK to restart my
> cron jobs?" the cron jobs could shut themselves down for increments of
> so many hours.
>
> Chris Babcock
>
> PS - Of course it could be misused, but I think the benefit is net.
