Just wanted to report that we are back up and running for the most part as well, BUT quite a number of our servers are still experiencing some BlackOut periods where twitter fails to respond and connections time out. They seem to last about 5-10 minutes each. We are running quite a few servers during this time to mitigate the issue propagating to our users, but it is a costly proposition to continue running this many servers for extended periods of time.

Here is a traceroute to twitter.com on one of the servers during the "Blackout" when it is not getting any response from twitter

traceroute to twitter.com (, 30 hops max, 40 byte packets 1 67-207-128-2.slicehost.net ( 0.000 ms 0.000 ms 0.000 ms 2 209-20-79-2.slicehost.net ( 0.000 ms 0.000 ms 0.000 ms 3 ge-6-10-193.car1.StLouis1.Level3.net ( 0.000 ms 0.000 ms 0.000 ms 4 ae-11-11.car2.StLouis1.Level3.net ( 0.000 ms 0.000 ms 0.000 ms 5 ae-4-4.ebr2.Chicago1.Level3.net ( 7.999 ms 7.999 ms 7.999 ms 6 ae-2-54.edge3.Chicago3.Level3.net ( 7.999 ms ae-2-52.edge3.Chicago3.Level3.net ( 7.999 ms ae-2-54.edge3.Chicago3.Level3.net ( 7.999 ms
 7 (  7.999 ms  7.999 ms  8.000 ms
8 ae-1.r21.chcgil09.us.bb.gin.ntt.net ( 8.000 ms 8.000 ms 8.000 ms 9 as-5.r20.snjsca04.us.bb.gin.ntt.net ( 51.996 ms 51.996 ms 51.996 ms 10 xe-1-3.r02.mlpsca01.us.bb.gin.ntt.net ( 55.995 ms 55.995 ms 55.995 ms 11 mg-1.c00.mlpsca01.us.da.verio.net ( 55.995 ms 55.995 ms 59.995 ms
12 (  55.995 ms  51.995 ms  51.995 ms
13 (  51.996 ms !X * *

On Aug 10, 2009, at 2:57 PM, Ryan Sarver wrote:

Wanted to send out a status update and let everyone know where the situation stands as of today at noon.

- Most developers are reporting being back in operation as of noon on Sunday - We have changed our defenses to make sure API developers are better supported. As such the system has more general strain on it and thus will produce some more 502/503 errors. If you see them, you should do a geometric back off instead of just sending a new request.
- OAuth should be fully operational
- If you continue to have unexpected errors, please produce a packet trace so we can help debug and define the issue.

I will continue to give periodic updates throughout the day as we know more, but as most apps are back in action the updates will be more based on new news. Please continue to let us know of any unexpected issues you may have.

Thanks again for your continued patience and support.

Best, Ryan

Reply via email to