Setting up external monitoring isn't a huge undertaking. Their are plenty of existing services out there already. Even free ones like . Perhaps Twitter should set up a couple of those until they have their full QoS implementation in place. Pingdom for example would have started sending them emails and SMS messages at 2am pst this morning, when the issue started. They could have then escalated to their ISP and this whole thing wouldn't have lasted for close to 9 hours.

On Oct 18, 2009, at 10:35 AM, Dewald Pretorius <> wrote:

I don't really blame Twitter Ops for not knowing. It's probably a new
edge defense that was installed by their service provider during
Sunday night.

However, a while ago Alex said the Platform team were working on an
external monitoring solution. Hopefully Ryan, who is now Director of
the Platform team, will quickly move this forward to completion.


On Oct 18, 1:30 pm, Michael Steuer <> wrote:
Amen to that. I find it kind of curious that as per John K., 5-6 hours
into this issue, the Twitter ops team was still blissfully unaware of
anything going on... Also weird that they apparently are unable to
reproduce the issue without our help, ie. they really haven't set up
any monitoring outside of their network...

On Oct 18, 2009, at 9:05 AM, Dewald Pretorius <> wrote:

I'd be more than happy to wait longer for snazzy API 2.0 features so
that the Platform team can build a QoS system that monitors the API's
availability and performance from the outside. That will enable
Twitter to catch these kinds of issues long before we do.


On Oct 18, 12:47 pm, Michael Steuer <> wrote:
This outage is now going on 7 hours. Any word from Twitter as to an
ETA for resolution?

On Oct 18, 2009, at 8:08 AM, John Meyer <>

John Kalucki wrote:
And here's the next question:

Is anyone having trouble from non-service, non-hosted endpoints. In
other words, problem from home ISPs and desktop clients?

-John Kalucki
Services, Twitter Inc.

Yep. (comcast, cannot access through either the website or desktop

Reply via email to