Ryan, These 502s happen in my high-volume processes. I can't manually reproduce them.
Most calls don't 502 after the second geometric back-off. I'm guessing it's just everyone doing a 9-hour catch-up against the API. Dewald On Oct 18, 4:54 pm, Ryan Sarver <[email protected]> wrote: > Dewald, > > Can you produce some TCP dumps and requests with headers so we can better > debug? > > Thanks, Ryan > > On Sun, Oct 18, 2009 at 12:05 PM, Dewald Pretorius <[email protected]> wrote: > > > I'm seeing lots of 502's now. Is the API overloaded? > > > Dewald > > > On Oct 18, 2:58 pm, Ryan Sarver <[email protected]> wrote: > >> Michael, > > >> We are still working on getting the full picture, but once we have the > >> details I will report to the group what the issue was. > > >> Thanks for updating us. > > >> Best, Ryan > > >> On Sun, Oct 18, 2009 at 10:55 AM, Michael Steuer <[email protected]> wrote: > >> > It is working for me. Would you mind sharing with the group what exactly > >> > happened? > >> > Thanks. > >> > Michael > > >> > On Oct 18, 2009, at 10:53 AM, Atul Kulkarni <[email protected]> > >> > wrote: > > >> > Works Fine... Duluth, MN. > > >> > On Sun, Oct 18, 2009 at 12:51 PM, Ryan Sarver <[email protected]> > >> > wrote: > > >> >> I wanted to check in and see if everyone is back to normal? We think > >> >> things have been fixed but its hard to confirm without your help. > > >> >> Let me know if you are still experiencing any issues and if so, where > >> >> you are located. > > >> >> Best, Ryan > > >> >> On Sun, Oct 18, 2009 at 10:48 AM, Dewald Pretorius <[email protected]> > >> >> wrote: > > >> >> > Same here. Can connect again. > > >> >> > On Oct 18, 2:46 pm, Michael Steuer <[email protected]> wrote: > >> >> >> The situation seems to have been resolved, at least for me, as of a > >> >> >> few minutes ago. My Rackspace hosted servers can reach the API > >> >> >> again... > > >> >> >> On Oct 18, 2009, at 10:35 AM, Dewald Pretorius <[email protected]> > >> >> >> wrote: > > >> >> >> > I don't really blame Twitter Ops for not knowing. It's probably a > >> >> >> > new > >> >> >> > edge defense that was installed by their service provider during > >> >> >> > Sunday night. > > >> >> >> > However, a while ago Alex said the Platform team were working on an > >> >> >> > external monitoring solution. Hopefully Ryan, who is now Director > >> >> >> > of > >> >> >> > the Platform team, will quickly move this forward to completion. > > >> >> >> > Dewald > > >> >> >> > On Oct 18, 1:30 pm, Michael Steuer <[email protected]> wrote: > >> >> >> >> Amen to that. I find it kind of curious that as per John K., 5-6 > >> >> >> >> hours > >> >> >> >> into this issue, the Twitter ops team was still blissfully unaware > >> >> >> >> of > >> >> >> >> anything going on... Also weird that they apparently are unable to > >> >> >> >> reproduce the issue without our help, ie. they really haven't set > >> >> >> >> up > >> >> >> >> any monitoring outside of their network... > > >> >> >> >> On Oct 18, 2009, at 9:05 AM, Dewald Pretorius <[email protected]> > >> >> >> >> wrote: > > >> >> >> >>> I'd be more than happy to wait longer for snazzy API 2.0 features > >> >> >> >>> so > >> >> >> >>> that the Platform team can build a QoS system that monitors the > >> >> >> >>> API's > >> >> >> >>> availability and performance from the outside. That will enable > >> >> >> >>> Twitter to catch these kinds of issues long before we do. > > >> >> >> >>> Dewald > > >> >> >> >>> On Oct 18, 12:47 pm, Michael Steuer <[email protected]> wrote: > >> >> >> >>>> This outage is now going on 7 hours. Any word from Twitter as to > >> >> >> >>>> an > >> >> >> >>>> ETA for resolution? > > >> >> >> >>>> On Oct 18, 2009, at 8:08 AM, John Meyer <[email protected]> > >> >> >> >>>> wrote: > > >> >> >> >>>>> John Kalucki wrote: > >> >> >> >>>>>> And here's the next question: > > >> >> >> >>>>>> Is anyone having trouble from non-service, non-hosted > >> >> >> >>>>>> endpoints. In > >> >> >> >>>>>> other words, problem from home ISPs and desktop clients? > > >> >> >> >>>>>> -John Kalucki > >> >> >> >>>>>>http://twitter.com/jkalucki > >> >> >> >>>>>> Services, Twitter Inc. > > >> >> >> >>>>> Yep. (comcast, cannot access through either the website or > >> >> >> >>>>> desktop > >> >> >> >>>>> clients). > > >> > -- > >> > Regards, > >> > Atul Kulkarni > >> >www.d.umn.edu/~kulka053
