[twitter-dev] Re: Timeouts and API Errors, Tuesday August 11th
For me the API behavior is now back again to this morning, with around 2 to 5 rejections per second, and most being successful on the first or second backed-off retry. Dewald On Aug 11, 6:22 pm, Alex Payne wrote: > Just found out that our hosting provider put some hardware in place that may > cause disruptions. Our operations team just spoke them, and they should be > taking it down in 15 - 30 minutes. > > > > On Tue, Aug 11, 2009 at 14:03, Andrew Badera wrote: > > > On Tue, Aug 11, 2009 at 5:01 PM, Andrew Badera wrote: > > > On Tue, Aug 11, 2009 at 4:57 PM, Alex Payne wrote: > > >> Yes, I've just been informed that the attack has resumed, and that our > > >> service provider is putting network hardware in place to counter the > > attack. > > >> We're trying to work with them to ensure minimal impact to the API, but > > in > > >> the near term there may be issues with OAuth and the Streaming API. > > >> This is a bit of a juggling act, as we're trying to coordinate our team, > > the > > >> operations team, our service provider's staff, and specialists that > > they've > > >> brought in for this issue. Please bear with us. > > > > Thanks for the update Alex. No worries here, you all seem to be > > > keeping us much more comfortably updated than most of Friday. > > > > --ab > > > *Thursday, not Friday > > -- > Alex Payne - Platform Lead, Twitter, Inc.http://twitter.com/al3x
[twitter-dev] Re: Timeouts and API Errors, Tuesday August 11th
Just found out that our hosting provider put some hardware in place that may cause disruptions. Our operations team just spoke them, and they should be taking it down in 15 - 30 minutes. On Tue, Aug 11, 2009 at 14:03, Andrew Badera wrote: > > On Tue, Aug 11, 2009 at 5:01 PM, Andrew Badera wrote: > > On Tue, Aug 11, 2009 at 4:57 PM, Alex Payne wrote: > >> Yes, I've just been informed that the attack has resumed, and that our > >> service provider is putting network hardware in place to counter the > attack. > >> We're trying to work with them to ensure minimal impact to the API, but > in > >> the near term there may be issues with OAuth and the Streaming API. > >> This is a bit of a juggling act, as we're trying to coordinate our team, > the > >> operations team, our service provider's staff, and specialists that > they've > >> brought in for this issue. Please bear with us. > >> > > > > > > Thanks for the update Alex. No worries here, you all seem to be > > keeping us much more comfortably updated than most of Friday. > > > > --ab > > > > > *Thursday, not Friday > -- Alex Payne - Platform Lead, Twitter, Inc. http://twitter.com/al3x
[twitter-dev] Re: Timeouts and API Errors, Tuesday August 11th
On Tue, Aug 11, 2009 at 5:01 PM, Andrew Badera wrote: > On Tue, Aug 11, 2009 at 4:57 PM, Alex Payne wrote: >> Yes, I've just been informed that the attack has resumed, and that our >> service provider is putting network hardware in place to counter the attack. >> We're trying to work with them to ensure minimal impact to the API, but in >> the near term there may be issues with OAuth and the Streaming API. >> This is a bit of a juggling act, as we're trying to coordinate our team, the >> operations team, our service provider's staff, and specialists that they've >> brought in for this issue. Please bear with us. >> > > > Thanks for the update Alex. No worries here, you all seem to be > keeping us much more comfortably updated than most of Friday. > > --ab > *Thursday, not Friday
[twitter-dev] Re: Timeouts and API Errors, Tuesday August 11th
On Tue, Aug 11, 2009 at 4:57 PM, Alex Payne wrote: > Yes, I've just been informed that the attack has resumed, and that our > service provider is putting network hardware in place to counter the attack. > We're trying to work with them to ensure minimal impact to the API, but in > the near term there may be issues with OAuth and the Streaming API. > This is a bit of a juggling act, as we're trying to coordinate our team, the > operations team, our service provider's staff, and specialists that they've > brought in for this issue. Please bear with us. > Thanks for the update Alex. No worries here, you all seem to be keeping us much more comfortably updated than most of Friday. --ab
[twitter-dev] Re: Timeouts and API Errors, Tuesday August 11th
Also, please be sure to provide packet dumps and full headers where possible so we can more easily determine the source of the issue. Thanks, Ryan On Tue, Aug 11, 2009 at 2:00 PM, Alex Payne wrote: > We're aware of these issues; sorry. > Our ops team tells me that the countermeasures that are being put in place > should not cause the 302 redirect behavior that impacted OAuth and other > services late last week. If you're seeing that behavior, please post here > and we'll coordinate with them to eliminate it. > > On Tue, Aug 11, 2009 at 13:58, Sean Callahan wrote: >> >> Alex, >> >> Did not see this post and posted a new message. Still receiving lots >> of errors and no one can login on our site, tweetphoto.com, right now >> along with a handful of others (that I've tried myself). Just wanted >> to give you a heads up. Thanks! >> >> Sean >> >> On Aug 11, 1:11 pm, Alex Payne wrote: >> > Our operations staff has informed me that the attack ceased several >> > minutes >> > ago. Site performance should be returning to normal. >> > >> > On Tue, Aug 11, 2009 at 12:23, Alex Payne wrote: >> > > We're currently experiencing another wave of Distributed Denial of >> > > Service >> > > (DDoS) attacks against our system. Expect periodic slowness and errors >> > > until >> > > the attack passes or is countered by our operations team and hosting >> > > provider. Updates will be provided as we get them. >> > > Thanks for your patience. >> > >> > > -- >> > > Alex Payne - Platform Lead, Twitter, Inc. >> > >http://twitter.com/al3x >> > >> > -- >> > Alex Payne - Platform Lead, Twitter, Inc.http://twitter.com/al3x > > > > -- > Alex Payne - Platform Lead, Twitter, Inc. > http://twitter.com/al3x >
[twitter-dev] Re: Timeouts and API Errors, Tuesday August 11th
We're aware of these issues; sorry. Our ops team tells me that the countermeasures that are being put in place should not cause the 302 redirect behavior that impacted OAuth and other services late last week. If you're seeing that behavior, please post here and we'll coordinate with them to eliminate it. On Tue, Aug 11, 2009 at 13:58, Sean Callahan wrote: > > Alex, > > Did not see this post and posted a new message. Still receiving lots > of errors and no one can login on our site, tweetphoto.com, right now > along with a handful of others (that I've tried myself). Just wanted > to give you a heads up. Thanks! > > Sean > > On Aug 11, 1:11 pm, Alex Payne wrote: > > Our operations staff has informed me that the attack ceased several > minutes > > ago. Site performance should be returning to normal. > > > > On Tue, Aug 11, 2009 at 12:23, Alex Payne wrote: > > > We're currently experiencing another wave of Distributed Denial of > Service > > > (DDoS) attacks against our system. Expect periodic slowness and errors > until > > > the attack passes or is countered by our operations team and hosting > > > provider. Updates will be provided as we get them. > > > Thanks for your patience. > > > > > -- > > > Alex Payne - Platform Lead, Twitter, Inc. > > >http://twitter.com/al3x > > > > -- > > Alex Payne - Platform Lead, Twitter, Inc.http://twitter.com/al3x > -- Alex Payne - Platform Lead, Twitter, Inc. http://twitter.com/al3x
[twitter-dev] Re: Timeouts and API Errors, Tuesday August 11th
Alex, Did not see this post and posted a new message. Still receiving lots of errors and no one can login on our site, tweetphoto.com, right now along with a handful of others (that I've tried myself). Just wanted to give you a heads up. Thanks! Sean On Aug 11, 1:11 pm, Alex Payne wrote: > Our operations staff has informed me that the attack ceased several minutes > ago. Site performance should be returning to normal. > > On Tue, Aug 11, 2009 at 12:23, Alex Payne wrote: > > We're currently experiencing another wave of Distributed Denial of Service > > (DDoS) attacks against our system. Expect periodic slowness and errors until > > the attack passes or is countered by our operations team and hosting > > provider. Updates will be provided as we get them. > > Thanks for your patience. > > > -- > > Alex Payne - Platform Lead, Twitter, Inc. > >http://twitter.com/al3x > > -- > Alex Payne - Platform Lead, Twitter, Inc.http://twitter.com/al3x
[twitter-dev] Re: Timeouts and API Errors, Tuesday August 11th
Yes, I've just been informed that the attack has resumed, and that our service provider is putting network hardware in place to counter the attack. We're trying to work with them to ensure minimal impact to the API, but in the near term there may be issues with OAuth and the Streaming API. This is a bit of a juggling act, as we're trying to coordinate our team, the operations team, our service provider's staff, and specialists that they've brought in for this issue. Please bear with us. On Tue, Aug 11, 2009 at 13:54, Dewald Pretorius wrote: > > My guess is it's still ongoing. I'm seeing far more rejections per > second, and the number of backed-off retries have also increased. > > Dewald > > On Aug 11, 5:37 pm, Andrew Badera wrote: > > On Tue, Aug 11, 2009 at 4:11 PM, Alex Payne wrote: > > > Our operations staff has informed me that the attack ceased several > minutes > > > ago. Site performance should be returning to normal. > > > > > On Tue, Aug 11, 2009 at 12:23, Alex Payne wrote: > > > > >> We're currently experiencing another wave of Distributed Denial of > Service > > >> (DDoS) attacks against our system. Expect periodic slowness and errors > until > > >> the attack passes or is countered by our operations team and hosting > > >> provider. Updates will be provided as we get them. > > >> Thanks for your patience. > > > > >> -- > > >> Alex Payne - Platform Lead, Twitter, Inc. > > >>http://twitter.com/al3x > > > > Has it resumed? Still getting lots of intermittency here. > > > > ∞ Andy Badera > > ∞ This email is: [ ] bloggable [x] ask first [ ] private > > ∞ Google me: > http://www.google.com/search?q=(andrew+badera)+OR+(andy+badera) > -- Alex Payne - Platform Lead, Twitter, Inc. http://twitter.com/al3x
[twitter-dev] Re: Timeouts and API Errors, Tuesday August 11th
My guess is it's still ongoing. I'm seeing far more rejections per second, and the number of backed-off retries have also increased. Dewald On Aug 11, 5:37 pm, Andrew Badera wrote: > On Tue, Aug 11, 2009 at 4:11 PM, Alex Payne wrote: > > Our operations staff has informed me that the attack ceased several minutes > > ago. Site performance should be returning to normal. > > > On Tue, Aug 11, 2009 at 12:23, Alex Payne wrote: > > >> We're currently experiencing another wave of Distributed Denial of Service > >> (DDoS) attacks against our system. Expect periodic slowness and errors > >> until > >> the attack passes or is countered by our operations team and hosting > >> provider. Updates will be provided as we get them. > >> Thanks for your patience. > > >> -- > >> Alex Payne - Platform Lead, Twitter, Inc. > >>http://twitter.com/al3x > > Has it resumed? Still getting lots of intermittency here. > > ∞ Andy Badera > ∞ This email is: [ ] bloggable [x] ask first [ ] private > ∞ Google me:http://www.google.com/search?q=(andrew+badera)+OR+(andy+badera)
[twitter-dev] Re: Timeouts and API Errors, Tuesday August 11th
On Tue, Aug 11, 2009 at 4:11 PM, Alex Payne wrote: > Our operations staff has informed me that the attack ceased several minutes > ago. Site performance should be returning to normal. > > On Tue, Aug 11, 2009 at 12:23, Alex Payne wrote: >> >> We're currently experiencing another wave of Distributed Denial of Service >> (DDoS) attacks against our system. Expect periodic slowness and errors until >> the attack passes or is countered by our operations team and hosting >> provider. Updates will be provided as we get them. >> Thanks for your patience. >> >> -- >> Alex Payne - Platform Lead, Twitter, Inc. >> http://twitter.com/al3x Has it resumed? Still getting lots of intermittency here. ∞ Andy Badera ∞ This email is: [ ] bloggable [x] ask first [ ] private ∞ Google me: http://www.google.com/search?q=(andrew+badera)+OR+(andy+badera)
[twitter-dev] Re: Timeouts and API Errors, Tuesday August 11th
I'm so happy gmail has a star feature, that deserved one. Tim. On Wed, Aug 12, 2009 at 7:28 AM, Dewald Pretorius wrote: > > I have a spare bazooka in my basement. Let me know. I can FedEx it to > you. > > Dewald > > On Aug 11, 4:23 pm, Alex Payne wrote: >> We're currently experiencing another wave of Distributed Denial of Service >> (DDoS) attacks against our system. Expect periodic slowness and errors until >> the attack passes or is countered by our operations team and hosting >> provider. Updates will be provided as we get them. >> Thanks for your patience. >> >> -- >> Alex Payne - Platform Lead, Twitter, Inc.http://twitter.com/al3x
[twitter-dev] Re: Timeouts and API Errors, Tuesday August 11th
Our operations staff has informed me that the attack ceased several minutes ago. Site performance should be returning to normal. On Tue, Aug 11, 2009 at 12:23, Alex Payne wrote: > We're currently experiencing another wave of Distributed Denial of Service > (DDoS) attacks against our system. Expect periodic slowness and errors until > the attack passes or is countered by our operations team and hosting > provider. Updates will be provided as we get them. > Thanks for your patience. > > -- > Alex Payne - Platform Lead, Twitter, Inc. > http://twitter.com/al3x > -- Alex Payne - Platform Lead, Twitter, Inc. http://twitter.com/al3x
[twitter-dev] Re: Timeouts and API Errors, Tuesday August 11th
I have a spare bazooka in my basement. Let me know. I can FedEx it to you. Dewald On Aug 11, 4:23 pm, Alex Payne wrote: > We're currently experiencing another wave of Distributed Denial of Service > (DDoS) attacks against our system. Expect periodic slowness and errors until > the attack passes or is countered by our operations team and hosting > provider. Updates will be provided as we get them. > Thanks for your patience. > > -- > Alex Payne - Platform Lead, Twitter, Inc.http://twitter.com/al3x
