Amazon diagnosis

2011-10-10 Thread Kristen Eisenberg
Sure they can, but as a thought exercise fully 2n redundancy is difficult on a small scale for anything web facing. I've seen a very simple implementation for a website requiring 5 9's that consumed over $50k in equipment, and this wasn't even geographically diverse. I have to believe that

Re: Amazon diagnosis

2011-05-06 Thread Ryan Malayter
On May 5, 3:51 pm, Jay Ashworth j...@baylink.com wrote: - Original Message - From: Ryan Malayter malay...@gmail.com I like to bag on my developers for not knowing anything about the infrastructure, but sometimes you just can't do it right because of physics. Or you can't do it

Re: Amazon diagnosis

2011-05-06 Thread Jay Ashworth
- Original Message - From: Ryan Malayter malay...@gmail.com On May 5, 3:51 pm, Jay Ashworth j...@baylink.com wrote: - Original Message - From: Ryan Malayter malay...@gmail.com I like to bag on my developers for not knowing anything about the infrastructure, but

RE: Amazon diagnosis

2011-05-06 Thread Kenneth M. Chipps Ph.D.
I am preparing a graduate level course for network managers. As part of this course I would like to use a series of case studies looking at problems such as described in the report from Amazon. If anyone has something similar or knows where I could find such things, I would appreciate a copy or

Re: Amazon diagnosis

2011-05-05 Thread Ryan Malayter
On May 1, 2:29 pm, Jeff Wheeler j...@inconcepts.biz wrote: What it really boils down to is this: if application developers are doing their jobs, a given service can be easy and inexpensive to distribute to unrelated systems/networks without a huge infrastructure expense.  If the developers

Re: Amazon diagnosis

2011-05-05 Thread George Herbert
On Thu, May 5, 2011 at 10:45 AM, Ryan Malayter malay...@gmail.com wrote: On May 1, 2:29 pm, Jeff Wheeler j...@inconcepts.biz wrote: What it really boils down to is this: if application developers are doing their jobs, a given service can be easy and inexpensive to distribute to unrelated

Re: Amazon diagnosis

2011-05-05 Thread Jay Ashworth
- Original Message - From: Ryan Malayter malay...@gmail.com I like to bag on my developers for not knowing anything about the infrastructure, but sometimes you just can't do it right because of physics. Or you can't do it right without writing your own OS, networking stacks, file

Re: Amazon diagnosis

2011-05-03 Thread Jason Baugher
On 5/2/2011 4:11 PM, George Herbert wrote: On Mon, May 2, 2011 at 2:04 PM, Jeroen van Aartjer...@mompl.net wrote: valdis.kletni...@vt.edu wrote: On Mon, 02 May 2011 12:27:34 PDT, Jeroen van Aart said: It surprised me because I, perhaps naively, assumed IT workers in general have a rather

Re: Amazon diagnosis

2011-05-03 Thread Phil Pierotti
Unlike the US of A, here in Australia the industry has gone *very* heavily down the path of requiring/expecting certification. They have bought into the faith that unless your resume includes CC?? you're worthless. There are colleges (er, I mean training businesses) who will *guarantee* you will

Re: Amazon diagnosis

2011-05-02 Thread Jeroen van Aart
Jeff Wheeler wrote: IT managers would do well to understand that a few smart programmers, who understand how all their tools (web servers, databases, filesystems, load-balancers, etc.) actually work, can often do more to I fully agree. But much to my dismay and surprise I have learned that

Re: Amazon diagnosis

2011-05-02 Thread Valdis . Kletnieks
On Mon, 02 May 2011 12:27:34 PDT, Jeroen van Aart said: It surprised me because I, perhaps naively, assumed IT workers in general have a rather broad knowledge No, the average IT worker is always a mere 3 keystrokes away from getting their latest creation listed on www.thedailywtf.com.

Re: Amazon diagnosis

2011-05-02 Thread Paul Graydon
On 05/02/2011 09:27 AM, Jeroen van Aart wrote: Jeff Wheeler wrote: IT managers would do well to understand that a few smart programmers, who understand how all their tools (web servers, databases, filesystems, load-balancers, etc.) actually work, can often do more to I fully agree. But much

Re: Amazon diagnosis

2011-05-02 Thread Jeroen van Aart
valdis.kletni...@vt.edu wrote: On Mon, 02 May 2011 12:27:34 PDT, Jeroen van Aart said: It surprised me because I, perhaps naively, assumed IT workers in general have a rather broad knowledge Sorry to break it to you. That's ok, the past tense in my story testifies to the fact I was

Re: Amazon diagnosis

2011-05-02 Thread George Herbert
On Mon, May 2, 2011 at 2:04 PM, Jeroen van Aart jer...@mompl.net wrote: valdis.kletni...@vt.edu wrote: On Mon, 02 May 2011 12:27:34 PDT, Jeroen van Aart said: It surprised me because I, perhaps naively, assumed IT workers in general have a rather broad knowledge Sorry to break it to you.

Re: Amazon diagnosis

2011-05-02 Thread James Smith
@nanog.org Subject: Re: Amazon diagnosis Jeff Wheeler wrote: IT managers would do well to understand that a few smart programmers, who understand how all their tools (web servers, databases, filesystems, load-balancers, etc.) actually work, can often do more to I fully agree. But much to my dismay

Re: Amazon diagnosis

2011-05-01 Thread Mike
On 04/29/2011 12:35 PM, Joly MacFie wrote: *http://aws.amazon.com/message/65648/*http://aws.amazon.com/message/65648/ ___ So, in a nut shell, Amazon had a single point of failure which touched off this entire incident. I am still waiting for proof that single points of failure can

Re: Amazon diagnosis

2011-05-01 Thread Jay Ashworth
- Original Message - From: Mike mike-na...@tiedyenetworks.com On 04/29/2011 12:35 PM, Joly MacFie wrote: *http://aws.amazon.com/message/65648/*http://aws.amazon.com/message/65648/ So, in a nut shell, Amazon had a single point of failure which touched off this entire incident. I

Re: Amazon diagnosis

2011-05-01 Thread Andrew Kirch
On 5/1/2011 2:07 PM, Mike wrote: I am still waiting for proof that single points of failure can realistically be completely eliminated from any moderately complicated network environment / application. So far, I think murphy is still winning on this one. Sure they can, but as a thought

RE: Amazon diagnosis

2011-05-01 Thread George Bonser
I am still waiting for proof that single points of failure can realistically be completely eliminated from any moderately complicated network environment / application. So far, I think murphy is still winning on this one. Good job by the AWS team however, I am sure your new procedures and

Re: Amazon diagnosis

2011-05-01 Thread Paul Graydon
On 5/1/2011 9:29 AM, Jeff Wheeler wrote: On Sun, May 1, 2011 at 2:18 PM, Andrew Kirchtrel...@trelane.net wrote: Sure they can, but as a thought exercise fully 2n redundancy is difficult on a small scale for anything web facing. I've seen a very simple implementation for a website requiring 5

Re: Amazon diagnosis

2011-05-01 Thread Brett Frankenberger
On Sun, May 01, 2011 at 12:50:37PM -0700, George Bonser wrote: From my reading of what happened, it looks like they didn't have a single point of failure but ended up routing around their own redundancy. They apparently had a redundant primary network and, on top of that, a secondary

Re: Amazon diagnosis

2011-05-01 Thread Robert Bonomi
Date: Sun, 01 May 2011 11:07:56 -0700 From: Mike mike-na...@tiedyenetworks.com To: nanog@nanog.org Subject: Re: Amazon diagnosis On 04/29/2011 12:35 PM, Joly MacFie wrote: http://aws.amazon.com/message/65648/ ___ So, in a nut shell, Amazon had a single point of failure which

Re: Amazon diagnosis

2011-05-01 Thread Valdis . Kletnieks
On Sun, 01 May 2011 11:07:56 PDT, Mike said: I am still waiting for proof that single points of failure can realistically be completely eliminated from any moderately complicated network environment / application. So far, I think murphy is still winning on this one. For starters, you

RE: Amazon diagnosis

2011-05-01 Thread Robert Bonomi
Subject: RE: Amazon diagnosis Date: Sun, 1 May 2011 12:50:37 -0700 From: George Bonser gbon...@seven.com They apparently had a redundant primary network and, on top of that, a secondary network. The secondary network, however, did not have the capacity of the primary network. Rather

Re: Amazon diagnosis

2011-05-01 Thread Stefan
On Fri, Apr 29, 2011 at 2:35 PM, Joly MacFie j...@punkcast.com wrote: *http://aws.amazon.com/message/65648/*http://aws.amazon.com/message/65648/ ___ -- --- Joly MacFie  218 565 9365 Skype:punkcast WWWhatsup NYC -

Amazon diagnosis

2011-04-29 Thread Joly MacFie
*http://aws.amazon.com/message/65648/*http://aws.amazon.com/message/65648/ ___ -- --- Joly MacFie 218 565 9365 Skype:punkcast WWWhatsup NYC - http://wwwhatsup.com http://pinstand.com - http://punkcast.com VP (Admin) - ISOC-NY -