I'm much more unnerved by the term 'blast radius' Cal
-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Michael B. Smith Sent: March 2, 2017 3:36 PM To: [email protected] Subject: RE: [EXTERNAL]Re: [NTSysADM] AWS East Outage OMG. “we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years.” That sentence scares me. But perhaps it shouldn’t. From: [email protected] [mailto:[email protected]] On Behalf Of Kennedy, Jim Sent: Thursday, March 2, 2017 3:12 PM To: [email protected] Subject: RE: [EXTERNAL]Re: [NTSysADM] AWS East Outage So the facts are out. Short version, basically someone fat fingered a command and deleted a bunch of really important servers. https://aws.amazon.com/message/41926/ From: [email protected] [mailto:[email protected]] On Behalf Of Melvin Backus Sent: Thursday, March 2, 2017 9:47 AM To: [email protected] Subject: RE: [EXTERNAL]Re: [NTSysADM] AWS East Outage That’s probably what caused the problem to being with. All that conversion and somebody missed a decimal point. -- There are 10 kinds of people in the world... those who understand binary and those who don't. From: [email protected] [mailto:[email protected]] On Behalf Of David McSpadden Sent: Thursday, March 2, 2017 7:17 AM To: [email protected] Subject: RE: [EXTERNAL]Re: [NTSysADM] AWS East Outage I believe it was an US-Converted-Metric S-ton IMHO. From: [email protected] [mailto:[email protected]] On Behalf Of Richard Stovall Sent: Thursday, March 2, 2017 7:05 AM To: [email protected] Subject: [EXTERNAL]Re: [NTSysADM] AWS East Outage Is that a metric S-ton, or the other kind? The is a difference. On Mar 2, 2017 2:38 AM, "Don Ely" <[email protected]> wrote: It is pretty trivial if you're setup correctly, but the setup takes an S-Ton of work and testing... On Wed, Mar 1, 2017 at 3:30 PM Michael B. Smith <[email protected]> wrote: I have to say, what surprised me most about this outage was the lack of failover to alternate datacenters for some pretty big names. I have no idea how this works in AWS, but in Azure it’s fairly trivial; I would expect the same of AWS. From: [email protected] [mailto:[email protected]] On Behalf Of Andrew S. Baker Sent: Wednesday, March 1, 2017 12:22 PM To: [email protected] <mailto:[email protected]> Subject: Re: [NTSysADM] AWS East Outage If not S3, then what? You're always going to be relying on someone else's something. Some data center provider (okay, so you might run your own) Some power provider Some Internet provider It's not like they have internet outages every week, and it's not like various organizations relying upon them haven't had outages for their own reasons. Technology breaks, which is why we RAID, cluster, backup, failover and farm our systems, devices and data centers. Regards, ASB http://XeeMe.com/AndrewBaker <http://xeeme.com/AndrewBaker> Providing Expert Technology Consulting Services for the SMB market… GPG: 860D 40A1 4DA5 3AE1 B052 8F9F 07A1 F9D6 A549 8842 Sent with Mixmax <https://mixmax.com/s/WMB47Rd39yDNPFfWo?utm_source=mixmax&utm_medium=email&utm_campaign=signature_link&utm_content=sent_with_mixmax> On Wed, Mar 1, 2017 8:37 AM, J- P [email protected] <mailto:[email protected]> wrote: https://techcrunch.com/2017/03/01/the-day-amazon-s3-storage-stood-still/ Would / should you hold your IT vendor responsible for relying on S3? Jean-Paul Natola ________________________________ From: [email protected] <[email protected]> on behalf of Andrew S. Baker <[email protected]> Sent: Tuesday, February 28, 2017 5:36 PM To: [email protected] <mailto:[email protected]> Subject: Re: [NTSysADM] AWS East Outage Indeed. Regards, ASB GPG: 860D 40A1 4DA5 3AE1 B052 8F9F 07A1 F9D6 A549 8842 Sent with Mixmax <https://mixmax.com/s/WMB47Rd39yDNPFfWo?utm_source=mixmax&utm_medium=email&utm_campaign=signature_link&utm_content=sent_with_mixmax> <https://compose.mixmax.com/img/blank.png> On Tue, Feb 28, 2017 3:56 PM, David McSpadden [email protected] wrote: So the normal question 'is the Internet down?' Is valid today? Sent from my iPhone On Feb 28, 2017, at 3:44 PM, Andrew S. Baker <[email protected]> wrote: Notice: This email is from an outside source. Please do not open any attachments, click on any hyperlinks, or respond without first confirming the authenticity of the email. Indeed. It's like someone broke the whole Internet. Or, at least, 80% of it. Regards, ASB http://XeeMe.com/AndrewBaker <http://xeeme.com/AndrewBaker> Providing Expert Technology Consulting Services for the SMB market… GPG: 860D 40A1 4DA5 3AE1 B052 8F9F 07A1 F9D6 A549 8842 Sent with Mixmax <https://mixmax.com/s/WMB47Rd39yDNPFfWo?utm_source=mixmax&utm_medium=email&utm_campaign=signature_link&utm_content=sent_with_mixmax> On Tue, Feb 28, 2017 2:13 PM, Kennedy, Jim [email protected] wrote: Learning very quickly how many vendors we have that are using AWS. Lots is the first word that comes to mind. From: [email protected] <mailto:[email protected]> [mailto:[email protected] <mailto:[email protected]> ] On Behalf Of Charles F Sullivan Sent: Tuesday, February 28, 2017 1:57 PM To: [email protected] <mailto:[email protected]> Subject: [NTSysADM] AWS East Outage Any of your organizations being affected by this? The few services we have moved there so far are down. http://bgr.com/2017/02/28/internet-outage-amazon-web-services/ Charlie Sullivan Sr. Windows Systems Administrator Boston College 197 Foster St. Room 367 Brighton, MA 02135 This e-mail and any files transmitted with it are property of Indiana Members Credit Union, are confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not one of the named recipient(s) or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your computer. Any other use, retention, dissemination, forwarding, printing, or copying of this email is strictly prohibited. Please consider the environment before printing this email. This e-mail and any files transmitted with it are property of Indiana Members Credit Union, are confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not one of the named recipient(s) or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your computer. Any other use, retention, dissemination, forwarding, printing, or copying of this email is strictly prohibited. Please consider the environment before printing this email.
