OMG. “we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years.”
That sentence scares me. But perhaps it shouldn’t. From: listsad...@lists.myitforum.com [mailto:listsad...@lists.myitforum.com] On Behalf Of Kennedy, Jim Sent: Thursday, March 2, 2017 3:12 PM To: ntsysadm@lists.myitforum.com Subject: RE: [EXTERNAL]Re: [NTSysADM] AWS East Outage So the facts are out. Short version, basically someone fat fingered a command and deleted a bunch of really important servers. https://aws.amazon.com/message/41926/ From: listsad...@lists.myitforum.com<mailto:listsad...@lists.myitforum.com> [mailto:listsad...@lists.myitforum.com] On Behalf Of Melvin Backus Sent: Thursday, March 2, 2017 9:47 AM To: ntsysadm@lists.myitforum.com<mailto:ntsysadm@lists.myitforum.com> Subject: RE: [EXTERNAL]Re: [NTSysADM] AWS East Outage That’s probably what caused the problem to being with. All that conversion and somebody missed a decimal point. -- There are 10 kinds of people in the world... those who understand binary and those who don't. From: listsad...@lists.myitforum.com<mailto:listsad...@lists.myitforum.com> [mailto:listsad...@lists.myitforum.com] On Behalf Of David McSpadden Sent: Thursday, March 2, 2017 7:17 AM To: ntsysadm@lists.myitforum.com<mailto:ntsysadm@lists.myitforum.com> Subject: RE: [EXTERNAL]Re: [NTSysADM] AWS East Outage I believe it was an US-Converted-Metric S-ton IMHO. From: listsad...@lists.myitforum.com<mailto:listsad...@lists.myitforum.com> [mailto:listsad...@lists.myitforum.com] On Behalf Of Richard Stovall Sent: Thursday, March 2, 2017 7:05 AM To: ntsysadm@lists.myitforum.com<mailto:ntsysadm@lists.myitforum.com> Subject: [EXTERNAL]Re: [NTSysADM] AWS East Outage Is that a metric S-ton, or the other kind? The is a difference. On Mar 2, 2017 2:38 AM, "Don Ely" <don....@gmail.com<mailto:don....@gmail.com>> wrote: It is pretty trivial if you're setup correctly, but the setup takes an S-Ton of work and testing... On Wed, Mar 1, 2017 at 3:30 PM Michael B. Smith <mich...@smithcons.com<mailto:mich...@smithcons.com>> wrote: I have to say, what surprised me most about this outage was the lack of failover to alternate datacenters for some pretty big names. I have no idea how this works in AWS, but in Azure it’s fairly trivial; I would expect the same of AWS. From: listsad...@lists.myitforum.com<mailto:listsad...@lists.myitforum.com> [mailto:listsad...@lists.myitforum.com<mailto:listsad...@lists.myitforum.com>] On Behalf Of Andrew S. Baker Sent: Wednesday, March 1, 2017 12:22 PM To: ntsysadm@lists.myitforum.com<mailto:ntsysadm@lists.myitforum.com> Subject: Re: [NTSysADM] AWS East Outage If not S3, then what? You're always going to be relying on someone else's something. Some data center provider (okay, so you might run your own) Some power provider Some Internet provider It's not like they have internet outages every week, and it's not like various organizations relying upon them haven't had outages for their own reasons. Technology breaks, which is why we RAID, cluster, backup, failover and farm our systems, devices and data centers. Regards, ASB http://XeeMe.com/AndrewBaker<http://xeeme.com/AndrewBaker> Providing Expert Technology Consulting Services for the SMB market… GPG: 860D 40A1 4DA5 3AE1 B052 8F9F 07A1 F9D6 A549 8842 Sent with Mixmax<https://mixmax.com/s/WMB47Rd39yDNPFfWo?utm_source=mixmax&utm_medium=email&utm_campaign=signature_link&utm_content=sent_with_mixmax> On Wed, Mar 1, 2017 8:37 AM, J- P jnat...@hotmail.com<mailto:jnat...@hotmail.com> wrote: https://techcrunch.com/2017/03/01/the-day-amazon-s3-storage-stood-still/ Would / should you hold your IT vendor responsible for relying on S3? Jean-Paul Natola ________________________________ From: listsad...@lists.myitforum.com<mailto:listsad...@lists.myitforum.com> <listsad...@lists.myitforum.com<mailto:listsad...@lists.myitforum.com>> on behalf of Andrew S. Baker <asbz...@gmail.com<mailto:asbz...@gmail.com>> Sent: Tuesday, February 28, 2017 5:36 PM To: ntsysadm@lists.myitforum.com<mailto:ntsysadm@lists.myitforum.com> Subject: Re: [NTSysADM] AWS East Outage Indeed. Regards, ASB GPG: 860D 40A1 4DA5 3AE1 B052 8F9F 07A1 F9D6 A549 8842 Sent with Mixmax<https://mixmax.com/s/WMB47Rd39yDNPFfWo?utm_source=mixmax&utm_medium=email&utm_campaign=signature_link&utm_content=sent_with_mixmax> On Tue, Feb 28, 2017 3:56 PM, David McSpadden dav...@imcu.com<mailto:dav...@imcu.com> wrote: So the normal question 'is the Internet down?' Is valid today? Sent from my iPhone On Feb 28, 2017, at 3:44 PM, Andrew S. Baker <asbz...@gmail.com<mailto:asbz...@gmail.com>> wrote: Notice: This email is from an outside source. Please do not open any attachments, click on any hyperlinks, or respond without first confirming the authenticity of the email. Indeed. It's like someone broke the whole Internet. Or, at least, 80% of it. Regards, ASB http://XeeMe.com/AndrewBaker<http://xeeme.com/AndrewBaker> Providing Expert Technology Consulting Services for the SMB market… GPG: 860D 40A1 4DA5 3AE1 B052 8F9F 07A1 F9D6 A549 8842 Sent with Mixmax<https://mixmax.com/s/WMB47Rd39yDNPFfWo?utm_source=mixmax&utm_medium=email&utm_campaign=signature_link&utm_content=sent_with_mixmax> On Tue, Feb 28, 2017 2:13 PM, Kennedy, Jim kennedy...@elyriaschools.org<mailto:kennedy...@elyriaschools.org> wrote: Learning very quickly how many vendors we have that are using AWS. Lots is the first word that comes to mind. From: listsad...@lists.myitforum.com<mailto:listsad...@lists.myitforum.com> [mailto:listsad...@lists.myitforum.com] On Behalf Of Charles F Sullivan Sent: Tuesday, February 28, 2017 1:57 PM To: ntsysadm@lists.myitforum.com<mailto:ntsysadm@lists.myitforum.com> Subject: [NTSysADM] AWS East Outage Any of your organizations being affected by this? The few services we have moved there so far are down. http://bgr.com/2017/02/28/internet-outage-amazon-web-services/ Charlie Sullivan Sr. Windows Systems Administrator Boston College 197 Foster St. Room 367 Brighton, MA 02135 This e-mail and any files transmitted with it are property of Indiana Members Credit Union, are confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not one of the named recipient(s) or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your computer. Any other use, retention, dissemination, forwarding, printing, or copying of this email is strictly prohibited. Please consider the environment before printing this email. This e-mail and any files transmitted with it are property of Indiana Members Credit Union, are confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not one of the named recipient(s) or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your computer. Any other use, retention, dissemination, forwarding, printing, or copying of this email is strictly prohibited. Please consider the environment before printing this email.