Re: AWS EC2 us-west-2 reboot
For those interested, this is the Xen bug they were fixing with the reboots http://xenbits.xen.org/xsa/advisory-108.html -Grant On Wed, Sep 24, 2014 at 8:41 PM, Reed Loden r...@reedloden.com wrote: On Wed, 24 Sep 2014 21:39:39 -0400 Peter Beckman beck...@angryox.com wrote: Likely some sort of potentially serious bug or flaw in EC2 or Xen. AWS Security is really on the ball on such things and do everything they can to make invisible fixes with no customer impact, but sometimes a reboot is required in order to apply the changes necessary to keep customer instances safe from attacks and vulnerabilities. Rumor mill is that it's XSA-108, embargoed until 2014-10-01 12:00 (http://xenbits.xen.org/xsa/). Just somebody's guess, though, afaik. ~reed
Re: AWS EC2 us-west-2 reboot
On Wed, Oct 01, 2014 at 11:01:37AM -0700, Grant Ridder wrote: For those interested, this is the Xen bug they were fixing with the reboots http://xenbits.xen.org/xsa/advisory-108.html Ouch. Good thing Bashpocalypse is still capturing everyone's attention... Interestingly, Amazon *didn't* discover this bug, which makes one wonder why they, out of all the big Xen-based providers out there, got a heads-up in advance of the embargo end. If I was a big provider who didn't get advance notice, I'd be somewhat miffed. - Matt -- If you are a trauma surgeon and someone dies on your table, [...] everyone would know you did your best. When someone does something truly stupid with their system and it dies and you can't resuscitate it, you must be incompetent or an idiot. -- Julian Macassey, in the Monastery
Re: AWS EC2 us-west-2 reboot
On 01/10/2014 4:29 PM, Matt Palmer wrote: On Wed, Oct 01, 2014 at 11:01:37AM -0700, Grant Ridder wrote: For those interested, this is the Xen bug they were fixing with the reboots http://xenbits.xen.org/xsa/advisory-108.html Ouch. Good thing Bashpocalypse is still capturing everyone's attention... Interestingly, Amazon *didn't* discover this bug, which makes one wonder why they, out of all the big Xen-based providers out there, got a heads-up in advance of the embargo end. If I was a big provider who didn't get advance notice, I'd be somewhat miffed. Rackspace did reboots over the weekend for this as well - http://www.rackspace.com/blog/an-apology/ Bryan --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com
Re: AWS EC2 us-west-2 reboot
read: http://www.xenproject.org/security-policy.html they have a sensible, commonly used security policy that involves private notification to large customers in advance where it is practical and there is not evidence of ongoing exploits in the wild. this is kind of incident handling 101 and shouldn't be surprising to anyone. t On Wed, Oct 1, 2014 at 4:38 PM, Bryan Fullerton fehwal...@gmail.com wrote: On 01/10/2014 4:29 PM, Matt Palmer wrote: On Wed, Oct 01, 2014 at 11:01:37AM -0700, Grant Ridder wrote: For those interested, this is the Xen bug they were fixing with the reboots http://xenbits.xen.org/xsa/advisory-108.html Ouch. Good thing Bashpocalypse is still capturing everyone's attention... Interestingly, Amazon *didn't* discover this bug, which makes one wonder why they, out of all the big Xen-based providers out there, got a heads-up in advance of the embargo end. If I was a big provider who didn't get advance notice, I'd be somewhat miffed. Rackspace did reboots over the weekend for this as well - http://www.rackspace.com/blog/an-apology/ Bryan --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com
Re: AWS EC2 us-west-2 reboot
On 10/01/2014 02:59 PM, Todd Underwood wrote: read: http://www.xenproject.org/security-policy.html they have a sensible, commonly used security policy that involves private notification to large customers in advance where it is practical and there is not evidence of ongoing exploits in the wild. this is kind of incident handling 101 and shouldn't be surprising to anyone. You don't have to be that large to get on the list.
Re: AWS EC2 us-west-2 reboot
On Oct 1, 2014, at 4:59 PM, Todd Underwood toddun...@gmail.com wrote: this is kind of incident handling 101 and shouldn't be surprising to anyone. There’s always people who feel “left out of the loop” when these things occur. I’ve found there’s no one location for centralized data after many years of doing this from the ASN.1/ILMI days to present. It requires being professional and engaging when most people just want to consume the derived data. Having found a few of these issues myself over the years, the best bugs are the ones where the advisory comes out after the fixed software is broadly available and deployed. Nothing will be perfect as people always like their legacy system that requires no work, but in reality, there is no such thing. - Jared
AWS EC2 us-west-2 reboot
As an FYI, it looks like Amazon is doing a mass reboot of the physical hosts in us-west-2 across all AZ's and it is scheduled to start tomorrow and take a couple days. Go to *https://console.aws.amazon.com/ec2/v2/home?region=us-west-2#Events https://console.aws.amazon.com/ec2/v2/home?region=us-west-2#Events:* to see what instances are affected when. -Grant
Re: AWS EC2 us-west-2 reboot
Bash related? On Sep 24, 2014, at 4:47 PM, Grant Ridder shortdudey...@gmail.com wrote: As an FYI, it looks like Amazon is doing a mass reboot of the physical hosts in us-west-2 across all AZ's and it is scheduled to start tomorrow and take a couple days. Go to *https://console.aws.amazon.com/ec2/v2/home?region=us-west-2#Events https://console.aws.amazon.com/ec2/v2/home?region=us-west-2#Events:* to see what instances are affected when. -Grant
Re: AWS EC2 us-west-2 reboot
Likely not, since it's affecting Windows instances as well. Also not just us-west-2 -- we have tons of instances scheduled for downtime in us-east-1 and eu-west-1 as well. -Peter On 09/24/2014 04:51 PM, Gabriel Blanchard wrote: Bash related? On Sep 24, 2014, at 4:47 PM, Grant Ridder shortdudey...@gmail.com wrote: As an FYI, it looks like Amazon is doing a mass reboot of the physical hosts in us-west-2 across all AZ's and it is scheduled to start tomorrow and take a couple days. Go to *https://console.aws.amazon.com/ec2/v2/home?region=us-west-2#Events https://console.aws.amazon.com/ec2/v2/home?region=us-west-2#Events:* to see what instances are affected when. -Grant
Re: AWS EC2 us-west-2 reboot
Doubt it since a bash patch shouldn't require a reboot On Wed, Sep 24, 2014 at 1:51 PM, Gabriel Blanchard g...@teksavvy.ca wrote: Bash related? On Sep 24, 2014, at 4:47 PM, Grant Ridder shortdudey...@gmail.com wrote: As an FYI, it looks like Amazon is doing a mass reboot of the physical hosts in us-west-2 across all AZ's and it is scheduled to start tomorrow and take a couple days. Go to * https://console.aws.amazon.com/ec2/v2/home?region=us-west-2#Events https://console.aws.amazon.com/ec2/v2/home?region=us-west-2#Events:* to see what instances are affected when. -Grant
Re: AWS EC2 us-west-2 reboot
Just got the same email. Not just US. Servers in Sydney we have also. Why such short notice? On Sep 24, 2014 4:58 PM, Grant Ridder shortdudey...@gmail.com wrote: Doubt it since a bash patch shouldn't require a reboot On Wed, Sep 24, 2014 at 1:51 PM, Gabriel Blanchard g...@teksavvy.ca wrote: Bash related? On Sep 24, 2014, at 4:47 PM, Grant Ridder shortdudey...@gmail.com wrote: As an FYI, it looks like Amazon is doing a mass reboot of the physical hosts in us-west-2 across all AZ's and it is scheduled to start tomorrow and take a couple days. Go to * https://console.aws.amazon.com/ec2/v2/home?region=us-west-2#Events https://console.aws.amazon.com/ec2/v2/home?region=us-west-2#Events:* to see what instances are affected when. -Grant
Re: AWS EC2 us-west-2 reboot
On Wed, Sep 24, 2014 at 3:56 PM, Grant Ridder shortdudey...@gmail.com wrote: Doubt it since a bash patch shouldn't require a reboot Unless you have a long-running bash script in the background providing a vital system service, and that service is so important in your environment that you might as well reboot rather than kill and respawn it. -- -JH
Re: AWS EC2 us-west-2 reboot
Likely some sort of potentially serious bug or flaw in EC2 or Xen. AWS Security is really on the ball on such things and do everything they can to make invisible fixes with no customer impact, but sometimes a reboot is required in order to apply the changes necessary to keep customer instances safe from attacks and vulnerabilities. Another possibility: getting rid of older hardware. A reboot will keep you in the same class of service but may move you to a new physical machine. Unlikely though at this reported scale. Same thing happened in December 2011 [1]. Beckman [1] http://www.crn.com/news/cloud/232300111/widespread-amazon-ec2-cloud-instance-reboots-spark-questions-concerns.htm On Wed, 24 Sep 2014, Javier J wrote: Just got the same email. Not just US. Servers in Sydney we have also. Why such short notice? On Sep 24, 2014 4:58 PM, Grant Ridder shortdudey...@gmail.com wrote: Doubt it since a bash patch shouldn't require a reboot On Wed, Sep 24, 2014 at 1:51 PM, Gabriel Blanchard g...@teksavvy.ca wrote: Bash related? On Sep 24, 2014, at 4:47 PM, Grant Ridder shortdudey...@gmail.com wrote: As an FYI, it looks like Amazon is doing a mass reboot of the physical hosts in us-west-2 across all AZ's and it is scheduled to start tomorrow and take a couple days. Go to * https://console.aws.amazon.com/ec2/v2/home?region=us-west-2#Events https://console.aws.amazon.com/ec2/v2/home?region=us-west-2#Events:* to see what instances are affected when. -Grant --- Peter Beckman Internet Guy beck...@angryox.com http://www.angryox.com/ ---
Re: AWS EC2 us-west-2 reboot
On Wed, 24 Sep 2014 21:39:39 -0400 Peter Beckman beck...@angryox.com wrote: Likely some sort of potentially serious bug or flaw in EC2 or Xen. AWS Security is really on the ball on such things and do everything they can to make invisible fixes with no customer impact, but sometimes a reboot is required in order to apply the changes necessary to keep customer instances safe from attacks and vulnerabilities. Rumor mill is that it's XSA-108, embargoed until 2014-10-01 12:00 (http://xenbits.xen.org/xsa/). Just somebody's guess, though, afaik. ~reed
