What an epic thread. Luckily I'm still a few days away from launching a new app on Heroku.
- edward. On Thu, Apr 21, 2011 at 1:13 PM, Jeff Schmitz <[email protected]>wrote: > Latest: > > I suppose Heroku is in the unavailability zone that is still down. Sorry, > Freudian slip. > 12:30 PM PDT We have observed successful new launches of EBS backed > instances for the past 15 minutes in all but one of the availability zones > in the US-EAST-1 Region. The team is continuing to work to recover the > unavailable EBS volumes as quickly as possible. > > > On Thu, Apr 21, 2011 at 1:45 PM, John Norman <[email protected]> wrote: > >> Here's what you want -- from: http://status.aws.amazon.com/ >> >> The last three provide the most information. >> >> 1:41 AM PDT We are currently investigating latency and error rates with >> EBS volumes and connectivity issues reaching EC2 instances in the US-EAST-1 >> region. >> >> 2:18 AM PDT We can confirm connectivity errors impacting EC2 instances >> and increased latencies impacting EBS volumes in multiple availability zones >> in the US-EAST-1 region. Increased error rates are affecting EBS >> CreateVolume API calls. We continue to work towards resolution. >> >> 2:49 AM PDT We are continuing to see connectivity errors impacting EC2 >> instances, increased latencies impacting EBS volumes in multiple >> availability zones in the US-EAST-1 region, and increased error rates >> affecting EBS CreateVolume API calls. We are also experiencing delayed >> launches for EBS backed EC2 instances in affected availability zones in the >> US-EAST-1 region. We continue to work towards resolution. >> >> 3:20 AM PDT Delayed EC2 instance launches and EBS API error rates are >> recovering. We're continuing to work towards full resolution. >> >> 4:09 AM PDT EBS volume latency and API errors have recovered in one of >> the two impacted Availability Zones in US-EAST-1. We are continuing to work >> to resolve the issues in the second impacted Availability Zone. The errors, >> which started at 12:55AM PDT, began recovering at 2:55am PDT >> >> 5:02 AM PDT Latency has recovered for a portion of the impacted EBS >> volumes. We are continuing to work to resolve the remaining issues with EBS >> volume latency and error rates in a single Availability Zone. >> >> 6:09 AM PDT EBS API errors and volume latencies in the affected >> availability zone remain. We are continuing to work towards resolution. >> >> 6:59 AM PDT There has been a moderate increase in error rates for >> CreateVolume. This may impact the launch of new EBS-backed EC2 instances in >> multiple availability zones in the US-EAST-1 region. Launches of instance >> store AMIs are currently unaffected. We are continuing to work on resolving >> this issue. >> >> 7:40 AM PDT In addition to the EBS volume latencies, EBS-backed instances >> in the US-EAST-1 region are failing at a high rate. This is due to a high >> error rate for creating new volumes in this region. >> >> 8:54 AM PDT We'd like to provide additional color on what were working on >> right now (please note that we always know more and understand issues better >> after we fully recover and dive deep into the post mortem). A networking >> event early this morning triggered a large amount of re-mirroring of EBS >> volumes in US-EAST-1. This re-mirroring created a shortage of capacity in >> one of the US-EAST-1 Availability Zones, which impacted new EBS volume >> creation as well as the pace with which we could re-mirror and recover >> affected EBS volumes. Additionally, one of our internal control planes for >> EBS has become inundated such that it's difficult to create new EBS volumes >> and EBS backed instances. We are working as quickly as possible to add >> capacity to that one Availability Zone to speed up the re-mirroring, and >> working to restore the control plane issue. We're starting to see progress >> on these efforts, but are not there yet. We will continue to provide updates >> when we have them. >> >> 10:26 AM PDT We have made significant progress in stabilizing the >> affected EBS control plane service. EC2 API calls that do not involve EBS >> resources in the affected Availability Zone are now seeing significantly >> reduced failures and latency and are continuing to recover. We have also >> brought additional capacity online in the affected Availability Zone and >> stuck EBS volumes (those that were being remirrored) are beginning to >> recover. We cannot yet estimate when these volumes will be completely >> recovered, but we will provide an estimate as soon as we have sufficient >> data to estimate the recovery. We have all available resources working to >> restore full service functionality as soon as possible. We will continue to >> provide updates when we have them. >> >> 11:09 AM PDT A number of people have asked us for an ETA on when we'll be >> fully recovered. We deeply understand why this is important and promise to >> share this information as soon as we have an estimate that we believe is >> close to accurate. Our high-level ballpark right now is that the ETA is a >> few hours. We can assure you that all-hands are on deck to recover as >> quickly as possible. We will update the community as we have more >> information. >> >> >> >> >> On Thu, Apr 21, 2011 at 1:22 PM, Shannon Perkins < >> [email protected]> wrote: >> >>> I'm a total lurker on this list, but I give a strong second to Eric's >>> comment. >>> >>> Whatever the technical explanation/root-cause turns out to be this is not >>> acceptable platform behavior. >>> >>> Very troubling. >>> >>> --sp >>> >>> >>> On Thu, Apr 21, 2011 at 2:06 PM, Eric Anderson <[email protected]>wrote: >>> >>>> On Apr 21, 11:50 am, Rohit Dewan <[email protected]> wrote: >>>> > Does anyone know why Heroku not able to redeploy onto another cluster? >>>> In >>>> > general, it would seem prudent to spread applications across the >>>> various >>>> > clusters so all apps do not suffer an outage when a single cluster is >>>> > affected. >>>> >>>> I agree completely. I was surprised to see that problems in just one >>>> of Amazons MANY data centers took Heroku down. Even their own website >>>> and their own support system are down. I thought the point of the >>>> cloud is to have your app stay up even if there are problems at one >>>> data center. >>>> >>>> Eric >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Heroku" group. >>>> To post to this group, send email to [email protected]. >>>> To unsubscribe from this group, send email to >>>> [email protected]. >>>> For more options, visit this group at >>>> http://groups.google.com/group/heroku?hl=en. >>>> >>>> >>> >>> >>> -- >>> Shannon Perkins >>> Editor of Interactive News Technologies >>> Wired.com >>> 415-276-4914 >>> --_--_- >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Heroku" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/heroku?hl=en. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Heroku" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/heroku?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. > -- Regards, ======== Edward Hotchkiss http://groupthinklabs.com/ +1.415.200.7902 ======== -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
