Constructive posts are always helpful, especially if numbers are provided, so we welcome you posted when there are performance issues. Aggregate statistics are a 95% case in terms of getting us useful information - we may be missing localized phenomenons.
There are reports of issues right now. We're investigating these and will update downtime-notify if we decide to take more drastic action to address the short term issues. On Thu, Sep 16, 2010 at 12:59 PM, Jason C <[email protected]> wrote: > I completely agree Cameron. > > I know that Google will be working hard to correct any problems. > > However, unless I see communication to this effect, I'm am not sure > that Google is _aware_ there is a problem - more so if the App Engine > status page is not indicating a problem. > > This forces me to sort of panic post on this group. A few pieces of > timely communication would prevent me from doing this. > > j > > On Sep 15, 11:24 pm, Cameron <[email protected]> wrote: > > Ikai, > > > > Thanks for listening to and considering our feedback. I saw several > > of the spike descriptions on the status page were already updated > > earlier today to be more accurate. I have full confidence that Google > > engineers work as quickly as possible to resolve the issues during > > these periods of disruptions. Having good communication from you guys > > acknowledging the severity of the problem and the efforts being taken > > to fix it goes a long way towards being able to handle the situation > > appropriately, both for myself and the users of my app. > > > > Thanks for your hard work. > > > > -Cameron > > > > On Sep 15, 4:32 pm, "Ikai Lan (Google)" > > <[email protected]<ikai.l%[email protected]> > > > > wrote: > > > > > > > > > Hey guys, > > > > > I understand your frustration. We have many Google services deployed on > top > > > of App Engine as well, and we get pressure from both sides anytime > events > > > impact production. There are several issues around communication being > > > highlighted here: > > > > > - App Engine Status page was being updated when we were having latency > > > problems > > > - Status page did not accurately describe the impact > > > - Delay between when we recognized a production event and posting to > > > downtime notify > > > - Downtime-notify emails are being marked as spam > > > > > We've attempted to take steps in the past to resolve the spam issue, > though > > > users are reporting that it hasn't worked. As far as the delay between > us > > > accurately identifying a production event and updating the groups - > well, > > > we'll have to figure out how we can minimize that. At the very least, > there > > > will be a communications post-mortem internally about what we plan on > doing > > > to address or at least minimizing the impact of these issues. > > > > > The option to go into an unplanned maintenance period was on the table, > but > > > it was one of those situations where we assessed it as overkill, > especially > > > since there were period when the latency appeared to have died down, > only to > > > restart again. We don't want to be too trigger happy with unscheduled > > > downtime, as degraded performance is usually preferable to a completely > > > downtime state (many of you may disagree with me, but this is a bit of > a > > > judgment call depending on the level of degradation). We're cautiously > > > optimistic at the moment about performance, but at this point, if the > spikes > > > begin appearing again, we may initiate another full downtime period. > Stay > > > tuned to the downtime-notify list. I'll let my team members know to > post > > > using their @google.com accounts to avoid being marked as spam. > > > > > On Wed, Sep 15, 2010 at 4:10 PM, Cameron <[email protected]> wrote: > > > > Hi Ikai - > > > > > > Just to be clear the issues have NOT subsided since last night. I > > > > hope you guys are working on eliminating the root of the problem and > > > > not just "monitoring the service closely" as the latest App Engine > > > > Notify post suggests. > > > > > > > http://groups.google.com/group/google-appengine-downtime-notify/brows... > > > > > > And even though the status page says "this spike did not affect the > > > > performance or uptime of applications" - every spike DOES in fact > > > > affect the performance of applications (mine at least - GQueues, but > > > > probably all apps). These red spikes make my app inaccessible. Even > > > > the yellow spikes cause many 500 errors. Basically this makes my app > > > > unusable, because users can't get any consistent work done with the > > > > frequent errors. > > > > > > > http://code.google.com/status/appengine/detail/datastore/2010/09/15#a... > > > > > > Most of all, the frequent errors make my app seem very brittle and > > > > deteriorates user confidence. Sales drop and my own forum gets lots > > > > of complaints. And then of course people start posting on Twitter. > > > > > > Anyway, I'm sure you guys are working very hard to fix the issues and > > > > want App Engine to be as reliable as possible. My suggestion is that > > > > you also look to improve communication during these times. I have to > > > > respond to my own users during these situations. This becomes very > > > > difficult when all I can tell them is "Google thinks the issue is > > > > resolved and is just monitoring the situation" when clearly the > status > > > > graphs indicate otherwise and people can't access my app. Or "Google > > > > says the issue didn't affect performance or uptime" when clearly it > > > > has. > > > > > > -Cameron > > > > > > On Sep 15, 6:13 am, Arny <[email protected]> wrote: > > > > > We're still getting a lot of 500s (dashboard & front end). > > > > > > > Did you transferred our apps to lower-cost servers or why is > > > > > everything working that bad since the maintenance? > > > > > When are the REAL paid services coming? > > > > > > > On Sep 15, 6:17 am, "Ikai Lan (Google)" > > > > > <[email protected]<ikai.l%[email protected]> > <ikai.l%[email protected] <ikai.l%[email protected]>> > > > > > > > wrote: > > > > > > > > Hi Tim, > > > > > > > > You can track the progress here: > > > > > > > > > http://groups.google.com/group/google-appengine-downtime-notify/brows. > > > > .. > > > > > > > > It's pretty hard to give an ETA, but we'd like to resolve this as > soon > > > > as > > > > > > possible. We're seeing signs that the issues may have subsided, > but > > > > we'd > > > > > > like a bit more confidence before giving the all clear. > > > > > > > > On Tue, Sep 14, 2010 at 6:49 PM, Tim Hoffman <[email protected] > > > > > > wrote: > > > > > > > Hi > > > > > > > http://groups.google.com.au/group/google-appengine-downtime-notify/br... > > > > > > > was posted several hours ago, with no updates. > > > > > > > > > I certainly am experiencing significant ongoing issues with > > > > taskqueues > > > > > > > and datastore timeouts and half the time can't get to the > dashboard. > > > > > > > > > I know someone must be working hard on this, but a little more > detail > > > > > > > on progress, ie ETA to recovery would be really great > > > > > > > > > Thanks > > > > > > > > > Tim > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the > Google > > > > Groups > > > > > > > "Google App Engine" group. > > > > > > > To post to this group, send email to > > > > [email protected]. > > > > > > > To unsubscribe from this group, send email to > > > > > > > [email protected]<google-appengine%[email protected]><google-appengine%2Bunsubscrib > [email protected]><google-appengine%2Bunsubscrib > > > > [email protected]> > > > > > > > . > > > > > > > For more options, visit this group at > > > > > > >http://groups.google.com/group/google-appengine?hl=en. > > > > > > -- > > > > You received this message because you are subscribed to the Google > Groups > > > > "Google App Engine" group. > > > > To post to this group, send email to > [email protected]. > > > > To unsubscribe from this group, send email to > > > > [email protected]<google-appengine%[email protected]><google-appengine%2Bunsubscrib > [email protected]> > > > > . > > > > For more options, visit this group at > > > >http://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<google-appengine%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
