It may have settled down for you Tim, but we are being hammered by DeadlineExceededError right now. Log time: 8.27a Thursday 2010-09-02
AppID: steprep In the past, this problem seems to be localized to small (presumably) clusters of apps. Can someone please look into our specific cluster? Thanks, j On Sep 1, 6:06 pm, Tim Hoffman <[email protected]> wrote: > Hi > > It seems that the massive increase startup times has settled down. > > But we effectively had to app instances down for about 16 hours due to > the issue. > It is clear the startup peformance issue is not my code as it was > affecting a other people, but > was not widespread enough to get on the system status radar. > > And issue for it was > raisedhttp://code.google.com/p/googleappengine/issues/detail?id=3667 > > The outage was not good but to be honest the biggest problem my > management hand with it > was the complete absence of any response from google during this > period. > > Last time we had a major issue it took a week, and a large report > prepared by myself detailing what was > going on before I could get any response from google. The response was > also > limited to "This is a known issue, we are working to address it" which > is fine in it self at least some communication > was eventually made but it should not have taken a week and about 3 > days of effort on my part to actually get any form of response. > > See the final part of that saga > here.http://groups.google.com.au/group/google-appengine/browse_thread/thre... > > I am huge advocate of appengine and we are using it more and more, but > the total feeling of helplessness you > get when you have instances non functioning because they can't even > startup for an extended period and > no feedback what so ever makes it very hard to even manage > expectations, and impossible to report to management > about any form of progress in resolving the issue. > > In addition when this happened last time, a journalist from the > Register contacted me about the issue and asked for more information > so they could do a story about the lack of communication from google > on these sorts of issues. I declined, but maybe it does need some > more visibility and least with in the app engine user community. > > Regards > > Tim Hoffman > > On Sep 1, 11:54 pm, Tim Hoffman <[email protected]> wrote: > > > > > The rate of errors has dropped off somewhat. > > > It is still happening though. > > > T > > > On Sep 1, 7:24 pm, Tim Hoffman <[email protected]> wrote: > > > > Further investigation shows these excessive startup times are being > > > experienced by more than one instance. > > > > Appid's qtrack-dev, q-tracker. > > > > T > > > > On Sep 1, 5:15 pm, Tim Hoffman <[email protected]> wrote: > > > > > Hi > > > > > I am still having trouble getting instances to get past initial > > > > import (appid q-tracker) > > > > > I have even tried starting old versions to get around failed imports. > > > > > for the last 10 minutes all startup attempts end in DeadlineExceeded > > > > error. > > > > > T > > > > > On Sep 1, 2:45 pm, Tim Hoffman <[email protected]> wrote: > > > > > > Hi > > > > > > As I have pointed out in the past its quite possible to get pockets > > > > > of appengine instances experiencing extremely poor performance and > > > > > this doesn't show up in the system status. (http:// > > > > > groups.google.com.au/group/google-appengine/browse_thread/thread/ > > > > > 23e988d494144242/8ff964f7ffcc0b37?q=#8ff964f7ffcc0b37) > > > > > > Apps seemd to be sharded into different parts of appengine > > > > > infrastructure based on appid. > > > > > > I hope this problem is just transitory for tosay and I don't have to > > > > > spend days collecting evidence again. > > > > > > T > > > > > > On Sep 1, 2:30 pm, Tim Hoffman <[email protected]> wrote: > > > > > > > Hi > > > > > > > Except it isn't. I am still getting hit with extremeley long startup > > > > > > times. > > > > > > I am not even getting around to accessing the datastore. Failure is > > > > > > always during instance startup. > > > > > > > T > > > > > > > On Sep 1, 2:27 pm, Andi Albrecht <[email protected]> > > > > > > wrote: > > > > > > > > Sorry, I've missed this announcement on Downtime Notify: > > > > > > > >http://groups.google.com/group/google-appengine-downtime-notify/brows... > > > > > > > > As noted in my previous post, everything's fine now :) > > > > > > > > Andi > > > > > > > > Tim Hoffman <[email protected]> writes: > > > > > > > > Hi > > > > > > > > > I am still seeing them on and off, everything might be fine for > > > > > > > > 20min, > > > > > > > > then bam. > > > > > > > > I have even been having some dashboard failures. > > > > > > > > > T > > > > > > > > > On Sep 1, 1:55 pm, Andi Albrecht <[email protected]> > > > > > > > > wrote: > > > > > > > >> Tim Hoffman <[email protected]> writes: > > > > > > > >> > Hi > > > > > > > > >> > As of today I am getting a lot of DeadlineExceeded Errors on > > > > > > > >> > a couple > > > > > > > >> > of instances druing startup. > > > > > > > > >> > I am not even getting past the initial imports, > > > > > > > > >> > Nothing is being reported in system status, so it's probably > > > > > > > >> > restricted part of the appengine structure. > > > > > > > > >> > Tim > > > > > > > > >> Hi Tim, > > > > > > > > >> do you still see these issues? I've seen a couple of different > > > > > > > >> errors a > > > > > > > >> few hours ago, but now everything looks fine again. > > > > > > > > >> Andi -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
