Hi It seems that the massive increase startup times has settled down.
But we effectively had to app instances down for about 16 hours due to the issue. It is clear the startup peformance issue is not my code as it was affecting a other people, but was not widespread enough to get on the system status radar. And issue for it was raised http://code.google.com/p/googleappengine/issues/detail?id=3667 The outage was not good but to be honest the biggest problem my management hand with it was the complete absence of any response from google during this period. Last time we had a major issue it took a week, and a large report prepared by myself detailing what was going on before I could get any response from google. The response was also limited to "This is a known issue, we are working to address it" which is fine in it self at least some communication was eventually made but it should not have taken a week and about 3 days of effort on my part to actually get any form of response. See the final part of that saga here. http://groups.google.com.au/group/google-appengine/browse_thread/thread/23e988d494144242/8ff964f7ffcc0b37?q=#8ff964f7ffcc0b37 I am huge advocate of appengine and we are using it more and more, but the total feeling of helplessness you get when you have instances non functioning because they can't even startup for an extended period and no feedback what so ever makes it very hard to even manage expectations, and impossible to report to management about any form of progress in resolving the issue. In addition when this happened last time, a journalist from the Register contacted me about the issue and asked for more information so they could do a story about the lack of communication from google on these sorts of issues. I declined, but maybe it does need some more visibility and least with in the app engine user community. Regards Tim Hoffman On Sep 1, 11:54 pm, Tim Hoffman <[email protected]> wrote: > The rate of errors has dropped off somewhat. > > It is still happening though. > > T > > On Sep 1, 7:24 pm, Tim Hoffman <[email protected]> wrote: > > > > > > > > > Further investigation shows these excessive startup times are being > > experienced by more than one instance. > > > Appid's qtrack-dev, q-tracker. > > > T > > > On Sep 1, 5:15 pm, Tim Hoffman <[email protected]> wrote: > > > > Hi > > > > I am still having trouble getting instances to get past initial > > > import (appid q-tracker) > > > > I have even tried starting old versions to get around failed imports. > > > > for the last 10 minutes all startup attempts end in DeadlineExceeded > > > error. > > > > T > > > > On Sep 1, 2:45 pm, Tim Hoffman <[email protected]> wrote: > > > > > Hi > > > > > As I have pointed out in the past its quite possible to get pockets > > > > of appengine instances experiencing extremely poor performance and > > > > this doesn't show up in the system status. (http:// > > > > groups.google.com.au/group/google-appengine/browse_thread/thread/ > > > > 23e988d494144242/8ff964f7ffcc0b37?q=#8ff964f7ffcc0b37) > > > > > Apps seemd to be sharded into different parts of appengine > > > > infrastructure based on appid. > > > > > I hope this problem is just transitory for tosay and I don't have to > > > > spend days collecting evidence again. > > > > > T > > > > > On Sep 1, 2:30 pm, Tim Hoffman <[email protected]> wrote: > > > > > > Hi > > > > > > Except it isn't. I am still getting hit with extremeley long startup > > > > > times. > > > > > I am not even getting around to accessing the datastore. Failure is > > > > > always during instance startup. > > > > > > T > > > > > > On Sep 1, 2:27 pm, Andi Albrecht <[email protected]> wrote: > > > > > > > Sorry, I've missed this announcement on Downtime Notify: > > > > > > >http://groups.google.com/group/google-appengine-downtime-notify/brows... > > > > > > > As noted in my previous post, everything's fine now :) > > > > > > > Andi > > > > > > > Tim Hoffman <[email protected]> writes: > > > > > > > Hi > > > > > > > > I am still seeing them on and off, everything might be fine for > > > > > > > 20min, > > > > > > > then bam. > > > > > > > I have even been having some dashboard failures. > > > > > > > > T > > > > > > > > On Sep 1, 1:55 pm, Andi Albrecht <[email protected]> > > > > > > > wrote: > > > > > > >> Tim Hoffman <[email protected]> writes: > > > > > > >> > Hi > > > > > > > >> > As of today I am getting a lot of DeadlineExceeded Errors on a > > > > > > >> > couple > > > > > > >> > of instances druing startup. > > > > > > > >> > I am not even getting past the initial imports, > > > > > > > >> > Nothing is being reported in system status, so it's probably > > > > > > >> > restricted part of the appengine structure. > > > > > > > >> > Tim > > > > > > > >> Hi Tim, > > > > > > > >> do you still see these issues? I've seen a couple of different > > > > > > >> errors a > > > > > > >> few hours ago, but now everything looks fine again. > > > > > > > >> Andi -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
