We're seeing the DEEs come in in batches, and there is not commonality. Some have some url fetch involved, but others do not.
j On Sep 2, 12:25 pm, "Ikai L (Google)" <[email protected]> wrote: > Jason, what are you doing in your handler that is hitting > DeadlineExceededError? It looks like you're doing an URLFetch. Could the > resources you are fetching be slow? > > > > > > On Thu, Sep 2, 2010 at 8:38 AM, Jason C <[email protected]> wrote: > > I should have mentioned that in addition to the 30s DEEs, we are also > > seeing a number of the 10s "Request was aborted" messages (the ones > > that occur before any user code executes). > > > j > > > On Sep 2, 9:32 am, Jason C <[email protected]> wrote: > > > It may have settled down for you Tim, but we are being hammered by > > > DeadlineExceededError right now. Log time: 8.27a Thursday 2010-09-02 > > > > AppID: steprep > > > > In the past, this problem seems to be localized to small (presumably) > > > clusters of apps. > > > > Can someone please look into our specific cluster? > > > > Thanks, > > > j > > > > On Sep 1, 6:06 pm, Tim Hoffman <[email protected]> wrote: > > > > > Hi > > > > > It seems that the massive increase startup times has settled down. > > > > > But we effectively had to app instances down for about 16 hours due to > > > > the issue. > > > > It is clear the startup peformance issue is not my code as it was > > > > affecting a other people, but > > > > was not widespread enough to get on the system status radar. > > > > > And issue for it was raisedhttp:// > > code.google.com/p/googleappengine/issues/detail?id=3667 > > > > > The outage was not good but to be honest the biggest problem my > > > > management hand with it > > > > was the complete absence of any response from google during this > > > > period. > > > > > Last time we had a major issue it took a week, and a large report > > > > prepared by myself detailing what was > > > > going on before I could get any response from google. The response was > > > > also > > > > limited to "This is a known issue, we are working to address it" which > > > > is fine in it self at least some communication > > > > was eventually made but it should not have taken a week and about 3 > > > > days of effort on my part to actually get any form of response. > > > > > See the final part of that saga here. > >http://groups.google.com.au/group/google-appengine/browse_thread/thre... > > > > > I am huge advocate of appengine and we are using it more and more, but > > > > the total feeling of helplessness you > > > > get when you have instances non functioning because they can't even > > > > startup for an extended period and > > > > no feedback what so ever makes it very hard to even manage > > > > expectations, and impossible to report to management > > > > about any form of progress in resolving the issue. > > > > > In addition when this happened last time, a journalist from the > > > > Register contacted me about the issue and asked for more information > > > > so they could do a story about the lack of communication from google > > > > on these sorts of issues. I declined, but maybe it does need some > > > > more visibility and least with in the app engine user community. > > > > > Regards > > > > > Tim Hoffman > > > > > On Sep 1, 11:54 pm, Tim Hoffman <[email protected]> wrote: > > > > > > The rate of errors has dropped off somewhat. > > > > > > It is still happening though. > > > > > > T > > > > > > On Sep 1, 7:24 pm, Tim Hoffman <[email protected]> wrote: > > > > > > > Further investigation shows these excessive startup times are being > > > > > > experienced by more than one instance. > > > > > > > Appid's qtrack-dev, q-tracker. > > > > > > > T > > > > > > > On Sep 1, 5:15 pm, Tim Hoffman <[email protected]> wrote: > > > > > > > > Hi > > > > > > > > I am still having trouble getting instances to get past initial > > > > > > > import (appid q-tracker) > > > > > > > > I have even tried starting old versions to get around failed > > imports. > > > > > > > > for the last 10 minutes all startup attempts end in > > DeadlineExceeded > > > > > > > error. > > > > > > > > T > > > > > > > > On Sep 1, 2:45 pm, Tim Hoffman <[email protected]> wrote: > > > > > > > > > Hi > > > > > > > > > As I have pointed out in the past its quite possible to get > > pockets > > > > > > > > of appengine instances experiencing extremely poor performance > > and > > > > > > > > this doesn't show up in the system status. (http:// > > > groups.google.com.au/group/google-appengine/browse_thread/thread/ > > > > > > > > 23e988d494144242/8ff964f7ffcc0b37?q=#8ff964f7ffcc0b37) > > > > > > > > > Apps seemd to be sharded into different parts of appengine > > > > > > > > infrastructure based on appid. > > > > > > > > > I hope this problem is just transitory for tosay and I don't > > have to > > > > > > > > spend days collecting evidence again. > > > > > > > > > T > > > > > > > > > On Sep 1, 2:30 pm, Tim Hoffman <[email protected]> wrote: > > > > > > > > > > Hi > > > > > > > > > > Except it isn't. I am still getting hit with extremeley long > > startup > > > > > > > > > times. > > > > > > > > > I am not even getting around to accessing the datastore. > > Failure is > > > > > > > > > always during instance startup. > > > > > > > > > > T > > > > > > > > > > On Sep 1, 2:27 pm, Andi Albrecht < > > [email protected]> wrote: > > > > > > > > > > > Sorry, I've missed this announcement on Downtime Notify: > > >http://groups.google.com/group/google-appengine-downtime-notify/brows... > > > > > > > > > > > As noted in my previous post, everything's fine now :) > > > > > > > > > > > Andi > > > > > > > > > > > Tim Hoffman <[email protected]> writes: > > > > > > > > > > > Hi > > > > > > > > > > > > I am still seeing them on and off, everything might be > > fine for 20min, > > > > > > > > > > > then bam. > > > > > > > > > > > I have even been having some dashboard failures. > > > > > > > > > > > > T > > > > > > > > > > > > On Sep 1, 1:55 pm, Andi Albrecht < > > [email protected]> wrote: > > > > > > > > > > >> Tim Hoffman <[email protected]> writes: > > > > > > > > > > >> > Hi > > > > > > > > > > > >> > As of today I am getting a lot of DeadlineExceeded > > Errors on a couple > > > > > > > > > > >> > of instances druing startup. > > > > > > > > > > > >> > I am not even getting past the initial imports, > > > > > > > > > > > >> > Nothing is being reported in system status, so it's > > probably > > > > > > > > > > >> > restricted part of the appengine structure. > > > > > > > > > > > >> > Tim > > > > > > > > > > > >> Hi Tim, > > > > > > > > > > > >> do you still see these issues? I've seen a couple of > > different errors a > > > > > > > > > > >> few hours ago, but now everything looks fine again. > > > > > > > > > > > >> Andi > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]<google-appengine%2Bunsubscrib > > [email protected]> > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > Blog:http://googleappengine.blogspot.com > Twitter:http://twitter.com/app_engine > Reddit:http://www.reddit.com/r/appengine -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
