Can the other applications send me their application IDs? We'll investigate what's happening here.
Pedro, thanks for your feedback. I'm going to go ahead and close your issue. We have a long term solution in the works, which is to have a more intelligent automatic migration off a hot instance. On Tue, Feb 23, 2010 at 4:14 PM, Pedro Morais <[email protected]>wrote: > Thanks! > The migration has fixed the problem, error rates seem to be back to > normal. > > The problem started occurring 2 days ago; I would say the error rates > had a strong correlation with the latency chart at > > http://code.google.com/status/appengine/detail/serving/2010/02/23#ae-trust-detail-helloworld-get-latency > > The symptoms I was having were the same as in > http://code.google.com/p/googleappengine/issues/detail?id=1695 (see > comment 10, 13 and 14, from September 16). > > Now, on both occasions, I spot a pattern: the DeadlineException kicks > in correctly at about 30 seconds, but even though my app is not > sleeping the used CPU is always pretty low (in the order of two > seconds; on September 16 I was seeing only 1 second). > > From this observation, a brute force approach to detecting the problem > could be having watchdog process on a busy loop running on each shard > for 5 seconds (for example); if the watchdog is killed by the 30 > seconds deadline you probably have a starvation problem. > > RPC does not seem to be the problem here, as I think my app timed-out > before it had a chance to do any RPC calls. > > Also, for debugging purposes maybe the current shards information > could be exposed somewhere on the admin interface. > > Regards, and thanks again, > Pedro Morais > > > On Feb 23, 10:59 pm, "Ikai L (Google)" <[email protected]> wrote: > > Pedro, > > > > I've migrated your application. Can you check and report back what is > > happening? If your application errors at a rate that is more closer to > > normal parameters, then we have an issue with a hotspot. Do you know if > the > > errors increased more at certain times in the day, or were the increased > > number of errors at all times? > > > > > > > > > > > > On Tue, Feb 23, 2010 at 2:47 PM, John Gardner <[email protected]> wrote: > > > I'm seeing the same thing on one of my apps. The same codebase with > > > another appid appears to be fine. Is an appid tied to certain > > > instances, and some groups of instances are dead/dying? > > > > > On Feb 23, 1:46 pm, Pedro Morais <[email protected]> wrote: > > > > Can anyone from Google take a look at the issue please? > > > > > > My application is basically unusable right now. > > > > > > Regards, > > > > Pedro Morais > > > > > > On Feb 23, 3:32 pm, Pedro Morais <[email protected]> wrote: > > > > > > > Hi, > > > > > > > Looking athttp:// > > > code.google.com/status/appengine/detail/serving/2010/02/22#ae-... > > > > > you can see that there was a spike yesterday in request serving > > > > > latency. > > > > > > > The comment says "Investigation Complete - Issue Resolved > > > > > We have determined that this spike did not affect the performance > or > > > > > uptime of applications. If you feel we have incorrectly diagnosed > this > > > > > issue please inform us by posting in our developer forum." > > > > > > > Well, it did affect the performance and uptime of at least one of > my > > > > > (Python) applications, as reported in the following production > issue > > > > > (still in the new state): > > >http://code.google.com/p/googleappengine/issues/detail?id=2837 > > > > > > > Today, we are seeing the same spike (right now the chart is in a > spike > > > > > before 8AM and at about 650): > > >http://code.google.com/status/appengine/detail/serving/2010/02/23#ae-. > .. > > > > > > > And again, my app is having a lot of Deadline exceptions. > > > > > > > Looking at the logs, the thing that I think is most relevant is > that > > > > > my app only gets to spend about 2000 CPU ms before de 30000 ms > period > > > > > finished - so it's severely starved for resources - so starved that > it > > > > > doesn't even manage to load Django 1.1 before the Deadline kicks > in. > > > > > > > Regards, > > > > > Pedro Morais > > > > > -- > > > You received this message because you are subscribed to the Google > Groups > > > "Google App Engine" group. > > > To post to this group, send email to [email protected] > . > > > To unsubscribe from this group, send email to > > > [email protected]<google-appengine%[email protected]><google-appengine%2Bunsubscrib > [email protected]> > > > . > > > For more options, visit this group at > > >http://groups.google.com/group/google-appengine?hl=en. > > > > -- > > Ikai Lan > > Developer Programs Engineer, Google App Enginehttp:// > googleappengine.blogspot.com|http://twitter.com/app_engine > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<google-appengine%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- Ikai Lan Developer Programs Engineer, Google App Engine http://googleappengine.blogspot.com | http://twitter.com/app_engine -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
