We are also experiencing very high query latency, sometimes enough that a non-compound query that matches one datastore entity will not complete after 30s and all our available serving instances get eaten up. We're going to try adding deadlines for all queries today and see if that saves our application instances.
On Jun 1, 7:22 pm, Millisecond <[email protected]> wrote: > Since I posted that my dashboard has been wonky and the errors > counting _backwards_. > > So, there are only ~2k errors at the moment, but there definitely were > ~3500 when I first posted. > > Really don't think I'm going crazy but I do feel like it with the > dashboard at times... > > -C > > On Jun 1, 3:41 pm, Millisecond <[email protected]> wrote: > > > > > Yes, app id noticeorange. 3500+ latency related errors today in 2 > > major and ~10 minor spikes. > > > Most days since the 25th have had 5k+ errors, with several 20k+. > > > It's usually related to ms/req spikes coinciding with the high latency > > spikes on the query latency graph. > > > -C > > > On Jun 1, 3:34 pm, "Ikai L (Google)" <[email protected]> wrote: > > > > The post-mortem should address the outage as well as the latency leading > > > up > > > to the outage. > > > > We're keeping an eye on latency now. Are you experience latency in your > > > application? > > > > On Tue, Jun 1, 2010 at 3:31 PM, Millisecond <[email protected]> wrote: > > > > Thanks Ikai, I look forward to reading the post-mortem. The last one > > > > was indeed very good. > > > > > Will the post-mortem address the dramatically increased latency since > > > > the 25th? It's ongoing and has already caused tens of thousands of > > > > errors on our site alone. As far as I can tell, there's been no > > > > official communication regarding that. > > > > > And it's still continuing with 1500ms+ for a lot of today: > > > > >http://code.google.com/status/appengine/detail/datastore/2010/06/01#a... > > > > > -C > > > > > On Jun 1, 3:16 pm, "Ikai L (Google)" <[email protected]> wrote: > > > > > I just wanted to let you guys know to hang in there. We're working on > > > > > a > > > > > post-mortem describing what went wrong and steps we are taking to > > > > > prevent > > > > > this from happening again. I don't have an ETA for this yet, but we've > > > > done > > > > > this before ( > > > >http://groups.google.com/group/google-appengine/browse_thread/thread/...) > > > > > and are committed to as much transparency as we can with regards to > > > > > these > > > > > issues with you guys. > > > > > > On Tue, Jun 1, 2010 at 2:59 PM, Mark <[email protected]> wrote: > > > > > > +1 as worried, I'm investing time in app engine and this is quite > > > > > > discouraging to say the least! How often does this happen? (I don't > > > > > > have a live app yet), but should I abandon app engine now before > > > > > > getting myself in trouble? > > > > > > > On May 27, 1:32 pm, Jody Belka <[email protected]> wrote: > > > > > > > And what do you know, no-one from Google has made any response > > > > whatsoever > > > > > > to > > > > > > > this thread. And yet we know they are around, they occasionally do > > > > write > > > > > > > messages after all. Very disappointing. > > > > > > > > On 25 May 2010 21:51, Waleed Abdulla <[email protected]> wrote: > > > > > > > > > It's definitely worrisome. My app is down at the moment, my task > > > > queue > > > > > > > > backlog is at 4 days, and I'm very disappointed. > > > > > > > > > On Tue, May 25, 2010 at 12:46 PM, Flips > > > > > > > > <[email protected]> > > > > > > wrote: > > > > > > > > >> These latency spikes are really annoying. Will the business app > > > > engine > > > > > > > >> edition use the same bigtable cluster as we do? > > > > > > > > >> On May 25, 9:03 pm, James <[email protected]> wrote: > > > > > > > >> > +1 > > > > > > > > >> > On May 25, 10:13 am, Jody Belka <[email protected]> wrote:> From > > > > > > reading > > > > > > > >> this list, there appear to have been quite few incidents lately > > > > > > > >> > > where multiple sites start reporting errors, with the > > > > > > > >> > > status > > > > page > > > > > > > >> showing > > > > > > > > >> > ... > > > > > > > > >> -- > > > > > > > >> You received this message because you are subscribed to the > > > > > > > >> Google > > > > > > Groups > > > > > > > >> "Google App Engine" group. > > > > > > > >> To post to this group, send email to > > > > > > [email protected]. > > > > > > > >> To unsubscribe from this group, send email to > > > > > > > >> [email protected]<google-appengine%2Bunsubscrib > > > > > > > >> [email protected]><google-appengine%2Bunsubscrib > > > > [email protected]><google-appengine%2Bunsubscrib > > > > > > [email protected]> > > > > > > > >> . > > > > > > > >> For more options, visit this group at > > > > > > > >>http://groups.google.com/group/google-appengine?hl=en. > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to the > > > > > > > > Google > > > > > > Groups > > > > > > > > "Google App Engine" group. > > > > > > > > To post to this group, send email to > > > > [email protected] > > > > > > . > > > > > > > > To unsubscribe from this group, send email to > > > > > > > > [email protected]<google-appengine%2Bunsubscrib > > > > > > > > [email protected]><google-appengine%2Bunsubscrib > > > > [email protected]><google-appengine%2Bunsubscrib > > > > > > [email protected]> > > > > > > > > . > > > > > > > > For more options, visit this group at > > > > > > > >http://groups.google.com/group/google-appengine?hl=en. > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > Groups > > > > > > "Google App Engine" group. > > > > > > To post to this group, send email to > > > > > > [email protected] > > > > . > > > > > > To unsubscribe from this group, send email to > > > > > > [email protected]<google-appengine%2Bunsubscrib > > > > > > [email protected]><google-appengine%2Bunsubscrib > > > > [email protected]> > > > > > > . > > > > > > For more options, visit this group at > > > > > >http://groups.google.com/group/google-appengine?hl=en. > > > > > > -- > > > > > Ikai Lan > > > > > Developer Programs Engineer, Google App Engine > > > > > Blog:http://googleappengine.blogspot.com > > > > > Twitter:http://twitter.com/app_engine > > > > > Reddit:http://www.reddit.com/r/appengine > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups > > > > "Google App Engine" group. > > > > To post to this group, send email to [email protected]. > > > > To unsubscribe from this group, send email to > > > > [email protected]<google-appengine%2Bunsubscrib > > > > [email protected]> > > > > . > > > > For more options, visit this group at > > > >http://groups.google.com/group/google-appengine?hl=en. > > > > -- > > > Ikai Lan > > > Developer Programs Engineer, Google App Engine > > > Blog:http://googleappengine.blogspot.com > > > Twitter:http://twitter.com/app_engine > > > Reddit:http://www.reddit.com/r/appengine -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
