Perhaps they're related, but Google generally doesn't do anything about datastore deadline exceeded errors. They claim that you should either be using HRD or make your app tolerant of M/S data-store spikes by making asynchronous calls. In my case, I studied appstat output thoroughly over a large sample, and the problem is specifically for memcache API calls.
FYI, here's a production issue I just logged, which you should consider starring, so that it gets attention: code.google.com/p/googleappengine/issues/detail?id=7554 On Tue, May 22, 2012 at 3:27 PM, Waleed Abdulla <[email protected]> wrote: > I'm noticing excessive datastore delays today (M/S), and generally a lot > of API calls timing out. Might be related. Errors like: > > *DeadlineExceededError: The API call logservice.Flush() took too long to > respond and was cancelled.* > > Also: > > *<class > 'google.appengine.runtime.apiproxy_errors.DeadlineExceededError'>: The API > call urlfetch.Fetch() took too long to respond and was cancelled.* > > > > > On Tue, May 22, 2012 at 12:04 PM, Gayle Laakmann <[email protected]> wrote: > >> I think I'm having the same issue. >> >> How did you get that look at memcache latency specifically? >> >> On May 22, 9:41 am, Rishi Arora <[email protected]> wrote: >> > Ah, I was mistaken. The restriction is the other way round. Py2.7 >> > requires HRD. >> > >> > On Tue, May 22, 2012 at 11:24 AM, Joshua Smith < >> [email protected]>wrote: >> > >> > >> > >> > >> > >> > >> > >> > > HRD does not require Py2.7 >> > >> > > On May 22, 2012, at 12:01 PM, Rishi Arora wrote: >> > >> > > Also, I imagine going to HRD would indirectly help my situation >> because >> > > I'll be forced to upgrade to python2.7. So, while my instance is >> waiting >> > > forever for memcache to return some data, it would be available to >> service >> > > other requests, and keep the number of active instances low. Am I >> correct >> > > in this reasoning? >> > >> > > On Tue, May 22, 2012 at 10:59 AM, Rishi Arora < >> [email protected]>wrote: >> > >> > >> My app usually consumes ~30 instance hours every day, and >> occasionally >> > >> spikes to ~35 when Master-Slave datastore latencies go up. We will >> be >> > >> transitioning to HRD soon, but today large latency spikes have >> caused my >> > >> instance hours to reach ~50 already, in the first 9 hours of the >> day. I >> > >> peeked at the appstats output to see where this spike is coming >> from, and >> > >> it appears to be memcache (see attached pic). This seems absurd and >> not >> > >> something app operators should be expected to pay for. Anybody else >> seen >> > >> this? >> > >> > >> Thanks in advance >> > >> - Rishi >> > >> > > -- >> > > You received this message because you are subscribed to the Google >> Groups >> > > "Google App Engine" group. >> > > To post to this group, send email to >> [email protected]. >> > > To unsubscribe from this group, send email to >> > > [email protected]. >> > > For more options, visit this group at >> > >http://groups.google.com/group/google-appengine?hl=en. >> > >> > > -- >> > > You received this message because you are subscribed to the Google >> Groups >> > > "Google App Engine" group. >> > > To post to this group, send email to >> [email protected]. >> > > To unsubscribe from this group, send email to >> > > [email protected]. >> > > For more options, visit this group at >> > >http://groups.google.com/group/google-appengine?hl=en. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
