I meant that the extra delays might not be just memcache, but an API delay in general because I'm noticing it with the logging service and the urlfetch service as well, as shown by the errors I listed. It's really hard to pinpoint these things because different apps have different use patterns so they might notice the problem in different ways.
On Tue, May 22, 2012 at 1:59 PM, Rishi Arora <[email protected]>wrote: > Perhaps they're related, but Google generally doesn't do anything about > datastore deadline exceeded errors. They claim that you should either be > using HRD or make your app tolerant of M/S data-store spikes by making > asynchronous calls. In my case, I studied appstat output thoroughly over a > large sample, and the problem is specifically for memcache API calls. > > FYI, here's a production issue I just logged, which you should consider > starring, so that it gets attention: > > code.google.com/p/googleappengine/issues/detail?id=7554 > > > > On Tue, May 22, 2012 at 3:27 PM, Waleed Abdulla <[email protected]> wrote: > >> I'm noticing excessive datastore delays today (M/S), and generally a lot >> of API calls timing out. Might be related. Errors like: >> >> *DeadlineExceededError: The API call logservice.Flush() took too long to >> respond and was cancelled.* >> >> Also: >> >> *<class >> 'google.appengine.runtime.apiproxy_errors.DeadlineExceededError'>: The API >> call urlfetch.Fetch() took too long to respond and was cancelled.* >> >> >> >> >> On Tue, May 22, 2012 at 12:04 PM, Gayle Laakmann <[email protected]>wrote: >> >>> I think I'm having the same issue. >>> >>> How did you get that look at memcache latency specifically? >>> >>> On May 22, 9:41 am, Rishi Arora <[email protected]> wrote: >>> > Ah, I was mistaken. The restriction is the other way round. Py2.7 >>> > requires HRD. >>> > >>> > On Tue, May 22, 2012 at 11:24 AM, Joshua Smith < >>> [email protected]>wrote: >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > > HRD does not require Py2.7 >>> > >>> > > On May 22, 2012, at 12:01 PM, Rishi Arora wrote: >>> > >>> > > Also, I imagine going to HRD would indirectly help my situation >>> because >>> > > I'll be forced to upgrade to python2.7. So, while my instance is >>> waiting >>> > > forever for memcache to return some data, it would be available to >>> service >>> > > other requests, and keep the number of active instances low. Am I >>> correct >>> > > in this reasoning? >>> > >>> > > On Tue, May 22, 2012 at 10:59 AM, Rishi Arora < >>> [email protected]>wrote: >>> > >>> > >> My app usually consumes ~30 instance hours every day, and >>> occasionally >>> > >> spikes to ~35 when Master-Slave datastore latencies go up. We will >>> be >>> > >> transitioning to HRD soon, but today large latency spikes have >>> caused my >>> > >> instance hours to reach ~50 already, in the first 9 hours of the >>> day. I >>> > >> peeked at the appstats output to see where this spike is coming >>> from, and >>> > >> it appears to be memcache (see attached pic). This seems absurd >>> and not >>> > >> something app operators should be expected to pay for. Anybody >>> else seen >>> > >> this? >>> > >>> > >> Thanks in advance >>> > >> - Rishi >>> > >>> > > -- >>> > > You received this message because you are subscribed to the Google >>> Groups >>> > > "Google App Engine" group. >>> > > To post to this group, send email to >>> [email protected]. >>> > > To unsubscribe from this group, send email to >>> > > [email protected]. >>> > > For more options, visit this group at >>> > >http://groups.google.com/group/google-appengine?hl=en. >>> > >>> > > -- >>> > > You received this message because you are subscribed to the Google >>> Groups >>> > > "Google App Engine" group. >>> > > To post to this group, send email to >>> [email protected]. >>> > > To unsubscribe from this group, send email to >>> > > [email protected]. >>> > > For more options, visit this group at >>> > >http://groups.google.com/group/google-appengine?hl=en. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
