Doh! Google marked this issue as "WontFix" blaming it as always on the M/S datastore. Here's my comment on this issue that will hopefully reach someone who cares:
"You've GOT to be kidding me!! Did you so much as glance at any logs to arrive at this conclusion? Or did you just read the letters "M/S" and decided it must be the datastore? Explain to us how M/S datastore latency also causes memcache latency. You guys seem to use M/S -> HRD migration way too leniently as a red herring explaining any issue that you can't explain otherwise. This makes no sense at all. Why is the M/S data-store even offered as an alternative if it causes so many problems completely unrelated to the datastore itself. Sounds like a cheap trick to force people to move to HRD to satisfy some hidden agenda. I'm definitely sold on the HRD and its value to me as a developer. But attributing such a vast number of issues to M/S without any proper investigation is unfair." On Tue, May 22, 2012 at 4:31 PM, Rishi Arora <[email protected]>wrote: > Have you tried installing appstats? > > > On Tue, May 22, 2012 at 4:27 PM, Waleed Abdulla <[email protected]> wrote: > >> I meant that the extra delays might not be just memcache, but an API >> delay in general because I'm noticing it with the logging service and the >> urlfetch service as well, as shown by the errors I listed. It's really hard >> to pinpoint these things because different apps have different use patterns >> so they might notice the problem in different ways. >> >> >> >> >> On Tue, May 22, 2012 at 1:59 PM, Rishi Arora >> <[email protected]>wrote: >> >>> Perhaps they're related, but Google generally doesn't do anything about >>> datastore deadline exceeded errors. They claim that you should either be >>> using HRD or make your app tolerant of M/S data-store spikes by making >>> asynchronous calls. In my case, I studied appstat output thoroughly over a >>> large sample, and the problem is specifically for memcache API calls. >>> >>> FYI, here's a production issue I just logged, which you should consider >>> starring, so that it gets attention: >>> >>> code.google.com/p/googleappengine/issues/detail?id=7554 >>> >>> >>> >>> On Tue, May 22, 2012 at 3:27 PM, Waleed Abdulla <[email protected]>wrote: >>> >>>> I'm noticing excessive datastore delays today (M/S), and generally a >>>> lot of API calls timing out. Might be related. Errors like: >>>> >>>> *DeadlineExceededError: The API call logservice.Flush() took too long >>>> to respond and was cancelled.* >>>> >>>> Also: >>>> >>>> *<class >>>> 'google.appengine.runtime.apiproxy_errors.DeadlineExceededError'>: The API >>>> call urlfetch.Fetch() took too long to respond and was cancelled.* >>>> >>>> >>>> >>>> >>>> On Tue, May 22, 2012 at 12:04 PM, Gayle Laakmann <[email protected]>wrote: >>>> >>>>> I think I'm having the same issue. >>>>> >>>>> How did you get that look at memcache latency specifically? >>>>> >>>>> On May 22, 9:41 am, Rishi Arora <[email protected]> wrote: >>>>> > Ah, I was mistaken. The restriction is the other way round. Py2.7 >>>>> > requires HRD. >>>>> > >>>>> > On Tue, May 22, 2012 at 11:24 AM, Joshua Smith < >>>>> [email protected]>wrote: >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > > HRD does not require Py2.7 >>>>> > >>>>> > > On May 22, 2012, at 12:01 PM, Rishi Arora wrote: >>>>> > >>>>> > > Also, I imagine going to HRD would indirectly help my situation >>>>> because >>>>> > > I'll be forced to upgrade to python2.7. So, while my instance is >>>>> waiting >>>>> > > forever for memcache to return some data, it would be available to >>>>> service >>>>> > > other requests, and keep the number of active instances low. Am I >>>>> correct >>>>> > > in this reasoning? >>>>> > >>>>> > > On Tue, May 22, 2012 at 10:59 AM, Rishi Arora < >>>>> [email protected]>wrote: >>>>> > >>>>> > >> My app usually consumes ~30 instance hours every day, and >>>>> occasionally >>>>> > >> spikes to ~35 when Master-Slave datastore latencies go up. We >>>>> will be >>>>> > >> transitioning to HRD soon, but today large latency spikes have >>>>> caused my >>>>> > >> instance hours to reach ~50 already, in the first 9 hours of the >>>>> day. I >>>>> > >> peeked at the appstats output to see where this spike is coming >>>>> from, and >>>>> > >> it appears to be memcache (see attached pic). This seems absurd >>>>> and not >>>>> > >> something app operators should be expected to pay for. Anybody >>>>> else seen >>>>> > >> this? >>>>> > >>>>> > >> Thanks in advance >>>>> > >> - Rishi >>>>> > >>>>> > > -- >>>>> > > You received this message because you are subscribed to the Google >>>>> Groups >>>>> > > "Google App Engine" group. >>>>> > > To post to this group, send email to >>>>> [email protected]. >>>>> > > To unsubscribe from this group, send email to >>>>> > > [email protected]. >>>>> > > For more options, visit this group at >>>>> > >http://groups.google.com/group/google-appengine?hl=en. >>>>> > >>>>> > > -- >>>>> > > You received this message because you are subscribed to the Google >>>>> Groups >>>>> > > "Google App Engine" group. >>>>> > > To post to this group, send email to >>>>> [email protected]. >>>>> > > To unsubscribe from this group, send email to >>>>> > > [email protected]. >>>>> > > For more options, visit this group at >>>>> > >http://groups.google.com/group/google-appengine?hl=en. >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Google App Engine" group. >>>>> To post to this group, send email to [email protected] >>>>> . >>>>> To unsubscribe from this group, send email to >>>>> [email protected]. >>>>> For more options, visit this group at >>>>> http://groups.google.com/group/google-appengine?hl=en. >>>>> >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Google App Engine" group. >>>> To post to this group, send email to [email protected]. >>>> To unsubscribe from this group, send email to >>>> [email protected]. >>>> For more options, visit this group at >>>> http://groups.google.com/group/google-appengine?hl=en. >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
