I'm seeing the same issue, also beginning on May 22. We did an emergency upgrade to the HR datastore this morning in hopes that this might fix our problem. Unsurprisingly, the memcache latency problem persist.
BTW, it's not all memcache requests that are slow. Sometimes everything will go back to being fine for up to an hour. On May 22, 11:53 pm, Rishi Arora <[email protected]> wrote: > Doh! Google marked this issue as "WontFix" blaming it as always on the M/S > datastore. Here's my comment on this issue that will hopefully reach > someone who cares: > > "You've GOT to be kidding me!! Did you so much as glance at any logs to > arrive at this conclusion? Or did you just read the letters "M/S" and > decided it must be the datastore? Explain to us how M/S datastore latency > also causes memcache latency. You guys seem to use M/S -> HRD migration way > too leniently as a red herring explaining any issue that you can't explain > otherwise. This makes no sense at all. Why is the M/S data-store even > offered as an alternative if it causes so many problems completely > unrelated to the datastore itself. Sounds like a cheap trick to force > people to move to HRD to satisfy some hidden agenda. I'm definitely sold on > the HRD and its value to me as a developer. But attributing such a vast > number of issues to M/S without any proper investigation is unfair." > > On Tue, May 22, 2012 at 4:31 PM, Rishi Arora <[email protected]>wrote: > > > > > > > > > Have you tried installing appstats? > > > On Tue, May 22, 2012 at 4:27 PM, Waleed Abdulla <[email protected]> wrote: > > >> I meant that the extra delays might not be just memcache, but an API > >> delay in general because I'm noticing it with the logging service and the > >> urlfetch service as well, as shown by the errors I listed. It's really hard > >> to pinpoint these things because different apps have different use patterns > >> so they might notice the problem in different ways. > > >> On Tue, May 22, 2012 at 1:59 PM, Rishi Arora > >> <[email protected]>wrote: > > >>> Perhaps they're related, but Google generally doesn't do anything about > >>> datastore deadline exceeded errors. They claim that you should either be > >>> using HRD or make your app tolerant of M/S data-store spikes by making > >>> asynchronous calls. In my case, I studied appstat output thoroughly over > >>> a > >>> large sample, and the problem is specifically for memcache API calls. > > >>> FYI, here's a production issue I just logged, which you should consider > >>> starring, so that it gets attention: > > >>> code.google.com/p/googleappengine/issues/detail?id=7554 > > >>> On Tue, May 22, 2012 at 3:27 PM, Waleed Abdulla <[email protected]>wrote: > > >>>> I'm noticing excessive datastore delays today (M/S), and generally a > >>>> lot of API calls timing out. Might be related. Errors like: > > >>>> *DeadlineExceededError: The API call logservice.Flush() took too long > >>>> to respond and was cancelled.* > > >>>> Also: > > >>>> *<class > >>>> 'google.appengine.runtime.apiproxy_errors.DeadlineExceededError'>: The > >>>> API > >>>> call urlfetch.Fetch() took too long to respond and was cancelled.* > > >>>> On Tue, May 22, 2012 at 12:04 PM, Gayle Laakmann <[email protected]>wrote: > > >>>>> I think I'm having the same issue. > > >>>>> How did you get that look at memcache latency specifically? > > >>>>> On May 22, 9:41 am, Rishi Arora <[email protected]> wrote: > >>>>> > Ah, I was mistaken. The restriction is the other way round. Py2.7 > >>>>> > requires HRD. > > >>>>> > On Tue, May 22, 2012 at 11:24 AM, Joshua Smith < > >>>>> [email protected]>wrote: > > >>>>> > > HRD does not require Py2.7 > > >>>>> > > On May 22, 2012, at 12:01 PM, Rishi Arora wrote: > > >>>>> > > Also, I imagine going to HRD would indirectly help my situation > >>>>> because > >>>>> > > I'll be forced to upgrade to python2.7. So, while my instance is > >>>>> waiting > >>>>> > > forever for memcache to return some data, it would be available to > >>>>> service > >>>>> > > other requests, and keep the number of active instances low. Am I > >>>>> correct > >>>>> > > in this reasoning? > > >>>>> > > On Tue, May 22, 2012 at 10:59 AM, Rishi Arora < > >>>>> [email protected]>wrote: > > >>>>> > >> My app usually consumes ~30 instance hours every day, and > >>>>> occasionally > >>>>> > >> spikes to ~35 when Master-Slave datastore latencies go up. We > >>>>> will be > >>>>> > >> transitioning to HRD soon, but today large latency spikes have > >>>>> caused my > >>>>> > >> instance hours to reach ~50 already, in the first 9 hours of the > >>>>> day. I > >>>>> > >> peeked at the appstats output to see where this spike is coming > >>>>> from, and > >>>>> > >> it appears to be memcache (see attached pic). This seems absurd > >>>>> and not > >>>>> > >> something app operators should be expected to pay for. Anybody > >>>>> else seen > >>>>> > >> this? > > >>>>> > >> Thanks in advance > >>>>> > >> - Rishi > > >>>>> > > -- > >>>>> > > You received this message because you are subscribed to the Google > >>>>> Groups > >>>>> > > "Google App Engine" group. > >>>>> > > To post to this group, send email to > >>>>> [email protected]. > >>>>> > > To unsubscribe from this group, send email to > >>>>> > > [email protected]. > >>>>> > > For more options, visit this group at > >>>>> > >http://groups.google.com/group/google-appengine?hl=en. > > >>>>> > > -- > >>>>> > > You received this message because you are subscribed to the Google > >>>>> Groups > >>>>> > > "Google App Engine" group. > >>>>> > > To post to this group, send email to > >>>>> [email protected]. > >>>>> > > To unsubscribe from this group, send email to > >>>>> > > [email protected]. > >>>>> > > For more options, visit this group at > >>>>> > >http://groups.google.com/group/google-appengine?hl=en. > > >>>>> -- > >>>>> You received this message because you are subscribed to the Google > >>>>> Groups "Google App Engine" group. > >>>>> To post to this group, send email to [email protected] > >>>>> . > >>>>> To unsubscribe from this group, send email to > >>>>> [email protected]. > >>>>> For more options, visit this group at > >>>>>http://groups.google.com/group/google-appengine?hl=en. > > >>>> -- > >>>> You received this message because you are subscribed to the Google > >>>> Groups "Google App Engine" group. > >>>> To post to this group, send email to [email protected]. > >>>> To unsubscribe from this group, send email to > >>>> [email protected]. > >>>> For more options, visit this group at > >>>>http://groups.google.com/group/google-appengine?hl=en. > > >>> -- > >>> You received this message because you are subscribed to the Google > >>> Groups "Google App Engine" group. > >>> To post to this group, send email to [email protected]. > >>> To unsubscribe from this group, send email to > >>> [email protected]. > >>> For more options, visit this group at > >>>http://groups.google.com/group/google-appengine?hl=en. > > >> -- > >> You received this message because you are subscribed to the Google Groups > >> "Google App Engine" group. > >> To post to this group, send email to [email protected]. > >> To unsubscribe from this group, send email to > >> [email protected]. > >> For more options, visit this group at > >>http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
