Have you tried installing appstats? On Tue, May 22, 2012 at 4:27 PM, Waleed Abdulla <[email protected]> wrote:
> I meant that the extra delays might not be just memcache, but an API delay > in general because I'm noticing it with the logging service and the > urlfetch service as well, as shown by the errors I listed. It's really hard > to pinpoint these things because different apps have different use patterns > so they might notice the problem in different ways. > > > > > On Tue, May 22, 2012 at 1:59 PM, Rishi Arora <[email protected]>wrote: > >> Perhaps they're related, but Google generally doesn't do anything about >> datastore deadline exceeded errors. They claim that you should either be >> using HRD or make your app tolerant of M/S data-store spikes by making >> asynchronous calls. In my case, I studied appstat output thoroughly over a >> large sample, and the problem is specifically for memcache API calls. >> >> FYI, here's a production issue I just logged, which you should consider >> starring, so that it gets attention: >> >> code.google.com/p/googleappengine/issues/detail?id=7554 >> >> >> >> On Tue, May 22, 2012 at 3:27 PM, Waleed Abdulla <[email protected]> wrote: >> >>> I'm noticing excessive datastore delays today (M/S), and generally a lot >>> of API calls timing out. Might be related. Errors like: >>> >>> *DeadlineExceededError: The API call logservice.Flush() took too long >>> to respond and was cancelled.* >>> >>> Also: >>> >>> *<class >>> 'google.appengine.runtime.apiproxy_errors.DeadlineExceededError'>: The API >>> call urlfetch.Fetch() took too long to respond and was cancelled.* >>> >>> >>> >>> >>> On Tue, May 22, 2012 at 12:04 PM, Gayle Laakmann <[email protected]>wrote: >>> >>>> I think I'm having the same issue. >>>> >>>> How did you get that look at memcache latency specifically? >>>> >>>> On May 22, 9:41 am, Rishi Arora <[email protected]> wrote: >>>> > Ah, I was mistaken. The restriction is the other way round. Py2.7 >>>> > requires HRD. >>>> > >>>> > On Tue, May 22, 2012 at 11:24 AM, Joshua Smith < >>>> [email protected]>wrote: >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > > HRD does not require Py2.7 >>>> > >>>> > > On May 22, 2012, at 12:01 PM, Rishi Arora wrote: >>>> > >>>> > > Also, I imagine going to HRD would indirectly help my situation >>>> because >>>> > > I'll be forced to upgrade to python2.7. So, while my instance is >>>> waiting >>>> > > forever for memcache to return some data, it would be available to >>>> service >>>> > > other requests, and keep the number of active instances low. Am I >>>> correct >>>> > > in this reasoning? >>>> > >>>> > > On Tue, May 22, 2012 at 10:59 AM, Rishi Arora < >>>> [email protected]>wrote: >>>> > >>>> > >> My app usually consumes ~30 instance hours every day, and >>>> occasionally >>>> > >> spikes to ~35 when Master-Slave datastore latencies go up. We >>>> will be >>>> > >> transitioning to HRD soon, but today large latency spikes have >>>> caused my >>>> > >> instance hours to reach ~50 already, in the first 9 hours of the >>>> day. I >>>> > >> peeked at the appstats output to see where this spike is coming >>>> from, and >>>> > >> it appears to be memcache (see attached pic). This seems absurd >>>> and not >>>> > >> something app operators should be expected to pay for. Anybody >>>> else seen >>>> > >> this? >>>> > >>>> > >> Thanks in advance >>>> > >> - Rishi >>>> > >>>> > > -- >>>> > > You received this message because you are subscribed to the Google >>>> Groups >>>> > > "Google App Engine" group. >>>> > > To post to this group, send email to >>>> [email protected]. >>>> > > To unsubscribe from this group, send email to >>>> > > [email protected]. >>>> > > For more options, visit this group at >>>> > >http://groups.google.com/group/google-appengine?hl=en. >>>> > >>>> > > -- >>>> > > You received this message because you are subscribed to the Google >>>> Groups >>>> > > "Google App Engine" group. >>>> > > To post to this group, send email to >>>> [email protected]. >>>> > > To unsubscribe from this group, send email to >>>> > > [email protected]. >>>> > > For more options, visit this group at >>>> > >http://groups.google.com/group/google-appengine?hl=en. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Google App Engine" group. >>>> To post to this group, send email to [email protected]. >>>> To unsubscribe from this group, send email to >>>> [email protected]. >>>> For more options, visit this group at >>>> http://groups.google.com/group/google-appengine?hl=en. >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
