Thanks for reporting guys! Ideally, the memcache service should have steady latencies, but unfortunately it is not as stable as HR datastore. Additionally, our SLA is not covering the memcache service now. In general, we're working hard for improving stability of our service. However, of course we prioritize SLA-covered serviced first.
Thus, you may want to file a feature request for making the memcache service covered by SLA. That way, others can show their interests/business needs by staring it, and if there's many stars on the issue, our engineering team will eventually prioritize providing SLA-covered memcache service. For the time being, you can put appropriate deadline on your memcache call to prevent those kind of latency problems happening. Python memcache Client class has some methods like get_multi_async and set_multi_async that you can put a deadline(by specifying an rpc object). For more details, please see: https://developers.google.com/appengine/docs/python/memcache/clientclass#Client_get_multi_async -- Takashi On Wed, May 23, 2012 at 6:36 AM, Rishi Arora <[email protected]> wrote: > Here's my appstats output for a specific request showing my problem is > squarely related to memcache: > > > > > On Tue, May 22, 2012 at 4:33 PM, Rishi Arora <[email protected]> > wrote: >> >> Hmmm. That means the problem isn't specific to memcache. And that means >> it might take longer for Google to fix this, if indeed they determine >> there's a problem. >> >> >> On Tue, May 22, 2012 at 4:15 PM, Robert Morgan <[email protected]> >> wrote: >>> >>> My graph is attached -- interesting, eh? >>> I don't think we're using memcache in this instance. >>> >>> :R >>> >>> >>> On Tue, May 22, 2012 at 2:04 PM, Rishi Arora <[email protected]> >>> wrote: >>>> >>>> Here's what my milliseconds/request looks like, although the #instances >>>> follows a similar pattern, and translates directly into higher costs (5x >>>> normal usage today) >>>> >>>> >>>> On Tue, May 22, 2012 at 4:01 PM, Rishi Arora <[email protected]> >>>> wrote: >>>>> >>>>> Perhaps similar to what I'm seeing. Although I was able to use the >>>>> appstats tool to determine that the deadlines were being missed >>>>> specifically >>>>> because of memcache API calls. Perhaps you can determine from appstats if >>>>> its the same for you? >>>>> >>>>> If so, please star this issue that I logged: >>>>> >>>>> code.google.com/p/googleappengine/issues/detail?id=7554 >>>>> >>>>> >>>>> >>>>> On Tue, May 22, 2012 at 2:02 PM, Robert Morgan <[email protected]> >>>>> wrote: >>>>>> >>>>>> Beginning about 18 hours ago, the milliseconds/Request on one of our >>>>>> apps began to ramp and is now over 100 seconds for a trivial response >>>>>> (which >>>>>> historically takes 2 seconds). Our other apps are fine, and the AE status >>>>>> page is clear. >>>>>> >>>>>> It's an app that we use for development, it's been around for ages and >>>>>> still runs the Master-Slave datastore. Python. The ramping just started >>>>>> -- >>>>>> we had not deployed a new server version in about a week, and our client >>>>>> usage remains low and patterns haven't changed. >>>>>> >>>>>> I've poked at it trying to see if I can determine some threshold, but >>>>>> even a very simple return now fails due to DeadlineExceeded. >>>>>> >>>>>> This really feels like it's a system issue -- any ideas on what I can >>>>>> do next? >>>>>> >>>>>> Thanks, >>>>>> :R >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Google App Engine" group. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msg/google-appengine/-/rYY5Gw9UuE8J. >>>>>> To post to this group, send email to >>>>>> [email protected]. >>>>>> To unsubscribe from this group, send email to >>>>>> [email protected]. >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/group/google-appengine?hl=en. >>>>> >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Google App Engine" group. >>>> To post to this group, send email to [email protected]. >>>> To unsubscribe from this group, send email to >>>> [email protected]. >>>> For more options, visit this group at >>>> http://groups.google.com/group/google-appengine?hl=en. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >> >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. -- Takashi Matsuo | Developer Advocate | [email protected] -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
