[google-appengine] Re: Memcache API Timing out
Hi Gayle, Would you mind sending either your app_id or a stack trace to [EMAIL PROTECTED] We haven't seen this globally and would like to look into it. Thanks! Pete, App Engine Team On Nov 13, 9:29 am, Gayle Laakmann [EMAIL PROTECTED] wrote: Despite what the post at the link below says, memcache is actually timing out and throwing a DeadlineExceededError. Furthermore, it hardly seems accurate to say that our apps should continue serving normally. The quota limits are so ridiculously low that we CAN'T serve pages without caching. http://groups.google.com/group/google-appengine-downtime-notify/brows... We will be taking memcache offline tomorrow morning from 9-10am PST (GMT-8) for routine maintenance. Calls to the memcache API will *not* throw exceptions but will instead return false for set() calls and None for get() calls (just like any other cache miss.) Your app should continue serving normally during this period, and we'll keep you updated on our progress. Additonally, I'd like to make two suggestions: 1. If you're going to take down memcache, wouldn't it make sense to remove the quota limits? It hardly seems fair to penalized for exceeding quota when caching is disabled. 2. If you have to take down our apps for an hour, can you pick something like 2am - 3am? I know you don't want to go to work at 2am, but it's not really ok to take down our apps for an hour during prime- time. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Memcache API Timing out
Pete, Please also check this post: http://groups.google.com/group/google-appengine/browse_thread/thread/d8a3a54df62e1b04 I guess, you've messed something up with versions. Denis --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Memcache API Timing out
Hi Gayle, I'm sorry to hear you're seeing issues. It's important to note that there are two types of DeadlineExceededErrors. One is defined in apiproxy_errors.py. The other is in runtime/__init__.py: http://code.google.com/p/googleappengine/source/browse/trunk/google/appengine/runtime/__init__.py The names are the same, which makes this confusing. The latter error is raised when the request has been processing for too long (~10 seconds). Perhaps what's happening here is that the cache misses are causing a lot of extra work to be done, causing your application to hit this other deadline. The traceback sent by Pratham confirms this behavior. More detail on how to deal with this is here, in the section entitled The Request Timer: http://code.google.com/appengine/docs/python/requestsandcgi.html -Brett On Nov 13, 9:29 am, Gayle Laakmann [EMAIL PROTECTED] wrote: Despite what the post at the link below says, memcache is actually timing out and throwing a DeadlineExceededError. Furthermore, it hardly seems accurate to say that our apps should continue serving normally. The quota limits are so ridiculously low that we CAN'T serve pages without caching. http://groups.google.com/group/google-appengine-downtime-notify/brows... We will be taking memcache offline tomorrow morning from 9-10am PST (GMT-8) for routine maintenance. Calls to the memcache API will *not* throw exceptions but will instead return false for set() calls and None for get() calls (just like any other cache miss.) Your app should continue serving normally during this period, and we'll keep you updated on our progress. Additonally, I'd like to make two suggestions: 1. If you're going to take down memcache, wouldn't it make sense to remove the quota limits? It hardly seems fair to penalized for exceeding quota when caching is disabled. 2. If you have to take down our apps for an hour, can you pick something like 2am - 3am? I know you don't want to go to work at 2am, but it's not really ok to take down our apps for an hour during prime- time. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Memcache API Timing out
They're timing out for me as well
App-ID - lifesizedpicx
class 'google.appengine.runtime.DeadlineExceededError':
Traceback (most recent call last):
File /base/data/home/apps/lifesizedpicx/4.26/home.py, line 1316,
in main
run_wsgi_app (application)
File /base/python_lib/versions/1/google/appengine/ext/webapp/
util.py, line 76, in run_wsgi_app
result = application(env, _start_response)
File /base/python_lib/versions/1/google/appengine/ext/webapp/
__init__.py, line 499, in __call__
handler.get(*groups)
File /base/data/home/apps/lifesizedpicx/4.26/home.py, line 1078,
in get
memcache.set ('pic'+id, pix, CACHE_EXPIRE)
File /base/python_lib/versions/1/google/appengine/api/memcache/
__init__.py, line 539, in set
return self._set_with_policy(MemcacheSetRequest.SET, key, value,
time=time)
File /base/python_lib/versions/1/google/appengine/api/memcache/
__init__.py, line 609, in _set_with_policy
self._make_sync_call('memcache', 'Set', request, response)
File /base/python_lib/versions/1/google/appengine/api/
apiproxy_stub_map.py, line 46, in MakeSyncCall
stub.MakeSyncCall(service, call, request, response)
File /base/python_lib/versions/1/google/appengine/runtime/
apiproxy.py, line 245, in MakeSyncCall
rpc.Wait()
File /base/python_lib/versions/1/google/appengine/runtime/
apiproxy.py, line 161, in Wait
rpc_completed = _apphosting_runtime___python__apiproxy.Wait(self)
On Nov 13, 10:37 pm, Pete Koomen [EMAIL PROTECTED] wrote:
Hi Gayle,
Would you mind sending either your app_id or a stack trace to
[EMAIL PROTECTED] We haven't seen this globally and would like to
look into it.
Thanks!
Pete, App Engine Team
On Nov 13, 9:29 am, Gayle Laakmann [EMAIL PROTECTED] wrote:
Despite what the post at the link below says, memcache is actually
timing out and throwing a DeadlineExceededError. Furthermore, it
hardly seems accurate to say that our apps should continue serving
normally. The quota limits are so ridiculously low that we CAN'T
serve pages without caching.
http://groups.google.com/group/google-appengine-downtime-notify/brows...
We will be taking memcache offline tomorrow morning from 9-10am PST
(GMT-8) for routine maintenance. Calls to the memcache API will *not*
throw exceptions but will instead return false for set() calls and
None for get() calls (just like any other cache miss.)
Your app should continue serving normally during this period, and
we'll keep you updated on our progress.
Additonally, I'd like to make two suggestions:
1. If you're going to take down memcache, wouldn't it make sense to
remove the quota limits? It hardly seems fair to penalized for
exceeding quota when caching is disabled.
2. If you have to take down our apps for an hour, can you pick
something like 2am - 3am? I know you don't want to go to work at 2am,
but it's not really ok to take down our apps for an hour during prime-
time.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: Memcache API Timing out
I second Gayle's points entirely. We're getting hit with a huge number of deadline exceeded errors at the moment, all due more or less to cache misses. Site is basically unusable for our users. Gee On Nov 13, 9:29 am, Gayle Laakmann [EMAIL PROTECTED] wrote: Despite what the post at the link below says, memcache is actually timing out and throwing a DeadlineExceededError. Furthermore, it hardly seems accurate to say that our apps should continue serving normally. The quota limits are so ridiculously low that we CAN'T serve pages without caching. http://groups.google.com/group/google-appengine-downtime-notify/brows... We will be taking memcache offline tomorrow morning from 9-10am PST (GMT-8) for routine maintenance. Calls to the memcache API will *not* throw exceptions but will instead return false for set() calls and None for get() calls (just like any other cache miss.) Your app should continue serving normally during this period, and we'll keep you updated on our progress. Additonally, I'd like to make two suggestions: 1. If you're going to take down memcache, wouldn't it make sense to remove the quota limits? It hardly seems fair to penalized for exceeding quota when caching is disabled. 2. If you have to take down our apps for an hour, can you pick something like 2am - 3am? I know you don't want to go to work at 2am, but it's not really ok to take down our apps for an hour during prime- time. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Memcache API Timing out
It was timing out within memcache.get and memcache.set -
consistently. If I understand you correctly - you're saying that
memcache isn't technically timing out, on the grounds that it would
complete if given enough time. But, the cache misses take SO much
time that it consistently timeouts during the operations of
memcache.get and memcache.set.
Ok, sure, that may be the case. I'm afraid I don't really see the
difference between that and timing out. Either way, memcache.get and
memcache.set aren't exactly just return false and letting your
application proceed. The fact is they are causing timeouts.
obj = memcache.get(cache_key)
File /base/data/home/apps/careercup/13.329261908074364999/memcache/
__init__.py, line 362, in get
self._make_sync_call('memcache', 'Get', request, response)
File /base/python_lib/versions/1/google/appengine/api/
apiproxy_stub_map.py, line 46, in MakeSyncCall
stub.MakeSyncCall(service, call, request, response)
File /base/python_lib/versions/1/google/appengine/runtime/
apiproxy.py, line 245, in MakeSyncCall
rpc.Wait()
File /base/python_lib/versions/1/google/appengine/runtime/
apiproxy.py, line 161, in Wait
rpc_completed = _apphosting_runtime___python__apiproxy.Wait(self)
On Nov 13, 9:45 am, Brett [EMAIL PROTECTED] wrote:
Hi Gayle,
I'm sorry to hear you're seeing issues. It's important to note that
there are two types of DeadlineExceededErrors. One is defined in
apiproxy_errors.py. The other is in runtime/__init__.py:
http://code.google.com/p/googleappengine/source/browse/trunk/google/a...
The names are the same, which makes this confusing. The latter error
is raised when the request has been processing for too long (~10
seconds). Perhaps what's happening here is that the cache misses are
causing a lot of extra work to be done, causing your application to
hit this other deadline.
The traceback sent by Pratham confirms this behavior. More detail on
how to deal with this is here, in the section entitled The Request
Timer:
http://code.google.com/appengine/docs/python/requestsandcgi.html
-Brett
On Nov 13, 9:29 am, Gayle Laakmann [EMAIL PROTECTED] wrote:
Despite what the post at the link below says, memcache is actually
timing out and throwing a DeadlineExceededError. Furthermore, it
hardly seems accurate to say that our apps should continue serving
normally. The quota limits are so ridiculously low that we CAN'T
serve pages without caching.
http://groups.google.com/group/google-appengine-downtime-notify/brows...
We will be taking memcache offline tomorrow morning from 9-10am PST
(GMT-8) for routine maintenance. Calls to the memcache API will *not*
throw exceptions but will instead return false for set() calls and
None for get() calls (just like any other cache miss.)
Your app should continue serving normally during this period, and
we'll keep you updated on our progress.
Additonally, I'd like to make two suggestions:
1. If you're going to take down memcache, wouldn't it make sense to
remove the quota limits? It hardly seems fair to penalized for
exceeding quota when caching is disabled.
2. If you have to take down our apps for an hour, can you pick
something like 2am - 3am? I know you don't want to go to work at 2am,
but it's not really ok to take down our apps for an hour during prime-
time.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: Memcache API Timing out
Memcache calls should now be failing fast. That should significantly
reduce the number of request deadlines people are seeing. Otherwise,
we're working to bring the memcache API back online for all
applications as soon as possible. Thanks again for your patience,
everyone!
On Nov 13, 10:12 am, Brett (Google) [EMAIL PROTECTED]
wrote:
Hi Gayle,
Yes, the *request* times out during the API call. It's not the API
call itself that's timing out. But there is no effective difference to
your application if you're caught in this situation (as you pointed
out). I'd assume the first few calls you make to memcache during a
request actually return False and None. But later on in your request
handler, so much time has already passed that your API call gets
interrupted by the DeadlineExceededException for the entire request.
I think for now your best bet is to catch the runtime
DeadlineExceededException and immediately return with some kind of
error code. We're looking into the cause of the delayed memcache calls
on failure. They should be immediately failing instead of waiting up
to a second to return. That's the root cause of the issue here.
I'm sorry for the trouble this is causing for your application. We're
working right now to address this issue. Someone from the team will
follow up when everything has been resolved.
-Brett
On Nov 13, 9:57 am, Gayle Laakmann [EMAIL PROTECTED] wrote:
It was timing out within memcache.get and memcache.set -
consistently. If I understand you correctly - you're saying that
memcache isn't technically timing out, on the grounds that it would
complete if given enough time. But, the cache misses take SO much
time that it consistently timeouts during the operations of
memcache.get and memcache.set.
Ok, sure, that may be the case. I'm afraid I don't really see the
difference between that and timing out. Either way, memcache.get and
memcache.set aren't exactly just return false and letting your
application proceed. The fact is they are causing timeouts.
obj = memcache.get(cache_key)
File /base/data/home/apps/careercup/13.329261908074364999/memcache/
__init__.py, line 362, in get
self._make_sync_call('memcache', 'Get', request, response)
File /base/python_lib/versions/1/google/appengine/api/
apiproxy_stub_map.py, line 46, in MakeSyncCall
stub.MakeSyncCall(service, call, request, response)
File /base/python_lib/versions/1/google/appengine/runtime/
apiproxy.py, line 245, in MakeSyncCall
rpc.Wait()
File /base/python_lib/versions/1/google/appengine/runtime/
apiproxy.py, line 161, in Wait
rpc_completed = _apphosting_runtime___python__apiproxy.Wait(self)
On Nov 13, 9:45 am, Brett [EMAIL PROTECTED] wrote:
Hi Gayle,
I'm sorry to hear you're seeing issues. It's important to note that
there are two types of DeadlineExceededErrors. One is defined in
apiproxy_errors.py. The other is in runtime/__init__.py:
http://code.google.com/p/googleappengine/source/browse/trunk/google/a...
The names are the same, which makes this confusing. The latter error
is raised when the request has been processing for too long (~10
seconds). Perhaps what's happening here is that the cache misses are
causing a lot of extra work to be done, causing your application to
hit this other deadline.
The traceback sent by Pratham confirms this behavior. More detail on
how to deal with this is here, in the section entitled The Request
Timer:
http://code.google.com/appengine/docs/python/requestsandcgi.html
-Brett
On Nov 13, 9:29 am, Gayle Laakmann [EMAIL PROTECTED] wrote:
Despite what the post at the link below says, memcache is actually
timing out and throwing a DeadlineExceededError. Furthermore, it
hardly seems accurate to say that our apps should continue serving
normally. The quota limits are so ridiculously low that we CAN'T
serve pages without caching.
http://groups.google.com/group/google-appengine-downtime-notify/brows...
We will be taking memcache offline tomorrow morning from 9-10am PST
(GMT-8) for routine maintenance. Calls to the memcache API will *not*
throw exceptions but will instead return false for set() calls and
None for get() calls (just like any other cache miss.)
Your app should continue serving normally during this period, and
we'll keep you updated on our progress.
Additonally, I'd like to make two suggestions:
1. If you're going to take down memcache, wouldn't it make sense to
remove the quota limits? It hardly seems fair to penalized for
exceeding quota when caching is disabled.
2. If you have to take down our apps for an hour, can you pick
something like 2am - 3am? I know you don't want to go to work at 2am,
but it's not really ok to take down our apps for an hour during prime-
time.
--~--~-~--~~~---~--~~
You received this message
[google-appengine] Re: Memcache API Timing out
Hi Gee,
Can you please reply to me with your app ID and I can look in to it.
-Marzia
On Thu, Nov 13, 2008 at 10:50 AM, gee [EMAIL PROTECTED] wrote:
Hi Brett,
During this time, it would have been very extremely helpful if CPU/
response time quotas were relaxed.
All our cache misses are adding up, and we're now over quota.
We've had significant downtime over the last two days due to the
scheduled downtimes going awry (our app was one of those taken down
unexpectedly Tuesday night), and it's definitely hurting our users'
experience with our growing service. Any help would be much
appreciated.
Thanks
Gee
On Nov 13, 10:30 am, Brett (Google) [EMAIL PROTECTED]
wrote:
Memcache calls should now be failing fast. That should significantly
reduce the number of request deadlines people are seeing. Otherwise,
we're working to bring the memcache API back online for all
applications as soon as possible. Thanks again for your patience,
everyone!
On Nov 13, 10:12 am, Brett (Google) [EMAIL PROTECTED]
wrote:
Hi Gayle,
Yes, the *request* times out during the API call. It's not the API
call itself that's timing out. But there is no effective difference to
your application if you're caught in this situation (as you pointed
out). I'd assume the first few calls you make to memcache during a
request actually return False and None. But later on in your request
handler, so much time has already passed that your API call gets
interrupted by the DeadlineExceededException for the entire request.
I think for now your best bet is to catch the runtime
DeadlineExceededException and immediately return with some kind of
error code. We're looking into the cause of the delayed memcache calls
on failure. They should be immediately failing instead of waiting up
to a second to return. That's the root cause of the issue here.
I'm sorry for the trouble this is causing for your application. We're
working right now to address this issue. Someone from the team will
follow up when everything has been resolved.
-Brett
On Nov 13, 9:57 am, Gayle Laakmann [EMAIL PROTECTED] wrote:
It was timing out within memcache.get and memcache.set -
consistently. If I understand you correctly - you're saying that
memcache isn't technically timing out, on the grounds that it would
complete if given enough time. But, the cache misses take SO much
time that it consistently timeouts during the operations of
memcache.get and memcache.set.
Ok, sure, that may be the case. I'm afraid I don't really see the
difference between that and timing out. Either way, memcache.get and
memcache.set aren't exactly just return false and letting your
application proceed. The fact is they are causing timeouts.
obj = memcache.get(cache_key)
File
/base/data/home/apps/careercup/13.329261908074364999/memcache/
__init__.py, line 362, in get
self._make_sync_call('memcache', 'Get', request, response)
File /base/python_lib/versions/1/google/appengine/api/
apiproxy_stub_map.py, line 46, in MakeSyncCall
stub.MakeSyncCall(service, call, request, response)
File /base/python_lib/versions/1/google/appengine/runtime/
apiproxy.py, line 245, in MakeSyncCall
rpc.Wait()
File /base/python_lib/versions/1/google/appengine/runtime/
apiproxy.py, line 161, in Wait
rpc_completed = _apphosting_runtime___python__apiproxy.Wait(self)
On Nov 13, 9:45 am, Brett [EMAIL PROTECTED] wrote:
Hi Gayle,
I'm sorry to hear you're seeing issues. It's important to note that
there are two types of DeadlineExceededErrors. One is defined in
apiproxy_errors.py. The other is in runtime/__init__.py:
http://code.google.com/p/googleappengine/source/browse/trunk/google/a...
The names are the same, which makes this confusing. The latter
error
is raised when the request has been processing for too long (~10
seconds). Perhaps what's happening here is that the cache misses
are
causing a lot of extra work to be done, causing your application to
hit this other deadline.
The traceback sent by Pratham confirms this behavior. More detail
on
how to deal with this is here, in the section entitled The Request
Timer:
http://code.google.com/appengine/docs/python/requestsandcgi.html
-Brett
On Nov 13, 9:29 am, Gayle Laakmann [EMAIL PROTECTED] wrote:
Despite what the post at the link below says, memcache is
actually
timing out and throwing a DeadlineExceededError. Furthermore, it
hardly seems accurate to say that our apps should continue
serving
normally. The quota limits are so ridiculously low that we CAN'T
serve pages without caching.
http://groups.google.com/group/google-appengine-downtime-notify/brows...
We will be taking memcache offline tomorrow morning from
[google-appengine] Re: Memcache API Timing out
Hi Gee,
Definitely understood that there's been quite a bit of trouble the
last two days, especially for you and a few other apps. We're sorry
you guys have been hurting from this. I can imagine it's especially
frustrating for a growing service.
In the specific case of this memcache maintenance, the fail-fast issue
lead to high CPU and response time problems for applications that rely
heavily on memcache. We're working to resolve this fail-fast issue so
it will not happen again. We also came up with an interim solution
that will prevent this problem from happening during future memcache
maintenances.
Otherwise, I understand the pain you're feeling here, and relaxing
response times and CPU times would be one way to alleviate that. Our
team is working hard to improve this in general. Feel free to email
Pete, Paul, or me if you have any more questions.
The good news is that memcache is now back up and serving. Please let
us know if you encounter any more issues!
-Brett
On Nov 13, 10:50 am, gee [EMAIL PROTECTED] wrote:
Hi Brett,
During this time, it would have been very extremely helpful if CPU/
response time quotas were relaxed.
All our cache misses are adding up, and we're now over quota.
We've had significant downtime over the last two days due to the
scheduled downtimes going awry (our app was one of those taken down
unexpectedly Tuesday night), and it's definitely hurting our users'
experience with our growing service. Any help would be much
appreciated.
Thanks
Gee
On Nov 13, 10:30 am, Brett (Google) [EMAIL PROTECTED]
wrote:
Memcache calls should now be failing fast. That should significantly
reduce the number of request deadlines people are seeing. Otherwise,
we're working to bring the memcache API back online for all
applications as soon as possible. Thanks again for your patience,
everyone!
On Nov 13, 10:12 am, Brett (Google) [EMAIL PROTECTED]
wrote:
Hi Gayle,
Yes, the *request* times out during the API call. It's not the API
call itself that's timing out. But there is no effective difference to
your application if you're caught in this situation (as you pointed
out). I'd assume the first few calls you make to memcache during a
request actually return False and None. But later on in your request
handler, so much time has already passed that your API call gets
interrupted by the DeadlineExceededException for the entire request.
I think for now your best bet is to catch the runtime
DeadlineExceededException and immediately return with some kind of
error code. We're looking into the cause of the delayed memcache calls
on failure. They should be immediately failing instead of waiting up
to a second to return. That's the root cause of the issue here.
I'm sorry for the trouble this is causing for your application. We're
working right now to address this issue. Someone from the team will
follow up when everything has been resolved.
-Brett
On Nov 13, 9:57 am, Gayle Laakmann [EMAIL PROTECTED] wrote:
It was timing out within memcache.get and memcache.set -
consistently. If I understand you correctly - you're saying that
memcache isn't technically timing out, on the grounds that it would
complete if given enough time. But, the cache misses take SO much
time that it consistently timeouts during the operations of
memcache.get and memcache.set.
Ok, sure, that may be the case. I'm afraid I don't really see the
difference between that and timing out. Either way, memcache.get and
memcache.set aren't exactly just return false and letting your
application proceed. The fact is they are causing timeouts.
obj = memcache.get(cache_key)
File /base/data/home/apps/careercup/13.329261908074364999/memcache/
__init__.py, line 362, in get
self._make_sync_call('memcache', 'Get', request, response)
File /base/python_lib/versions/1/google/appengine/api/
apiproxy_stub_map.py, line 46, in MakeSyncCall
stub.MakeSyncCall(service, call, request, response)
File /base/python_lib/versions/1/google/appengine/runtime/
apiproxy.py, line 245, in MakeSyncCall
rpc.Wait()
File /base/python_lib/versions/1/google/appengine/runtime/
apiproxy.py, line 161, in Wait
rpc_completed = _apphosting_runtime___python__apiproxy.Wait(self)
On Nov 13, 9:45 am, Brett [EMAIL PROTECTED] wrote:
Hi Gayle,
I'm sorry to hear you're seeing issues. It's important to note that
there are two types of DeadlineExceededErrors. One is defined in
apiproxy_errors.py. The other is in runtime/__init__.py:
http://code.google.com/p/googleappengine/source/browse/trunk/google/a...
The names are the same, which makes this confusing. The latter error
is raised when the request has been processing for too long (~10
seconds). Perhaps what's happening here is that the cache misses are
causing a lot of extra work to
[google-appengine] Re: Memcache API Timing out
I'm now hitting TONS of over quota errors, and I wasn't before.
This is really hurting my revenue today.
http://www.careercup.com
On Nov 13, 11:09 am, Brett (Google) [EMAIL PROTECTED]
wrote:
Hi Gee,
Definitely understood that there's been quite a bit of trouble the
last two days, especially for you and a few other apps. We're sorry
you guys have been hurting from this. I can imagine it's especially
frustrating for a growing service.
In the specific case of this memcache maintenance, the fail-fast issue
lead to high CPU and response time problems for applications that rely
heavily on memcache. We're working to resolve this fail-fast issue so
it will not happen again. We also came up with an interim solution
that will prevent this problem from happening during future memcache
maintenances.
Otherwise, I understand the pain you're feeling here, and relaxing
response times and CPU times would be one way to alleviate that. Our
team is working hard to improve this in general. Feel free to email
Pete, Paul, or me if you have any more questions.
The good news is that memcache is now back up and serving. Please let
us know if you encounter any more issues!
-Brett
On Nov 13, 10:50 am, gee [EMAIL PROTECTED] wrote:
Hi Brett,
During this time, it would have been very extremely helpful if CPU/
response time quotas were relaxed.
All our cache misses are adding up, and we're now over quota.
We've had significant downtime over the last two days due to the
scheduled downtimes going awry (our app was one of those taken down
unexpectedly Tuesday night), and it's definitely hurting our users'
experience with our growing service. Any help would be much
appreciated.
Thanks
Gee
On Nov 13, 10:30 am, Brett (Google) [EMAIL PROTECTED]
wrote:
Memcache calls should now be failing fast. That should significantly
reduce the number of request deadlines people are seeing. Otherwise,
we're working to bring the memcache API back online for all
applications as soon as possible. Thanks again for your patience,
everyone!
On Nov 13, 10:12 am, Brett (Google) [EMAIL PROTECTED]
wrote:
Hi Gayle,
Yes, the *request* times out during the API call. It's not the API
call itself that's timing out. But there is no effective difference to
your application if you're caught in this situation (as you pointed
out). I'd assume the first few calls you make to memcache during a
request actually return False and None. But later on in your request
handler, so much time has already passed that your API call gets
interrupted by the DeadlineExceededException for the entire request.
I think for now your best bet is to catch the runtime
DeadlineExceededException and immediately return with some kind of
error code. We're looking into the cause of the delayed memcache calls
on failure. They should be immediately failing instead of waiting up
to a second to return. That's the root cause of the issue here.
I'm sorry for the trouble this is causing for your application. We're
working right now to address this issue. Someone from the team will
follow up when everything has been resolved.
-Brett
On Nov 13, 9:57 am, Gayle Laakmann [EMAIL PROTECTED] wrote:
It was timing out within memcache.get and memcache.set -
consistently. If I understand you correctly - you're saying that
memcache isn't technically timing out, on the grounds that it would
complete if given enough time. But, the cache misses take SO much
time that it consistently timeouts during the operations of
memcache.get and memcache.set.
Ok, sure, that may be the case. I'm afraid I don't really see the
difference between that and timing out. Either way, memcache.get and
memcache.set aren't exactly just return false and letting your
application proceed. The fact is they are causing timeouts.
obj = memcache.get(cache_key)
File /base/data/home/apps/careercup/13.329261908074364999/memcache/
__init__.py, line 362, in get
self._make_sync_call('memcache', 'Get', request, response)
File /base/python_lib/versions/1/google/appengine/api/
apiproxy_stub_map.py, line 46, in MakeSyncCall
stub.MakeSyncCall(service, call, request, response)
File /base/python_lib/versions/1/google/appengine/runtime/
apiproxy.py, line 245, in MakeSyncCall
rpc.Wait()
File /base/python_lib/versions/1/google/appengine/runtime/
apiproxy.py, line 161, in Wait
rpc_completed = _apphosting_runtime___python__apiproxy.Wait(self)
On Nov 13, 9:45 am, Brett [EMAIL PROTECTED] wrote:
Hi Gayle,
I'm sorry to hear you're seeing issues. It's important to note that
there are two types of DeadlineExceededErrors. One is defined in
apiproxy_errors.py. The other is in runtime/__init__.py:
