Hi,
I've been seeing the following phenomenon: Datastore or Memcache
operations (e.g. PersistentManager#getObjectById, Query#execute, or
Cache#get/put) freezes repeatedly for a few certain users (source IP
address), while all the other requests are not affected. Sometimes it
lasts just for a 10 - 15 minutes, and sometimes it lasts a few hours.
As a result, those phenomenons becomes a major part of the CPU cost of
our app.
As the failures are repeating, I don't think it is the 0.1%
spontaneous failures mentioned in the past discussions. I suspect that
a certain App Server node or cluster would be having a trouble on its
tablet servers or memcache service, and it is taking so long time to
fail over. What should I do for this?
Thanks,
Kaz
Example logs:
-----
com.google.apphosting.runtime.HardDeadlineExceededError: This request
(f2e1077c0b151110) started at 2009/09/24 13:14:29.281 UTC and was
still executing at 2009/09/24 13:14:59.071 UTC.
at com.google.appengine.runtime.Request.process-c9366ab62e22ae39
(Request.java)
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos
(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos
(Unknown Source)
at java.util.concurrent.CountDownLatch.await(Unknown Source)
at com.google.net.rpc.util.RpcWaiter.waitForRpcsToFinish
(RpcWaiter.java:96)
at com.google.apphosting.api.ApiProxy.makeSyncCall(ApiProxy.java:79)
at com.google.appengine.api.datastore.DatastoreApiHelper.makeSyncCall
(DatastoreApiHelper.java:48)
at com.google.appengine.api.datastore.DatastoreServiceImpl$1.run
(DatastoreServiceImpl.java:110)
at
com.google.appengine.api.datastore.TransactionRunner.runInTransaction
(TransactionRunner.java:30)
at com.google.appengine.api.datastore.DatastoreServiceImpl.get
(DatastoreServiceImpl.java:96)
at com.google.appengine.api.datastore.DatastoreServiceImpl.get
(DatastoreServiceImpl.java:69)
at com.google.appengine.api.datastore.DatastoreServiceImpl.get
(DatastoreServiceImpl.java:62)
at
org.datanucleus.store.appengine.RuntimeExceptionWrappingDatastoreService.get
(RuntimeExceptionWrappingDatastoreService.java:53)
at org.datanucleus.store.appengine.DatastorePersistenceHandler.get
(DatastorePersistenceHandler.java:79)
at org.datanucleus.store.appengine.DatastorePersistenceHandler.get
(DatastorePersistenceHandler.java:91)
at
org.datanucleus.store.appengine.DatastorePersistenceHandler.locateObject
(DatastorePersistenceHandler.java:559)
at org.datanucleus.state.JDOStateManagerImpl.validate
(JDOStateManagerImpl.java:4273)
at org.datanucleus.ObjectManagerImpl.findObject
(ObjectManagerImpl.java:2444)
at org.datanucleus.jdo.JDOPersistenceManager.getObjectById
(JDOPersistenceManager.java:1671)
at org.datanucleus.jdo.JDOPersistenceManager.getObjectById
(JDOPersistenceManager.java:1767)
at
org.datanucleus.store.appengine.jdo.DatastoreJDOPersistenceManager.getObjectById
(DatastoreJDOPersistenceManager.java:71)
-----
com.google.apphosting.runtime.HardDeadlineExceededError: This request
(517b68e8231edfc1) started at 2009/09/24 13:12:52.711 UTC and was
still executing at 2009/09/24 13:13:28.846 UTC.
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos
(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos
(Unknown Source)
at java.util.concurrent.CountDownLatch.await(Unknown Source)
at com.google.net.rpc.util.RpcWaiter.waitForRpcsToFinish
(RpcWaiter.java:96)
at com.google.apphosting.runtime.ApiProxyImpl.doSyncCall
(ApiProxyImpl.java:99)
at com.google.apphosting.runtime.ApiProxyImpl.access$000
(ApiProxyImpl.java:36)
at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:
68)
at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:
64)
at java.security.AccessController.doPrivileged(Native Method)
at com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall
(ApiProxyImpl.java:64)
at com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall
(ApiProxyImpl.java:36)
at com.google.apphosting.api.ApiProxy.makeSyncCall(ApiProxy.java:79)
at com.google.appengine.api.memcache.MemcacheServiceImpl.makeSyncCall
(MemcacheServiceImpl.java:162)
at com.google.appengine.api.memcache.MemcacheServiceImpl.get
(MemcacheServiceImpl.java:233)
at com.google.appengine.api.memcache.stdimpl.GCache.get(GCache.java:
157)
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en
-~----------~----~----~----~------~----~------~--~---
