Dan Smith created GEODE-2808:
--------------------------------
Summary: Deadlock in tomcat session module during session
expiration
Key: GEODE-2808
URL: https://issues.apache.org/jira/browse/GEODE-2808
Project: Geode
Issue Type: Bug
Components: http session
Reporter: Dan Smith
We observed a deadlock in the tomcat session state replication module due to
the way we do expiration. Here are the threads that are involved:
{noformat}
Found one Java-level deadlock:
=============================
"http-nio-21064-exec-2":
waiting to lock monitor 0x00007f9a04bde298 (object 0x00000007947da1f0, a
org.apache.geode.internal.cache.VersionedStatsLRURegionEntryHeapObjectKey),
which is held by "Timer-1"
"Timer-1":
waiting to lock monitor 0x00007f99a4004e38 (object 0x00000007947da240, a
org.apache.geode.modules.session.catalina.DeltaSession8),
which is held by "http-nio-21064-exec-2"
Java stack information for the threads listed above:
===================================================
"http-nio-21064-exec-2":
at
org.apache.geode.internal.cache.AbstractRegionMap.destroy(AbstractRegionMap.java:1228)
- waiting to lock <0x00000007947da1f0> (a
org.apache.geode.internal.cache.VersionedStatsLRURegionEntryHeapObjectKey)
at
org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6785)
at
org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6762)
at
org.apache.geode.internal.cache.LocalRegionDataView.destroyExistingEntry(LocalRegionDataView.java:55)
at
org.apache.geode.internal.cache.LocalRegion.basicDestroy(LocalRegion.java:6724)
at
org.apache.geode.internal.cache.LocalRegion.validatedDestroy(LocalRegion.java:1110)
at
org.apache.geode.internal.cache.LocalRegion.destroy(LocalRegion.java:1095)
at
org.apache.geode.internal.cache.AbstractRegion.destroy(AbstractRegion.java:271)
at
org.apache.geode.modules.session.catalina.AbstractSessionCache.destroySession(AbstractSessionCache.java:72)
at
org.apache.geode.modules.session.catalina.DeltaSessionManager.remove(DeltaSessionManager.java:405)
at
org.apache.catalina.session.StandardSession.expire(StandardSession.java:850)
- locked <0x00000007947da240> (a
org.apache.geode.modules.session.catalina.DeltaSession8)
at
org.apache.catalina.session.StandardSession.isValid(StandardSession.java:682)
at org.apache.catalina.connector.Request.doGetSession(Request.java:2917)
at org.apache.catalina.connector.Request.getSession(Request.java:2367)
at
org.apache.geode.modules.session.catalina.CommitSessionValve.invoke(CommitSessionValve.java:50)
at
org.apache.geode.modules.session.catalina.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:46)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:474)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at
org.apache.geode.modules.session.catalina.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:46)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349)
at
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:783)
at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:789)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1437)
at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0x00000007948d6870> (a
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
"Timer-1":
at
org.apache.catalina.session.StandardSession.expire(StandardSession.java:791)
- waiting to lock <0x00000007947da240> (a
org.apache.geode.modules.session.catalina.DeltaSession8)
at
org.apache.catalina.session.StandardSession.expire(StandardSession.java:766)
at
org.apache.geode.modules.session.catalina.DeltaSession8.processExpired(DeltaSession8.java:320)
at
org.apache.geode.modules.session.catalina.callback.SessionExpirationCacheListener.afterDestroy(SessionExpirationCacheListener.java:56)
at
org.apache.geode.internal.cache.EnumListenerEvent$AFTER_DESTROY.dispatchEvent(EnumListenerEvent.java:151)
at
org.apache.geode.internal.cache.LocalRegion.dispatchEvent(LocalRegion.java:8751)
at
org.apache.geode.internal.cache.LocalRegion.dispatchListenerEvent(LocalRegion.java:7245)
at
org.apache.geode.internal.cache.LocalRegion.invokeDestroyCallbacks(LocalRegion.java:7052)
at
org.apache.geode.internal.cache.EntryEventImpl.invokeCallbacks(EntryEventImpl.java:2293)
at
org.apache.geode.internal.cache.AbstractRegionEntry.dispatchListenerEvents(AbstractRegionEntry.java:140)
at
org.apache.geode.internal.cache.LocalRegion.basicDestroyPart2(LocalRegion.java:6992)
at
org.apache.geode.internal.cache.AbstractRegionMap.destroy(AbstractRegionMap.java:1419)
- locked <0x00000007947da1f0> (a
org.apache.geode.internal.cache.VersionedStatsLRURegionEntryHeapObjectKey)
at
org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6785)
at
org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6762)
at
org.apache.geode.internal.cache.LocalRegionDataView.destroyExistingEntry(LocalRegionDataView.java:55)
at
org.apache.geode.internal.cache.LocalRegion.basicDestroy(LocalRegion.java:6724)
at
org.apache.geode.internal.cache.LocalRegion.expireDestroy(LocalRegion.java:6864)
at
org.apache.geode.internal.cache.EntryExpiryTask.destroy(EntryExpiryTask.java:119)
at
org.apache.geode.internal.cache.ExpiryTask.expire(ExpiryTask.java:278)
at
org.apache.geode.internal.cache.ExpiryTask.expire(ExpiryTask.java:232)
at
org.apache.geode.internal.cache.EntryExpiryTask.basicPerformTimeout(EntryExpiryTask.java:305)
at
org.apache.geode.internal.cache.LocalRegion.performExpiryTimeout(LocalRegion.java:801)
at
org.apache.geode.internal.cache.EntryExpiryTask.performTimeout(EntryExpiryTask.java:219)
at
org.apache.geode.internal.cache.ExpiryTask.runInThreadPool(ExpiryTask.java:370)
at org.apache.geode.internal.cache.ExpiryTask.run2(ExpiryTask.java:318)
at
org.apache.geode.internal.SystemTimer$SystemTimerTask.run(SystemTimer.java:444)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Found 1 deadlock.
{noformat}
The basic issue is that we configure expiration on the geode region to expire
sessions, and the tomcat container will also expire sessions when the get to
old. But those two threads get locks in a different order:
Tomcat expiration (http-nio-21064-exec-2)
# sync session in the tomcat code (StandardSession.expire)
# sync region entry in the geode code (AbstractRegionMap.destroy)
region expiration
# sync region entry in the geode code (AbstractRegionMap.destroy
# sync session in geode code (StandardSession.expire)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
