We've just release a production service using JBoss 3.2.1 on Solaris with Sun's 1.4.2 JVM, and we're seeing a deadlock on the AbstractInstanceCache for one of our stateful session beans that's used heavily.
Over a period of time more and more threads become locked trying to insert or get from the cache, until all the Jetty threads (currently set to a maximum of 255 on the AJP13 listener we're using) are locked and the entire site becomes unresponsive. This seems to be happening about once an hour.
I've attached a portion of the thread dump from the server when it has become unresponsive.
It shows one thread running org.jboss.ejb.plugins.LRUEnterpriseContextCachePolicy$OveragerTask.run with a lock passivating from the AbstractInstanceCache and then a lock and wait on a org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock. Another thread is trying to get from the cache, and one is inserting into the cache. In the entire thread dump there are more inserts than gets.
Because we are using mod_jk and the AJP13 listener we are able to use the HTTP listener to view the jmx-console after the site has become unresponsive, but I'm not sure what to look for.
We're going to download the source and start looking at the cache implementation, but we would appreciate any help or advice, as we need to fix this urgently.
Regards,
Andrew
"Thread-51" daemon prio=5 tid=0x00705960 nid=0x4c in Object.wait() [83901000..83901998]
at java.lang.Object.wait(Native Method)
- waiting on <0xc0143718> (a
org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock)
at java.lang.Object.wait(Object.java:429)
at org.jboss.ejb.plugins.lock.BeanLockSupport.sync(BeanLockSupport.java:70)
- locked <0xc0143718> (a org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock)
at
org.jboss.ejb.plugins.AbstractInstanceCache.tryToPassivate(AbstractInstanceCache.java:151)
at
org.jboss.ejb.plugins.AbstractInstanceCache.release(AbstractInstanceCache.java:192)
at
org.jboss.ejb.plugins.LRUEnterpriseContextCachePolicy.ageOut(LRUEnterpriseContextCachePolicy.java:274)
at
org.jboss.ejb.plugins.LRUEnterpriseContextCachePolicy$OveragerTask.kickOut(LRUEnterpriseContextCachePolicy.java:446)
at
org.jboss.ejb.plugins.LRUEnterpriseContextCachePolicy$OveragerTask.run(LRUEnterpriseContextCachePolicy.java:405)
- locked <0xbaba03c8> (a java.lang.Object)
at java.util.TimerThread.mainLoop(Timer.java:432)
at java.util.TimerThread.run(Timer.java:382)
"PoolThread-147" prio=5 tid=0x01bc8248 nid=0xbc6 waiting for monitor entry
[7a400000..7a401998]
at
org.jboss.ejb.plugins.AbstractInstanceCache.get(AbstractInstanceCache.java:103)
- waiting to lock <0xbaba03c8> (a java.lang.Object)
at
org.jboss.ejb.plugins.StatefulSessionInstanceInterceptor.invoke(StatefulSessionInstanceInterceptor.java:212)
at
org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:84)
at
org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:216)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:104)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:191)
at
org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
at
org.jboss.ejb.StatefulSessionContainer.internalInvoke(StatefulSessionContainer.java:410)
at org.jboss.ejb.Container.invoke(Container.java:674)
at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at
org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:549)
at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:101)
at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:83)
at
org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:45)
at
org.jboss.proxy.ejb.StatefulSessionInterceptor.invoke(StatefulSessionInterceptor.java:104)
at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:85)
at $Proxy272.search(Unknown Source)
at
com.ingenta.content.handler.BrowseHandlerUtilities.getTableOfContents(BrowseHandlerUtilities.java:225)
- locked <0xbf2a7098> (a com.ingenta.search.handler.SearchSession)
at
com.ingenta.content.handler.TOCHandler.handleAuthenticated(TOCHandler.java:149)
at
com.ingenta.authentication.handler.AuthenticationHandler.handle(AuthenticationHandler.java:244)
at com.ingenta.servlet.Controller.dispatch(Controller.java:213)
at com.ingenta.servlet.Controller.doGet(Controller.java:125)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:356)
at
org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:294)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:558)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1723)
at
org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:514)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1673)
at org.mortbay.http.HttpServer.service(HttpServer.java:879)
at org.jboss.jetty.Jetty.service(Jetty.java:460)
at org.mortbay.http.HttpConnection.service(HttpConnection.java:783)
at org.mortbay.http.ajp.AJP13Connection.handleNext(AJP13Connection.java:273)
at org.mortbay.http.HttpConnection.handle(HttpConnection.java:800)
at org.mortbay.http.ajp.AJP13Listener.handleConnection(AJP13Listener.java:204)
at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:289)
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:454)
"PoolThread-146" prio=5 tid=0x01bc7768 nid=0xbc5 waiting for monitor entry
[7a4ff000..7a501998]
at
org.jboss.ejb.plugins.AbstractInstanceCache.insert(AbstractInstanceCache.java:140)
- waiting to lock <0xbaba03c8> (a java.lang.Object)
at
org.jboss.ejb.StatefulSessionContainer.createSession(StatefulSessionContainer.java:539)
at
org.jboss.ejb.StatefulSessionContainer.createHome(StatefulSessionContainer.java:553)
at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at
org.jboss.ejb.StatefulSessionContainer$ContainerInterceptor.invokeHome(StatefulSessionContainer.java:836)
at
org.jboss.ejb.plugins.SecurityInterceptor.invokeHome(SecurityInterceptor.java:92)
at
org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invokeHome(CachedConnectionInterceptor.java:215)
at
org.jboss.ejb.plugins.StatefulSessionInstanceInterceptor.invokeHome(StatefulSessionInstanceInterceptor.java:128)
at
org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:88)
at
org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:216)
at org.jboss.ejb.plugins.TxInterceptorCMT.invokeHome(TxInterceptorCMT.java:74)
at org.jboss.ejb.plugins.LogInterceptor.invokeHome(LogInterceptor.java:120)
at
org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invokeHome(ProxyFactoryFinderInterceptor.java:93)
at
org.jboss.ejb.StatefulSessionContainer.internalInvokeHome(StatefulSessionContainer.java:398)
at org.jboss.ejb.Container.invoke(Container.java:694)
at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at
org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:549)
at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:101)
at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:83)
at
org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:45)
at org.jboss.proxy.ejb.HomeInterceptor.invoke(HomeInterceptor.java:173)
at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:85)
at $Proxy252.create(Unknown Source)
at com.ingenta.search.handler.SearchSession.<init>(SearchSession.java:80)
at
com.ingenta.search.handler.SearchSession.getOrCreateSearchSession(SearchSession.java:147)
at
com.ingenta.content.handler.BrowseHandlerUtilities.getTableOfContents(BrowseHandlerUtilities.java:219)
at
com.ingenta.content.handler.TOCHandler.handleAuthenticated(TOCHandler.java:149)
at
com.ingenta.authentication.handler.AuthenticationHandler.handle(AuthenticationHandler.java:244)
at com.ingenta.servlet.Controller.dispatch(Controller.java:213)
at com.ingenta.servlet.Controller.doGet(Controller.java:125)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:356)
at
org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:294)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:558)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1723)
at
org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:514)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1673)
at org.mortbay.http.HttpServer.service(HttpServer.java:879)
at org.jboss.jetty.Jetty.service(Jetty.java:460)
at org.mortbay.http.HttpConnection.service(HttpConnection.java:783)
at org.mortbay.http.ajp.AJP13Connection.handleNext(AJP13Connection.java:273)
at org.mortbay.http.HttpConnection.handle(HttpConnection.java:800)
at org.mortbay.http.ajp.AJP13Listener.handleConnection(AJP13Listener.java:204)
at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:289)
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:454)
