Here is the rest of the stack trace.
I had profiled the application using OptimizeIt earlier and it was spending
80-90% of its time in trying to acquire a connection from the datasource.
I had even bumped up min/max connection to be more than the concurrent requests
and it still did not help. The acceptCount and maxThreads in Tomcat is also
high enough.
"http-0.0.0.0-80-51" daemon prio=5 tid=0x3002eca0 nid=0x948 in Object.wait()
[0x31c0f000..0x31c0f9e8]
at java.lang.Object.wait(Native Method)
- waiting on <0x07e1a388> (a
org.apache.tomcat.util.net.MasterSlaveWorkerThread)
at java.lang.Object.wait(Object.java:474)
at
org.apache.tomcat.util.net.MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
- locked <0x07e1a388> (a
org.apache.tomcat.util.net.MasterSlaveWorkerThread)
at
org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
at java.lang.Thread.run(Thread.java:595)
"http-0.0.0.0-80-50" daemon prio=5 tid=0x2f643008 nid=0x944 runnable
[0x31bcf000..0x31bcfa68]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at
org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:737)
at
org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:398)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:818)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at
org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
"http-0.0.0.0-80-8" daemon prio=5 tid=0x2f9de558 nid=0x89c runnable
[0x3114d000..0x3114fb68]
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x05af86d8> (a java.io.BufferedOutputStream)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:2692)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:2621)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1552)
at
com.mysql.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:1124)
- locked <0x05af33c0> (a java.lang.Object)
at
com.mysql.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:676)
at
com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1030)
- locked <0x05af33c0> (a java.lang.Object)
- locked <0x088c61b0> (a com.mysql.jdbc.ServerPreparedStatement)
at
org.jboss.resource.adapter.jdbc.CachedPreparedStatement.executeQuery(CachedPreparedStatement.java:76)
at
org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:211)
at
org.jboss.ejb.plugins.cmp.jdbc.JDBCAbstractQueryCommand.execute(JDBCAbstractQueryCommand.java:228)
at
org.jboss.ejb.plugins.cmp.jdbc.JDBCAbstractQueryCommand.execute(JDBCAbstractQueryCommand.java:128)
at
org.jboss.ejb.plugins.cmp.jdbc.JDBCFindEntityCommand.execute(JDBCFindEntityCommand.java:47)
at
org.jboss.ejb.plugins.cmp.jdbc.JDBCStoreManager.findEntity(JDBCStoreManager.java:589)
at
org.jboss.ejb.plugins.CMPPersistenceManager.findEntity(CMPPersistenceManager.java:300)
at
org.jboss.resource.connectionmanager.CachedConnectionInterceptor.findEntity(CachedConnectionInterceptor.java:236)
at
org.jboss.ejb.EntityContainer.findSingleObject(EntityContainer.java:1086)
at org.jboss.ejb.EntityContainer.findLocal(EntityContainer.java:663)
at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.invocation.Invocation.performCall(Invocation.java:345)
at
org.jboss.ejb.EntityContainer$ContainerInterceptor.invokeHome(EntityContainer.java:1113)
at
org.jboss.ejb.plugins.AbstractInterceptor.invokeHome(AbstractInterceptor.java:90)
at
org.jboss.ejb.plugins.EntitySynchronizationInterceptor.invokeHome(EntitySynchronizationInterceptor.java:189)
at
org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invokeHome(CachedConnectionInterceptor.java:180)
at
org.jboss.ejb.plugins.AbstractInterceptor.invokeHome(AbstractInterceptor.java:90)
at
org.jboss.ejb.plugins.EntityInstanceInterceptor.invokeHome(EntityInstanceInterceptor.java:119)
at
org.jboss.ejb.plugins.EntityLockInterceptor.invokeHome(EntityLockInterceptor.java:61)
at
org.jboss.ejb.plugins.EntityCreationInterceptor.invokeHome(EntityCreationInterceptor.java:28)
at
org.jboss.ejb.plugins.CallValidationInterceptor.invokeHome(CallValidationInterceptor.java:41)
at
org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:110)
at
org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:420)
at
org.jboss.ejb.plugins.TxInterceptorCMT.invokeHome(TxInterceptorCMT.java:146)
at
org.jboss.ejb.plugins.SecurityInterceptor.invokeHome(SecurityInterceptor.java:130)
at
org.jboss.ejb.plugins.LogInterceptor.invokeHome(LogInterceptor.java:121)
at
org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invokeHome(ProxyFactoryFinderInterceptor.java:93)
at
org.jboss.ejb.EntityContainer.internalInvokeHome(EntityContainer.java:508)
at org.jboss.ejb.Container.invoke(Container.java:894)
at
org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invokeHome(BaseLocalProxyFactory.java:344)
at
org.jboss.ejb.plugins.local.LocalHomeProxy.invoke(LocalHomeProxy.java:118)
at $Proxy145.findByUserIdAndKey(Unknown Source)
"http-0.0.0.0-80" daemon prio=5 tid=0x2f97ece8 nid=0x129c runnable
[0x30f4f000..0x30f4fbe8]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
- locked <0x05669fc8> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:450)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at
org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:407)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.run(PoolTcpEndpoint.java:647)
at java.lang.Thread.run(Thread.java:595)
"JBossLifeThread" prio=5 tid=0x26ffad80 nid=0x1298 in Object.wait()
[0x30f0f000..0x30f0fc68]
at java.lang.Object.wait(Native Method)
- waiting on <0x06cf1ce8> (a java.lang.Object)
at java.lang.Object.wait(Object.java:474)
at
org.jboss.system.server.ServerImpl$LifeThread.run(ServerImpl.java:900)
- locked <0x06cf1ce8> (a java.lang.Object)
"Timer-3" daemon prio=5 tid=0x2705e838 nid=0x1294 in Object.wait()
[0x30ecf000..0x30ecfce8]
at java.lang.Object.wait(Native Method)
- waiting on <0x05fec178> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x05fec178> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"JCA PoolFiller" prio=5 tid=0x27115ee0 nid=0xfdc in Object.wait()
[0x2fe4f000..0x2fe4f9e8]
at java.lang.Object.wait(Native Method)
- waiting on <0x05dfa9e0> (a java.util.LinkedList)
at java.lang.Object.wait(Object.java:474)
at
org.jboss.resource.connectionmanager.PoolFiller.run(PoolFiller.java:82)
- locked <0x05dfa9e0> (a java.util.LinkedList)
at java.lang.Thread.run(Thread.java:595)
"IdleRemover" daemon prio=5 tid=0x2f6f3bd0 nid=0xf94 in Object.wait()
[0x2fe0f000..0x2fe0fa68]
at java.lang.Object.wait(Native Method)
- waiting on <0x05dfaa58> (a java.util.ArrayList)
at
org.jboss.resource.connectionmanager.IdleRemover$1.run(IdleRemover.java:77)
- locked <0x05dfaa58> (a java.util.ArrayList)
at java.lang.Thread.run(Thread.java:595)
"ContainerBackgroundProcessor[StandardEngine[jboss.web]]" daemon prio=5
tid=0x2f59c828 nid=0xfc8 waiting on condition [0
x2fdcf000..0x2fdcfae8]
at java.lang.Thread.sleep(Native Method)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1546)
at java.lang.Thread.run(Thread.java:595)
"PooledInvokerAcceptor#499-4445" prio=5 tid=0x27436088 nid=0xfd0 waiting for
monitor entry [0x2f58f000..0x2f58fb68]
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:382)
- waiting to lock <0x0537f428> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:450)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at
org.jboss.invocation.pooled.server.PooledInvoker.run(PooledInvoker.java:240)
at java.lang.Thread.run(Thread.java:595)
"RMI TCP Accept-4444" daemon prio=5 tid=0x27270b48 nid=0x1410 runnable
[0x2788f000..0x2788fd68]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
- locked <0x0539ca08> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:450)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
at java.lang.Thread.run(Thread.java:595)
"JBoss System Threads(1)-2" daemon prio=5 tid=0x27281008 nid=0x140c runnable
[0x2784f000..0x2784f9e8]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
- locked <0x0539cbf8> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:450)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.jnp.server.Main$AcceptHandler.run(Main.java:417)
at
org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:134)
at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
at java.lang.Thread.run(Thread.java:595)
"GC Daemon" daemon prio=2 tid=0x27135718 nid=0x1408 in Object.wait()
[0x2780f000..0x2780fa68]
at java.lang.Object.wait(Native Method)
- waiting on <0x052c2ae0> (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:100)
- locked <0x052c2ae0> (a sun.misc.GC$LatencyLock)
"RMI Reaper" prio=5 tid=0x27273808 nid=0x1404 in Object.wait()
[0x277cf000..0x277cfae8]
at java.lang.Object.wait(Native Method)
- waiting on <0x052c1d60> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
- locked <0x052c1d60> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:336)
at java.lang.Thread.run(Thread.java:595)
"Timer-1" daemon prio=5 tid=0x272628f8 nid=0x1400 in Object.wait()
[0x2778f000..0x2778fb68]
at java.lang.Object.wait(Native Method)
- waiting on <0x052c20e8> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:474)
at java.util.TimerThread.mainLoop(Timer.java:483)
- locked <0x052c20e8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"RMI TCP Accept-1098" daemon prio=5 tid=0x27274af8 nid=0x132c runnable
[0x2774f000..0x2774fbe8]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
- locked <0x052c1dc0> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:450)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
at java.lang.Thread.run(Thread.java:595)
"JBoss System Threads(1)-1" daemon prio=5 tid=0x27266e00 nid=0x13e4 runnable
[0x276df000..0x276dfc68]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
- locked <0x052ba838> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:450)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.jboss.web.WebServer.run(WebServer.java:310)
at
org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:134)
at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
at java.lang.Thread.run(Thread.java:595)
"ScannerThread" daemon prio=5 tid=0x27122b00 nid=0x13dc waiting on condition
[0x274df000..0x274dfce8]
at java.lang.Thread.sleep(Native Method)
at
org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.loop(AbstractDeploymentScanner.java:209)
at
org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:182)
"Timer-0" daemon prio=5 tid=0x271c0188 nid=0x13d4 in Object.wait()
[0x2748f000..0x2748fd68]
at java.lang.Object.wait(Native Method)
- waiting on <0x052c3360> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x052c3360> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"DestroyJavaVM" prio=5 tid=0x00036cd0 nid=0x13b8 waiting on condition
[0x00000000..0x0007fae8]
"Low Memory Detector" daemon prio=5 tid=0x00974068 nid=0xc5c runnable
[0x00000000..0x00000000]
"CompilerThread0" daemon prio=10 tid=0x00972c40 nid=0xcac waiting on condition
[0x00000000..0x26d7f8cc]
"Signal Dispatcher" daemon prio=10 tid=0x00971fc8 nid=0x278 waiting on
condition [0x00000000..0x00000000]
"Finalizer" daemon prio=9 tid=0x00969590 nid=0x494 in Object.wait()
[0x26cff000..0x26cffc68]
at java.lang.Object.wait(Native Method)
- waiting on <0x051807d0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
- locked <0x051807d0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x00968100 nid=0x13c0 in Object.wait()
[0x26cbf000..0x26cbfce8]
at java.lang.Object.wait(Native Method)
- waiting on <0x05180170> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:474)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x05180170> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=10 tid=0x00965730 nid=0x13bc runnable
"VM Periodic Task Thread" prio=10 tid=0x00975240 nid=0x13cc waiting on condition
View the original post :
http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3917359#3917359
Reply to the post :
http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=3917359
-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
JBoss-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/jboss-user
