Is the Tomcat instance making connections to MySQL? Have you checked MySQL to see if there are an abnormally large number of connections at the time Tomcat fails? Do the MySQL logs tell you anything that suggests some of these threads/sessions might be deadlocked?
-----Original Message----- From: Richard Mixon [mailto:[EMAIL PROTECTED] Sent: Friday, October 14, 2005 3:12 AM To: 'Tomcat Users List' Subject: RE: Tomcat hangs with lots of DBCP "Object.wait" processes One thing I forgot, that might be relevant. Another Tomcat instance on the server continued to function just fine. It also accesses a different MySQL database, but its served by the same MySQL instance. - Richard -----Original Message----- From: Richard Mixon [mailto:[EMAIL PROTECTED] Sent: Thursday, October 13, 2005 11:27 PM To: users@tomcat.apache.org Subject: Tomcat hangs with lots of DBCP "Object.wait" processes I have a customer running Tomcat 4.0.18. About twice a week their system is locking up - obviously we don't know why or I would not be asking. Symptoms 1) CPU usage per TOP is not very high. 2) When requesting the main web page, the page goes white and never returns. When this happened yesterday I was able to get a thread dump. It was quite different than I've seen on this system before, almost 4000 lines long. What was particularly noticable was that it had 74 threads that were almost identically waiting (my interpretation, not sure its accurate, but it seems to be saying so) in Commons DBCP objects. I've got an annotated (and somewhat shortened) version of the thread dump below. Additional info: Sun Java jdk1.5.0_04 Tomcat is of course fronted by Apache 2.0.48 and mod_jk. CPU does not appear to be high, but the # uname -a Linux acme2.acme.com 2.4.22-1.2115.nptlsmp #1 SMP Wed Oct 29 15:30:09 EST 2003 i686 i686 i386 GNU/Linux I believe its Linux Fedora FC2, dual Xeon processors, 2GB RAM Oh, the data source is defined with pretty much default parameters: <Context path="" docBase="httpdocs" reloadable="true" debug="0"> <Resource name="jdbc/acme.com" auth="Container" type="javax.sql.DataSource"/> <ResourceParams name="jdbc/acme.com"> <parameter> <name>url</name> <value>jdbc:mysql://localhost/acme?autoReconnect=true&zeroDateTimeBehavi or=convertToNull&noDatetimeStringSync=true</value> </parameter> <parameter> <name>driverClassName</name> <value>com.mysql.jdbc.Driver</value> </parameter> <parameter> <name>username</name> <value>acme</value> </parameter> <parameter> <name>password</name> <value>acme</value> </parameter> </ResourceParams> </Context> Any ideas or assistance in tracking this down are appreciated Thanks in advance - Richard ----- Annotated stack trace follows --- Full thread dump Java HotSpot(TM) Server VM (1.5.0_04-b05 mixed mode): "Thread-421" daemon prio=1 tid=0x085f8bc0 nid=0x686f in Object.wait() [0x85826000..0x85827680] at java.lang.Object.wait(Native Method) - waiting on <0x49d33258> (a org.apache.commons.dbcp.AbandonedObjectPool) at java.lang.Object.wait(Object.java:474) at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Unknown Source) - locked <0x49d33258> (a org.apache.commons.dbcp.AbandonedObjectPool) at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool .java:117) - locked <0x49d33258> (a org.apache.commons.dbcp.AbandonedObjectPool) at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.ja va:110) at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:3 12) at com.acme.Connector.CreateConnection(Connector.java:69) at com.acme.OpenMailData.insert(OpenMailData.java:282) at com.acme.trackopenmail.doPost(trackopenmail.java:48) at com.acme.trackopenmail.doGet(trackopenmail.java:19) at javax.servlet.http.HttpServlet.service(HttpServlet.java:740) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application FilterChain.java:247) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh ain.java:193) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja va:260) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok eNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja va:191) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok eNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2415) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180 ) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok eNext(StandardPipeline.java:643) at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve. java:170) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok eNext(StandardPipeline.java:641) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172 ) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok eNext(StandardPipeline.java:641) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java :174) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok eNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:223) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:261) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:360) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:632) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:590) at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:707) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav a:530) at java.lang.Thread.run(Thread.java:595) ... 51 more of these, though the actual application class varies between two or three different classes. "Thread-209" daemon prio=1 tid=0x086e0158 nid=0x7746 in Object.wait() [0x7f701000..0x7f701500] at java.lang.Object.wait(Native Method) - waiting on <0x4ab87400> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:474) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav a:509) - locked <0x4ab87400> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:595) "Thread-208" daemon prio=1 tid=0x08607f00 nid=0x7745 in Object.wait() [0x7f482000..0x7f482580] at java.lang.Object.wait(Native Method) - waiting on <0x4ab873e0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:474) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav a:509) - locked <0x4ab873e0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:595) "Thread-207" daemon prio=1 tid=0x087b2aa0 nid=0x7744 in Object.wait() [0x7d4c2000..0x7d4c2600] at java.lang.Object.wait(Native Method) - waiting on <0x4ab873c0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:474) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav a:509) - locked <0x4ab873c0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:595) "Thread-206" daemon prio=1 tid=0x08677068 nid=0x7743 in Object.wait() [0x7f680000..0x7f680680] at java.lang.Object.wait(Native Method) - waiting on <0x4ab873a0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:474) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav a:509) - locked <0x4ab873a0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:595)29 ... 20 more of the DBCP waiting threads, followed by the rest of the thread dump ... "MonitorRunnable" daemon prio=1 tid=0x7c9ae6f0 nid=0x769e in Object.wait() [0x7e449000..0x7e449680] at java.lang.Object.wait(Native Method) - waiting on <0x49d63f30> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable) at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.jav a:420) - locked <0x49d63f30> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable) at java.lang.Thread.run(Thread.java:595) "Thread-5" daemon prio=1 tid=0x7c9acc08 nid=0x769d in Object.wait() [0x7e3c8000..0x7e3c8700] at java.lang.Object.wait(Native Method) - waiting on <0x49d64888> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:474) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav a:509) - locked <0x49d64888> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:595) "Thread-4" daemon prio=1 tid=0x7c9abff8 nid=0x769c in Object.wait() [0x7e347000..0x7e347780] at java.lang.Object.wait(Native Method) - waiting on <0x49d64868> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:474) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav a:509) - locked <0x49d64868> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:595) "Thread-3" daemon prio=1 tid=0x7c7ff3d8 nid=0x769b runnable [0x7e2c6000..0x7e2c6800] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) - locked <0x49ca0038> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:450) at java.net.ServerSocket.accept(ServerSocket.java:421) at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultSe rverSocketFactory.java:107) at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java :341) at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:497) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav a:530) at java.lang.Thread.run(Thread.java:595) "Thread-2" daemon prio=1 tid=0x7c55fb88 nid=0x769a in Object.wait() [0x7e245000..0x7e245480] at java.lang.Object.wait(Native Method) - waiting on <0x49d64828> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:474) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav a:509) - locked <0x49d64828> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:595) "HostConfig[acmemarketing.com]" daemon prio=1 tid=0x7c55f798 nid=0x7699 waiting on condition [0x7e1c4000..0x7e1c4500] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.startup.HostConfig.threadSleep(HostConfig.java:985) at org.apache.catalina.startup.HostConfig.run(HostConfig.java:1009) at java.lang.Thread.run(Thread.java:595) "StandardManager[]" daemon prio=1 tid=0x7c1b0fe8 nid=0x7698 waiting on condition [0x7e143000..0x7e143580] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java :780) at org.apache.catalina.session.StandardManager.run(StandardManager.java:839) at java.lang.Thread.run(Thread.java:595) "WebappLoader[]" daemon prio=1 tid=0x7c9991a8 nid=0x7697 waiting on condition [0x7dae0000..0x7dae0600] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.loader.WebappLoader.threadSleep(WebappLoader.java:1221) at org.apache.catalina.loader.WebappLoader.run(WebappLoader.java:1341) at java.lang.Thread.run(Thread.java:595) "HostConfig[hospitalitytoolkitemail.com]" daemon prio=1 tid=0x7cae8e98 nid=0x7696 waiting on condition [0x7da09000..0x7da09680] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.startup.HostConfig.threadSleep(HostConfig.java:985) at org.apache.catalina.startup.HostConfig.run(HostConfig.java:1009) at java.lang.Thread.run(Thread.java:595) "StandardManager[]" daemon prio=1 tid=0x7be58710 nid=0x7695 waiting on condition [0x7d7bb000..0x7d7bb700] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java :780) at org.apache.catalina.session.StandardManager.run(StandardManager.java:839) at java.lang.Thread.run(Thread.java:595) "WebappLoader[]" daemon prio=1 tid=0x7cc610e8 nid=0x7694 waiting on condition [0x7d441000..0x7d441780] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.loader.WebappLoader.threadSleep(WebappLoader.java:1221) at org.apache.catalina.loader.WebappLoader.run(WebappLoader.java:1341) at java.lang.Thread.run(Thread.java:595) "HostConfig[localhost]" daemon prio=1 tid=0x7c8abba8 nid=0x7693 waiting on condition [0x7d184000..0x7d184800] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.startup.HostConfig.threadSleep(HostConfig.java:985) at org.apache.catalina.startup.HostConfig.run(HostConfig.java:1009) at java.lang.Thread.run(Thread.java:595) "StandardManager[]" daemon prio=1 tid=0x7cae9890 nid=0x7692 waiting on condition [0x7d103000..0x7d103480] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java :780) at org.apache.catalina.session.StandardManager.run(StandardManager.java:839) at java.lang.Thread.run(Thread.java:595) "StandardManager[/webdav]" daemon prio=1 tid=0x7bc62f30 nid=0x7691 waiting on condition [0x7d082000..0x7d082500] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java :780) at org.apache.catalina.session.StandardManager.run(StandardManager.java:839) at java.lang.Thread.run(Thread.java:595) "StandardManager[/examples]" daemon prio=1 tid=0x088c70b8 nid=0x7690 waiting on condition [0x7d001000..0x7d001580] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java :780) at org.apache.catalina.session.StandardManager.run(StandardManager.java:839) at java.lang.Thread.run(Thread.java:595) "StandardManager[/tomcat-docs]" daemon prio=1 tid=0x0820e688 nid=0x768f waiting on condition [0x7cf80000..0x7cf80600] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java :780) at org.apache.catalina.session.StandardManager.run(StandardManager.java:839) at java.lang.Thread.run(Thread.java:595) "StandardManager[/pinEdit]" daemon prio=1 tid=0x0850be60 nid=0x768e waiting on condition [0x7c4ff000..0x7c4ff680] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java :780) at org.apache.catalina.session.StandardManager.run(StandardManager.java:839) at java.lang.Thread.run(Thread.java:595) "StandardManager[/manager]" daemon prio=1 tid=0x08a42cf0 nid=0x768d waiting on condition [0x7bf80000..0x7bf80700] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java :780) at org.apache.catalina.session.StandardManager.run(StandardManager.java:839) at java.lang.Thread.run(Thread.java:595) "StandardManager[/admin]" daemon prio=1 tid=0x0844dba8 nid=0x768c waiting on condition [0x7bd80000..0x7bd80780] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java :780) at org.apache.catalina.session.StandardManager.run(StandardManager.java:839) at java.lang.Thread.run(Thread.java:595) "Low Memory Detector" daemon prio=1 tid=0x081268e0 nid=0x7689 runnable [0x00000000..0x00000000] "CompilerThread1" daemon prio=1 tid=0x08125468 nid=0x7688 waiting on condition [0x00000000..0x7a540718] "CompilerThread0" daemon prio=1 tid=0x08124418 nid=0x7687 waiting on condition [0x00000000..0x7a4bf798] "AdapterThread" daemon prio=1 tid=0x081232e8 nid=0x7686 waiting on condition [0x00000000..0x00000000] "Signal Dispatcher" daemon prio=1 tid=0x08122468 nid=0x7685 waiting on condition [0x00000000..0x00000000] "Finalizer" daemon prio=1 tid=0x08117b98 nid=0x7684 in Object.wait() [0x7a13d000..0x7a13d700] at java.lang.Object.wait(Native Method) - waiting on <0x49cea6d8> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116) - locked <0x49cea6d8> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) "Reference Handler" daemon prio=1 tid=0x08116f20 nid=0x7683 in Object.wait() [0x7a0bc000..0x7a0bc780] at java.lang.Object.wait(Native Method) - waiting on <0x49cc2568> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:474) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) - locked <0x49cc2568> (a java.lang.ref.Reference$Lock) "main" prio=1 tid=0x0805b220 nid=0x767c runnable [0xbfffc000..0xbfffd538] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) - locked <0x49d58e18> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:450) at java.net.ServerSocket.accept(ServerSocket.java:421) at org.apache.catalina.core.StandardServer.await(StandardServer.java:526) at org.apache.catalina.startup.Catalina.start(Catalina.java:521) at org.apache.catalina.startup.Catalina.execute(Catalina.java:400) at org.apache.catalina.startup.Catalina.process(Catalina.java:180) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39 ) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl .java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:203) "VM Thread" prio=1 tid=0x081142b8 nid=0x7682 runnable "GC task thread#0 (ParallelGC)" prio=1 tid=0x080e98b8 nid=0x767e runnable "GC task thread#1 (ParallelGC)" prio=1 tid=0x080e9ca0 nid=0x767f runnable "GC task thread#2 (ParallelGC)" prio=1 tid=0x080ea088 nid=0x7680 runnable "GC task thread#3 (ParallelGC)" prio=1 tid=0x080ea868 nid=0x7681 runnable "VM Periodic Task Thread" prio=1 tid=0x08127d70 nid=0x768a waiting on condition --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
