Tomcat shutdown issues
Good day, I've been working on closing all my threads in my applications that run on Tomcat. I recently upgraded to 7.0.14 to get better messages concerning why Tomcat wasn't shutting down properly. I have stopped all my servlets and threads, but I'm still having trouble. Below is the thread dump of the most recent shutdown and my catalina log. I don't see any of my threads in there, so what is keeping Tomcat alive? 2011-06-10 09:07:32 Full thread dump Java HotSpot(TM) 64-Bit Server VM (19.0-b09 mixed mode): DestroyJavaVM prio=3D10 tid=3D0x5d1e9800 nid=3D0x5155 waiting on condition [0x] java.lang.Thread.State: RUNNABLE Worker-JM prio=3D10 tid=3D0x5d7a1800 nid=3D0x521f in Object.wait() [0x42ce2000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0xb69c2050 (a java.util.ArrayList) at org.eclipse.core.internal.jobs.InternalWorker.run(InternalWorker.java:58) - locked 0xb69c2050 (a java.util.ArrayList) Bundle File Closer daemon prio=3D10 tid=3D0x5e223800 nid=3D0x521d in Object.wait() [0x42be1000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0xb6a67768 (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread) at java.lang.Object.wait(Object.java:485) at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent(EventManager.java:397) - locked 0xb6a67768 (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread) at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:333) Start Level Event Dispatcher daemon prio=3D10 tid=3D0x5d5fc000 nid=3D0x521b in Object.wait() [0x42ae] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0xb68c5d38 (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread) at java.lang.Object.wait(Object.java:485) at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent(EventManager.java:397) - locked 0xb68c5d38 (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread) at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:333) Framework Event Dispatcher daemon prio=3D10 tid=3D0x5d956800 nid=3D0x521a in Object.wait() [0x429df000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0xb68f7278 (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread) at java.lang.Object.wait(Object.java:485) at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent(EventManager.java:397) - locked 0xb68f7278 (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread) at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:333) GC Daemon daemon prio=3D10 tid=3D0x5d299000 nid=3D0x5160 in Object.wait() [0x40dea000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0xb48180b8 (a sun.misc.GC$LatencyLock) at sun.misc.GC$Daemon.run(GC.java:100) - locked 0xb48180b8 (a sun.misc.GC$LatencyLock) Low Memory Detector daemon prio=3D10 tid=3D0x5ce8b000 nid=3D0x515e runnable [0x] java.lang.Thread.State: RUNNABLE CompilerThread1 daemon prio=3D10 tid=3D0x5ce88800 nid=3D0x515d waiting on condition [0x] java.lang.Thread.State: RUNNABLE CompilerThread0 daemon prio=3D10 tid=3D0x5ce85000 nid=3D0x515c waiting on condition [0x] java.lang.Thread.State: RUNNABLE Signal Dispatcher daemon prio=3D10 tid=3D0x5ce82800 nid=3D0x515b runnable [0x] java.lang.Thread.State: RUNNABLE Finalizer daemon prio=3D10 tid=3D0x5ce5e000 nid=3D0x515a in Object.wait() [0x4136b000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0xb4818800 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) - locked 0xb4818800 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) Reference Handler daemon prio=3D10 tid=3D0x5ce5c000 nid=3D0x5159 in Object.wait() [0x4126a000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0xb48180a8 (a java.lang.ref.Reference$Lock)
RE: Tomcat shutdown issues
I just installed a plug-in to try and get YourKit going to find the leaks. I didn't realize that I had started Tomcat from inside Eclipse. I'll take out what I did. Thanks, April - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat shutdown issues
Taking out what I did for the YourKit plug-in didn't help. Next, I shutdown my Eclipse and then started and stopped Tomcat on the server. The org.eclipse threads were still listed in the thread dump. I recently upgraded B.I.R.T. to version 2.2 on Tomcat and it uses osgi jar files. I'll look to see if there is an issue there or if I need to do something else so Tomcat will shutdown properly. Thanks, April, Systems - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6 not shutting down
Chuck, Thank you greatly for the help. I searched our code and found the timer in question. I added a call to cancel this timer when the other thread dies and Tomcat now shuts down properly. Thank you, April -Original Message- From: April Easton Sent: Wednesday, December 15, 2010 3:58 PM To: 'users@tomcat.apache.org' Subject: Tomcat 6 not shutting down Good day, I've been searching for information on how to determine what process is keeping Tomcat from shutting down properly. From other posts, I've been able to determine that the threads started from an Axis2 1.5.3 servlet on Tomcat were still processing when the shutdown.sh command was issued. I have worked on that issue and now the threads created are being terminated no more than 2 minutes after the ServletContextListener executes the destroy method. I don't see the threads that I created in this new dump file. How do I determine what process is keeping Tomcat from shutting down properly? Is it a problem if I'm not terminating the Axis2 threads immediately after the servlet is destroyed? I'm using Tomcat 6.0.18 on RHEL5 update 5 with Java 6 update 7. I have included the dump file below. Thank you in advance for your help. 2010-12-15 14:46:38 Full thread dump Java HotSpot(TM) 64-Bit Server VM (10.0-b23 mixed mode): DestroyJavaVM prio=10 tid=0x51218400 nid=0x666e waiting on condition [0x..0x416a3d00] java.lang.Thread.State: RUNNABLE http-8443-1 daemon prio=10 tid=0x50fd0c00 nid=0x677e in Object.wait() [0x40fcb000..0x40fcbb10] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x2aaade50fbf0 (a org.apache.tomcat.util.net.JIoEndpoint$Worker) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416) - locked 0x2aaade50fbf0 (a org.apache.tomcat.util.net.JIoEndpoint$Worker) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442) at java.lang.Thread.run(Thread.java:619) http-8088-1 daemon prio=10 tid=0x50cd7800 nid=0x677d in Object.wait() [0x40b71000..0x40b71c90] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x2aaadff4b5a8 (a org.apache.tomcat.util.net.JIoEndpoint$Worker) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416) - locked 0x2aaadff4b5a8 (a org.apache.tomcat.util.net.JIoEndpoint$Worker) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442) at java.lang.Thread.run(Thread.java:619) TP-Processor4 daemon prio=10 tid=0x5127c800 nid=0x66ce in Object.wait() [0x41f5..0x41f50d90] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x2aaaed3fa590 (a org.apache.jk.common.ChannelSocket) at java.lang.Object.wait(Object.java:485) at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:300) - locked 0x2aaaed3fa590 (a org.apache.jk.common.ChannelSocket) at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:660) at org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:870) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690) at java.lang.Thread.run(Thread.java:619) http-8443-Acceptor-0 daemon prio=10 tid=0x509ac000 nid=0x66ca waiting on condition [0x41d4e000..0x41d4eb90] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:302) at java.lang.Thread.run(Thread.java:619) http-8088-Acceptor-0 daemon prio=10 tid=0x517c9c00 nid=0x66c7 waiting on condition [0x41aa7000..0x41aa7b10] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:302) at java.lang.Thread.run(Thread.java:619) Thread-6 daemon prio=10 tid=0x2aaaf89d6800 nid=0x66a2 waiting on condition [0x40dc9000..0x40dc9c90] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at oracle.jdbc.pool.OracleImplicitConnectionCacheThread.run(OracleImplicitConnectionCacheThread.java:87) Timer-2 prio=10 tid=0x50b72800 nid=0x6697 in Object.wait() [0x4096f000..0x4096fd10] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x2aaab3e20cf0
Tomcat 6 not shutting down
Good day, I've been searching for information on how to determine what process is keeping Tomcat from shutting down properly. From other posts, I've been able to determine that the threads started from an Axis2 1.5.3 servlet on Tomcat were still processing when the shutdown.sh command was issued. I have worked on that issue and now the threads created are being terminated no more than 2 minutes after the ServletContextListener executes the destroy method. I don't see the threads that I created in this new dump file. How do I determine what process is keeping Tomcat from shutting down properly? Is it a problem if I'm not terminating the Axis2 threads immediately after the servlet is destroyed? I'm using Tomcat 6.0.18 on RHEL5 update 5 with Java 6 update 7. I have included the dump file below. Thank you in advance for your help. 2010-12-15 14:46:38 Full thread dump Java HotSpot(TM) 64-Bit Server VM (10.0-b23 mixed mode): DestroyJavaVM prio=10 tid=0x51218400 nid=0x666e waiting on condition [0x..0x416a3d00] java.lang.Thread.State: RUNNABLE http-8443-1 daemon prio=10 tid=0x50fd0c00 nid=0x677e in Object.wait() [0x40fcb000..0x40fcbb10] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x2aaade50fbf0 (a org.apache.tomcat.util.net.JIoEndpoint$Worker) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416) - locked 0x2aaade50fbf0 (a org.apache.tomcat.util.net.JIoEndpoint$Worker) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442) at java.lang.Thread.run(Thread.java:619) http-8088-1 daemon prio=10 tid=0x50cd7800 nid=0x677d in Object.wait() [0x40b71000..0x40b71c90] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x2aaadff4b5a8 (a org.apache.tomcat.util.net.JIoEndpoint$Worker) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416) - locked 0x2aaadff4b5a8 (a org.apache.tomcat.util.net.JIoEndpoint$Worker) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442) at java.lang.Thread.run(Thread.java:619) TP-Processor4 daemon prio=10 tid=0x5127c800 nid=0x66ce in Object.wait() [0x41f5..0x41f50d90] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x2aaaed3fa590 (a org.apache.jk.common.ChannelSocket) at java.lang.Object.wait(Object.java:485) at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:300) - locked 0x2aaaed3fa590 (a org.apache.jk.common.ChannelSocket) at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:660) at org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:870) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690) at java.lang.Thread.run(Thread.java:619) http-8443-Acceptor-0 daemon prio=10 tid=0x509ac000 nid=0x66ca waiting on condition [0x41d4e000..0x41d4eb90] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:302) at java.lang.Thread.run(Thread.java:619) http-8088-Acceptor-0 daemon prio=10 tid=0x517c9c00 nid=0x66c7 waiting on condition [0x41aa7000..0x41aa7b10] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:302) at java.lang.Thread.run(Thread.java:619) Thread-6 daemon prio=10 tid=0x2aaaf89d6800 nid=0x66a2 waiting on condition [0x40dc9000..0x40dc9c90] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at oracle.jdbc.pool.OracleImplicitConnectionCacheThread.run(OracleImplicitConnectionCacheThread.java:87) Timer-2 prio=10 tid=0x50b72800 nid=0x6697 in Object.wait() [0x4096f000..0x4096fd10] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x2aaab3e20cf0 (a java.util.TaskQueue) at java.lang.Object.wait(Object.java:485) at java.util.TimerThread.mainLoop(Timer.java:483) - locked 0x2aaab3e20cf0 (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) Timer-0 daemon prio=10 tid=0x2aaaf89c4800 nid=0x6693 in Object.wait() [0x4086e000..0x4086ea10]
