I generated this with VisualVM. I will try to generate in a different way, or to inspect the threads within the 60 seconds window as you suggested.
Thank you, Igal On Jul 1, 2014 9:42 AM, "Mark Thomas" <ma...@apache.org> wrote: > On 01/07/2014 17:13, Igal @ getRailo.org wrote: > > Hi Mark -- Thank you for replying. > > > > Here is a thread dump from about 30 seconds after I request to stop the > > service. Can you tell which thread is preventing the JVM from exiting? > > No, because the thread dump doesn't show whether a thread is a daemon > thread or not (which is odd - I thought the standard thread dump did > show that). My guess would be the timer threads. Use one of the many > tools that come with the JDK (or a profiler) to look at the threads and > see which non-daemon threads are still running. > > Mark > > > Thanks again! > > > > > > 2014-07-01 09:09:00 > > Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.51-b03 mixed > mode): > > > > "DestroyJavaVM" - Thread t@78 > > java.lang.Thread.State: RUNNABLE > > > > Locked ownable synchronizers: > > - None > > > > "Thread-24" - Thread t@61 > > java.lang.Thread.State: RUNNABLE > > at sun.net.dns.ResolverConfigurationImpl.notifyAddrChange0(Native > > Method) > > at > > sun.net.dns.ResolverConfigurationImpl$AddressChangeListener.run(Unknown > > Source) > > > > Locked ownable synchronizers: > > - None > > > > "RMI TCP Connection(4)-192.168.0.7" - Thread t@48 > > java.lang.Thread.State: RUNNABLE > > at sun.management.ThreadImpl.dumpThreads0(Native Method) > > at sun.management.ThreadImpl.dumpAllThreads(Unknown Source) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > > at java.lang.reflect.Method.invoke(Unknown Source) > > at sun.reflect.misc.Trampoline.invoke(Unknown Source) > > at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > > at java.lang.reflect.Method.invoke(Unknown Source) > > at sun.reflect.misc.MethodUtil.invoke(Unknown Source) > > at > > com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(Unknown > > Source) > > at > > com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(Unknown > > Source) > > at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(Unknown > Source) > > at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(Unknown > Source) > > at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source) > > at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source) > > at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source) > > at javax.management.StandardMBean.invoke(Unknown Source) > > at > > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown > > Source) > > at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source) > > at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown > > Source) > > at javax.management.remote.rmi.RMIConnectionImpl.access$300(Unknown > > Source) > > at > > > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown > > Source) > > at > > > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown > > Source) > > at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown > Source) > > at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > > at java.lang.reflect.Method.invoke(Unknown Source) > > at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source) > > at sun.rmi.transport.Transport$1.run(Unknown Source) > > at sun.rmi.transport.Transport$1.run(Unknown Source) > > at java.security.AccessController.doPrivileged(Native Method) > > at sun.rmi.transport.Transport.serviceCall(Unknown Source) > > at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) > > at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown > > Source) > > at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown > > Source) > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > > at java.lang.Thread.run(Unknown Source) > > > > Locked ownable synchronizers: > > - locked <6cb0e9eb> (a > java.util.concurrent.ThreadPoolExecutor$Worker) > > > > "RMI TCP Connection(3)-192.168.0.7" - Thread t@47 > > java.lang.Thread.State: TIMED_WAITING > > at java.lang.Object.wait(Native Method) > > - waiting on <18e5eaed> (a > > com.sun.jmx.remote.internal.ArrayNotificationBuffer) > > at > > > com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(Unknown > > Source) > > at > > > com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(Unknown > > Source) > > at > > com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(Unknown > > Source) > > at javax.management.remote.rmi.RMIConnectionImpl$3.run(Unknown > Source) > > at javax.management.remote.rmi.RMIConnectionImpl$3.run(Unknown > Source) > > at > > javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(Unknown > > Source) > > at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > > at java.lang.reflect.Method.invoke(Unknown Source) > > at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source) > > at sun.rmi.transport.Transport$1.run(Unknown Source) > > at sun.rmi.transport.Transport$1.run(Unknown Source) > > at java.security.AccessController.doPrivileged(Native Method) > > at sun.rmi.transport.Transport.serviceCall(Unknown Source) > > at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) > > at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown > > Source) > > at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown > > Source) > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > > at java.lang.Thread.run(Unknown Source) > > > > Locked ownable synchronizers: > > - locked <5f717b35> (a > java.util.concurrent.ThreadPoolExecutor$Worker) > > > > "JMX server connection timeout 44" - Thread t@44 > > java.lang.Thread.State: TIMED_WAITING > > at java.lang.Object.wait(Native Method) > > - waiting on <3d104456> (a [I) > > at > > com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown > > Source) > > at java.lang.Thread.run(Unknown Source) > > > > Locked ownable synchronizers: > > - None > > > > "RMI TCP Connection(5)-192.168.0.7" - Thread t@43 > > java.lang.Thread.State: RUNNABLE > > at java.net.SocketInputStream.socketRead0(Native Method) > > at java.net.SocketInputStream.read(Unknown Source) > > at java.net.SocketInputStream.read(Unknown Source) > > at java.io.BufferedInputStream.fill(Unknown Source) > > at java.io.BufferedInputStream.read(Unknown Source) > > - locked <1e46a59e> (a java.io.BufferedInputStream) > > at java.io.FilterInputStream.read(Unknown Source) > > at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) > > at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown > > Source) > > at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown > > Source) > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > > at java.lang.Thread.run(Unknown Source) > > > > Locked ownable synchronizers: > > - locked <55173444> (a > java.util.concurrent.ThreadPoolExecutor$Worker) > > > > "RMI Scheduler(0)" - Thread t@42 > > java.lang.Thread.State: TIMED_WAITING > > at sun.misc.Unsafe.park(Native Method) > > - parking to wait for <76f6896b> (a > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > > at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown > > Source) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown > > Source) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown > > Source) > > at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > > at java.lang.Thread.run(Unknown Source) > > > > Locked ownable synchronizers: > > - None > > > > "RMI TCP Connection(idle)" - Thread t@41 > > java.lang.Thread.State: TIMED_WAITING > > at sun.misc.Unsafe.park(Native Method) > > - parking to wait for <6dd7c555> (a > > java.util.concurrent.SynchronousQueue$TransferStack) > > at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) > > at > > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown > > Source) > > at > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown > > Source) > > at java.util.concurrent.SynchronousQueue.poll(Unknown Source) > > at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > > at java.lang.Thread.run(Unknown Source) > > > > Locked ownable synchronizers: > > - None > > > > "Thread-12" - Thread t@34 > > java.lang.Thread.State: TIMED_WAITING > > at java.lang.Thread.sleep(Native Method) > > at > > > railo.runtime.schedule.ScheduledTaskThread.sleepEL(ScheduledTaskThread.java:148) > > > > at > > > railo.runtime.schedule.ScheduledTaskThread._run(ScheduledTaskThread.java:109) > > > > at > > > railo.runtime.schedule.ScheduledTaskThread.run(ScheduledTaskThread.java:70) > > > > Locked ownable synchronizers: > > - None > > > > "Thread-11" - Thread t@33 > > java.lang.Thread.State: TIMED_WAITING > > at java.lang.Thread.sleep(Native Method) > > at > > > railo.runtime.schedule.ScheduledTaskThread.sleepEL(ScheduledTaskThread.java:148) > > > > at > > > railo.runtime.schedule.ScheduledTaskThread._run(ScheduledTaskThread.java:109) > > > > at > > > railo.runtime.schedule.ScheduledTaskThread.run(ScheduledTaskThread.java:70) > > > > Locked ownable synchronizers: > > - None > > > > "Thread-10" - Thread t@32 > > java.lang.Thread.State: TIMED_WAITING > > at java.lang.Thread.sleep(Native Method) > > at > > > railo.runtime.schedule.ScheduledTaskThread.sleepEL(ScheduledTaskThread.java:148) > > > > at > > > railo.runtime.schedule.ScheduledTaskThread._run(ScheduledTaskThread.java:109) > > > > at > > > railo.runtime.schedule.ScheduledTaskThread.run(ScheduledTaskThread.java:70) > > > > Locked ownable synchronizers: > > - None > > > > "Thread-9" - Thread t@27 > > java.lang.Thread.State: TIMED_WAITING > > at java.lang.Thread.sleep(Native Method) > > at > > > railo.runtime.schedule.ScheduledTaskThread.sleepEL(ScheduledTaskThread.java:148) > > > > at > > > railo.runtime.schedule.ScheduledTaskThread._run(ScheduledTaskThread.java:109) > > > > at > > > railo.runtime.schedule.ScheduledTaskThread.run(ScheduledTaskThread.java:70) > > > > Locked ownable synchronizers: > > - None > > > > "Thread-8" - Thread t@26 > > java.lang.Thread.State: TIMED_WAITING > > at java.lang.Thread.sleep(Native Method) > > at > > > railo.runtime.schedule.ScheduledTaskThread.sleepEL(ScheduledTaskThread.java:148) > > > > at > > > railo.runtime.schedule.ScheduledTaskThread._run(ScheduledTaskThread.java:109) > > > > at > > > railo.runtime.schedule.ScheduledTaskThread.run(ScheduledTaskThread.java:70) > > > > Locked ownable synchronizers: > > - None > > > > "Thread-6" - Thread t@22 > > java.lang.Thread.State: TIMED_WAITING > > at java.lang.Thread.sleep(Native Method) > > at railo.runtime.engine.Controler.run(Controler.java:73) > > > > Locked ownable synchronizers: > > - None > > > > "ehcache.data" - Thread t@20 > > java.lang.Thread.State: TIMED_WAITING > > at sun.misc.Unsafe.park(Native Method) > > - parking to wait for <62ea9dd9> (a > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > > at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown > > Source) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown > > Source) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown > > Source) > > at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > > at java.lang.Thread.run(Unknown Source) > > > > Locked ownable synchronizers: > > - None > > > > "ehcache_c71f0802d667559429fa52d6db704210" - Thread t@19 > > java.lang.Thread.State: TIMED_WAITING > > at java.lang.Object.wait(Native Method) > > - waiting on <1921444b> (a java.util.TaskQueue) > > at java.util.TimerThread.mainLoop(Unknown Source) > > at java.util.TimerThread.run(Unknown Source) > > > > Locked ownable synchronizers: > > - None > > > > "GC Daemon" - Thread t@15 > > java.lang.Thread.State: TIMED_WAITING > > at java.lang.Object.wait(Native Method) > > - waiting on <2baf4a4a> (a sun.misc.GC$LatencyLock) > > at sun.misc.GC$Daemon.run(Unknown Source) > > > > Locked ownable synchronizers: > > - None > > > > "RMI TCP Accept-0" - Thread t@14 > > java.lang.Thread.State: RUNNABLE > > at java.net.DualStackPlainSocketImpl.accept0(Native Method) > > at java.net.DualStackPlainSocketImpl.socketAccept(Unknown Source) > > at java.net.AbstractPlainSocketImpl.accept(Unknown Source) > > at java.net.PlainSocketImpl.accept(Unknown Source) > > - locked <5521134e> (a java.net.SocksSocketImpl) > > at java.net.ServerSocket.implAccept(Unknown Source) > > at java.net.ServerSocket.accept(Unknown Source) > > at > > sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(Unknown > > Source) > > at > > sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown > > Source) > > at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source) > > at java.lang.Thread.run(Unknown Source) > > > > Locked ownable synchronizers: > > - None > > > > "RMI TCP Accept-8079" - Thread t@13 > > java.lang.Thread.State: RUNNABLE > > at java.net.DualStackPlainSocketImpl.accept0(Native Method) > > at java.net.DualStackPlainSocketImpl.socketAccept(Unknown Source) > > at java.net.AbstractPlainSocketImpl.accept(Unknown Source) > > at java.net.PlainSocketImpl.accept(Unknown Source) > > - locked <6e7a918b> (a java.net.SocksSocketImpl) > > at java.net.ServerSocket.implAccept(Unknown Source) > > at java.net.ServerSocket.accept(Unknown Source) > > at > > sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown > > Source) > > at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source) > > at java.lang.Thread.run(Unknown Source) > > > > Locked ownable synchronizers: > > - None > > > > "RMI TCP Accept-0" - Thread t@12 > > java.lang.Thread.State: RUNNABLE > > at java.net.DualStackPlainSocketImpl.accept0(Native Method) > > at java.net.DualStackPlainSocketImpl.socketAccept(Unknown Source) > > at java.net.AbstractPlainSocketImpl.accept(Unknown Source) > > at java.net.PlainSocketImpl.accept(Unknown Source) > > - locked <30e97557> (a java.net.SocksSocketImpl) > > at java.net.ServerSocket.implAccept(Unknown Source) > > at java.net.ServerSocket.accept(Unknown Source) > > at > > sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown > > Source) > > at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source) > > at java.lang.Thread.run(Unknown Source) > > > > Locked ownable synchronizers: > > - None > > > > "AsyncFileHandlerWriter-550055638" - Thread t@11 > > java.lang.Thread.State: TIMED_WAITING > > at sun.misc.Unsafe.park(Native Method) > > - parking to wait for <13656ae> (a > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > > at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown > > Source) > > at java.util.concurrent.LinkedBlockingDeque.pollFirst(Unknown Source) > > at java.util.concurrent.LinkedBlockingDeque.poll(Unknown Source) > > at > > > org.apache.juli.AsyncFileHandler$LoggerThread.run(AsyncFileHandler.java:144) > > > > > > Locked ownable synchronizers: > > - None > > > > "Attach Listener" - Thread t@5 > > java.lang.Thread.State: RUNNABLE > > > > Locked ownable synchronizers: > > - None > > > > "Signal Dispatcher" - Thread t@4 > > java.lang.Thread.State: RUNNABLE > > > > Locked ownable synchronizers: > > - None > > > > "Finalizer" - Thread t@3 > > java.lang.Thread.State: WAITING > > at java.lang.Object.wait(Native Method) > > - waiting on <1674ba21> (a java.lang.ref.ReferenceQueue$Lock) > > at java.lang.ref.ReferenceQueue.remove(Unknown Source) > > at java.lang.ref.ReferenceQueue.remove(Unknown Source) > > at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) > > > > Locked ownable synchronizers: > > - None > > > > "Reference Handler" - Thread t@2 > > java.lang.Thread.State: WAITING > > at java.lang.Object.wait(Native Method) > > - waiting on <4967d80b> (a java.lang.ref.Reference$Lock) > > at java.lang.Object.wait(Object.java:503) > > at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) > > > > Locked ownable synchronizers: > > - None > > > > > > > > > > On 6/30/2014 2:45 PM, Mark Thomas wrote: > >> On 30/06/2014 22:40, Igal @ getRailo.org wrote: > >>> Does anyone know why Tomcat takes a full minute to stop? > >> Tomcat doesn't. It takes a few seconds to stop. Your application might > >> well take a full minute to stop. Try taking some thread dumps. > >> > >> Mark > >> > >> > >>> Is there a paid support for Tomcat? > >>> > >>> > >>> > >>> On 6/27/2014 8:29 PM, Igal @ getRailo.org wrote: > >>>> And again there is a full minute between "Service stop thread > >>>> completed." and "Run service finished.", which doesn't look like a > >>>> coincidence. > >>>> > >>>> [2014-06-27 20:24:14] [info] [ 3156] Stopping service... > >>>> [2014-06-27 20:24:15] [info] [ 3156] Service stop thread completed. > >>>> [2014-06-27 20:25:15] [info] [ 2520] Run service finished. > >>>> [2014-06-27 20:25:15] [info] [ 2520] Commons Daemon procrun finished > >>>> Can anyone shed some light on this? > >>>> > >>>> > >>>> Igal > >>>> > >>>> On 6/27/2014 2:08 PM, Igal @ getRailo.org wrote: > >>>>> I installed Tomcat 8.09 on Windows 2008R2 > >>>>> > >>>>> When trying to stop the service it takes a long time, and usually I > >>>>> get a prompt on the screen saying that there was an error. > >>>>> > >>>>> Windows Event Log did not show anything about this, but > >>>>> commons-daemon log shows: > >>>>> > >>>>> [2014-06-27 13:59:53] [info] [ 5504] Commons Daemon procrun > >>>>> (1.0.15.0 64-bit) started > >>>>> [2014-06-27 13:59:53] [info] [ 5504] Running 'Tomcat8' Service... > >>>>> [2014-06-27 13:59:53] [info] [ 5176] Starting service... > >>>>> [2014-06-27 13:59:54] [info] [ 5176] Service started in 1185 ms. > >>>>> *[2014-06-27 14:00:08] [info] [ 3472] Stopping service... > >>>>> [2014-06-27 14:00:09] [info] [ 3472] Service stop thread completed. > >>>>> [2014-06-27 14:01:09] [info] [ 5504] Run service finished. > >>>>> [2014-06-27 14:01:09] [info] [ 5504] Commons Daemon procrun > finished* > >>>>> Why does it take a full minute from "Service stop thread completed." > >>>>> where the stop command has yet to return, until "Run service > >>>>> finished." when it actually does stop the service? > >>>>> > >>>>> Is there any way to speed that up? > >>>>> > >>>>> TIA > >>>>> > >>>>> > >>>>> -- > >>>>> Igal Sapir > >>>>> Railo Core Developer > >>>>> http://getRailo.org/ > >>>> -- > >>>> Igal Sapir > >>>> Railo Core Developer > >>>> http://getRailo.org/ > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > >> > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
