lee created AMQ-6041: ------------------------ Summary: Activemq have a deadLock in running for a long time Key: AMQ-6041 URL: https://issues.apache.org/jira/browse/AMQ-6041 Project: ActiveMQ Issue Type: Bug Components: activemq-leveldb-store Affects Versions: 5.11.0 Environment: development and product Reporter: lee
my activemq is a Cluster that has three activemq single instance and using replicated stored levelDB . when the cluster is running for a long time ( a few days), this situation is that JVM Old Generation is filled,and eden and one of survivor 0 or 1 is filled., but not privider service and all of port include 61616 is dead when producer is calling. this time three instance's process is running , about half an hour , zookeeper is selected and another instance start all port and providers services i see this stack trace in jstack, the content is follow: at java.lang.Thread.run(Thread.java:745) "qtp2054301670-138 Selector0" prio=10 tid=0x00007f5e3cccd800 nid=0xffb runnable [0x00007f5db3cfb000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x00000000b5628fe0> (a sun.nio.ch.Util$2) - locked <0x00000000b5628fc8> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000b55fb490> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) "HashSessionScavenger-2" daemon prio=10 tid=0x00007f5e3cc5a800 nid=0xff7 in Object.wait() [0x00007f5e08156000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000b5621138> (a java.util.TaskQueue) at java.util.TimerThread.mainLoop(Timer.java:552) - locked <0x00000000b5621138> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:505) "HashSessionScavenger-1" daemon prio=10 tid=0x00007f5e3cc63000 nid=0xff6 in Object.wait() [0x00007f5e08257000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000b5621168> (a java.util.TaskQueue) at java.util.TimerThread.mainLoop(Timer.java:552) - locked <0x00000000b5621168> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:505) "HashSessionScavenger-0" daemon prio=10 tid=0x00007f5e3cbac000 nid=0xff5 in Object.wait() [0x00007f5e08358000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000b5621198> (a java.util.TaskQueue) at java.util.TimerThread.mainLoop(Timer.java:552) - locked <0x00000000b5621198> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:505) "qtp819231520-130" prio=10 tid=0x00007f5e3cb2a800 nid=0xff1 waiting on condition [0x00007f5e08459000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b563b640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) "qtp819231520-129" prio=10 tid=0x00007f5e3cb28800 nid=0xff0 waiting on condition [0x00007f5e0855a000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b563b640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) "qtp819231520-128" prio=10 tid=0x00007f5e3cb26800 nid=0xfef waiting on condition [0x00007f5e0865b000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b563b640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) "qtp819231520-127" prio=10 tid=0x00007f5e3cb24800 nid=0xfee waiting on condition [0x00007f5e088f3000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b563b640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) "qtp819231520-126" prio=10 tid=0x00007f5e3cb22000 nid=0xfed waiting on condition [0x00007f5e089f4000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b563b640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) "qtp819231520-125 Acceptor0 SelectChannelConnector@0.0.0.0:61614" prio=10 tid=0x00007f5e3cb20800 nid=0xfec runnable [0x00007f5e08af5000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241) - locked <0x00000000b563cdb0> (a java.lang.Object) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) "qtp819231520-124" prio=10 tid=0x00007f5e3cb1d000 nid=0xfeb waiting on condition [0x00007f5e08bf6000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b563b640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) "qtp819231520-123 Selector0" prio=10 tid=0x00007f5e3cb1c800 nid=0xfea runnable [0x00007f5e08cf7000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x00000000b563d6e0> (a sun.nio.ch.Util$2) - locked <0x00000000b563d4e8> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000b5621958> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) "ActiveMQ Transport Server: mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=1048576000" daemon prio=10 tid=0x00007f5e3c64c000 nid=0xfe9 runnable [0x00007f5e0900d000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:530) at java.net.ServerSocket.accept(ServerSocket.java:498) at org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:350) at java.lang.Thread.run(Thread.java:745) "ActiveMQ Transport Server Thread Handler: mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=1048576000" daemon prio=10 tid=0x00007f5e3c754000 nid=0xfe8 waiting on condition [0x00007f5e0910e000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b5651560> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.activemq.transport.tcp.TcpTransportServer$2.run(TcpTransportServer.java:425) at java.lang.Thread.run(Thread.java:745) "ActiveMQ Transport Server: stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=1048576000" daemon prio=10 tid=0x00007f5e3c745800 nid=0xfe7 runnable [0x00007f5e0920f000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:530) at java.net.ServerSocket.accept(ServerSocket.java:498) at org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:350) at java.lang.Thread.run(Thread.java:745) "ActiveMQ Transport Server Thread Handler: stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=1048576000" daemon prio=10 tid=0x00007f5e3c231800 nid=0xfe6 runnable [0x00007f5e09310000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.createEntry(HashMap.java:897) at java.util.HashMap.addEntry(HashMap.java:884) at java.util.HashMap.put(HashMap.java:505) at org.apache.activemq.transport.tcp.TcpTransportServer.handleSocket(TcpTransportServer.java:488) at org.apache.activemq.transport.tcp.TcpTransportServer$2.run(TcpTransportServer.java:428) at java.lang.Thread.run(Thread.java:745) "ActiveMQ Transport Server: amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=1048576000" daemon prio=10 tid=0x00007f5e3c230000 nid=0xfe5 runnable [0x00007f5e09411000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:530) at java.net.ServerSocket.accept(ServerSocket.java:498) at org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:350) at java.lang.Thread.run(Thread.java:745) "ActiveMQ Transport Server Thread Handler: amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=1048576000" daemon prio=10 tid=0x00007f5e3c76b000 nid=0xfe4 waiting on condition [0x00007f5e09512000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b5657bc0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.activemq.transport.tcp.TcpTransportServer$2.run(TcpTransportServer.java:425) at java.lang.Thread.run(Thread.java:745) "hawtdispatch timer" daemon prio=10 tid=0x00007f5dc80da800 nid=0x3ac in Object.wait() [0x00007f5e28382000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000b521a678> (a java.lang.Object) at java.lang.Object.wait(Object.java:503) at org.fusesource.hawtdispatch.internal.TimerThread.run(TimerThread.java:186) - locked <0x00000000b521a678> (a java.lang.Object) "hawtdispatch-DEFAULT-4" daemon prio=10 tid=0x00007f5dc8042800 nid=0x3ab runnable [0x00007f5e28483000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x00000000b5206048> (a sun.nio.ch.Util$2) - locked <0x00000000b5206030> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000b5205ef8> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) at org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52) at org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182) at org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130) at org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75) "hawtdispatch-DEFAULT-3" daemon prio=10 tid=0x00007f5dc8041800 nid=0x3aa runnable [0x00007f5e28584000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x00000000b51f58f0> (a sun.nio.ch.Util$2) - locked <0x00000000b51f58d8> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000b51f57a0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) at org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52) at org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182) at org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130) at org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75) "hawtdispatch-DEFAULT-2" daemon prio=10 tid=0x00007f5dc8049800 nid=0x3a9 runnable [0x00007f5e28685000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x00000000b51e5198> (a sun.nio.ch.Util$2) - locked <0x00000000b51e5180> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000b51e5048> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) at org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52) at org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182) at org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130) at org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75) "hawtdispatch-DEFAULT-1" daemon prio=10 tid=0x00007f5dc803f000 nid=0x3a8 runnable [0x00007f5e28786000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x00000000b51d2aa8> (a sun.nio.ch.Util$2) - locked <0x00000000b51d2a90> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000b51d2958> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) at org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52) at org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182) at org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130) at org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75) "ZooKeeper state change dispatcher thread" daemon prio=10 tid=0x00007f5e3ca77800 nid=0x3a5 waiting on condition [0x00007f5e29190000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b4fb04b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.activemq.leveldb.replicated.groups.ZKClient$StateChangeDispatcher.run(ZKClient.java:347) "RMI RenewClean-[172.30.0.21:57440]" daemon prio=10 tid=0x00007f5dd4003800 nid=0x3a3 in Object.wait() [0x00007f5e28b8a000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000b55a2858> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) - locked <0x00000000b55a2858> (a java.lang.ref.ReferenceQueue$Lock) at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:535) at java.lang.Thread.run(Thread.java:745) "RMI Scheduler(0)" daemon prio=10 tid=0x00007f5de4018000 nid=0x3a2 waiting on condition [0x00007f5e28c8b000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b4d5bc38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) "GC Daemon" daemon prio=10 tid=0x00007f5de4004800 nid=0x3a0 in Object.wait() [0x00007f5e28e8d000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000b4f9fdc8> (a sun.misc.GC$LatencyLock) at sun.misc.GC$Daemon.run(GC.java:117) - locked <0x00000000b4f9fdc8> (a sun.misc.GC$LatencyLock) "RMI Reaper" prio=10 tid=0x00007f5de4002800 nid=0x39f in Object.wait() [0x00007f5e28f8e000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000b4d5adc8> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) - locked <0x00000000b4d5adc8> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:351) at java.lang.Thread.run(Thread.java:745) "RMI TCP Accept-0" daemon prio=10 tid=0x00007f5de4001000 nid=0x39e runnable [0x00007f5e2908f000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:530) at java.net.ServerSocket.accept(ServerSocket.java:498) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360) at java.lang.Thread.run(Thread.java:745) "RMI TCP Accept-1099" daemon prio=10 tid=0x00007f5e3ca22800 nid=0x39c runnable [0x00007f5e29291000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:530) at java.net.ServerSocket.accept(ServerSocket.java:498) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360) at java.lang.Thread.run(Thread.java:745) "RMI TCP Accept-0" daemon prio=10 tid=0x00007f5e3c29e800 nid=0x39a runnable [0x00007f5e296ce000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:530) at java.net.ServerSocket.accept(ServerSocket.java:498) at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360) at java.lang.Thread.run(Thread.java:745) "Service Thread" daemon prio=10 tid=0x00007f5e3c1ee000 nid=0x399 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread1" daemon prio=10 tid=0x00007f5e3c1eb800 nid=0x398 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread0" daemon prio=10 tid=0x00007f5e3c1e9800 nid=0x397 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" daemon prio=10 tid=0x00007f5e3c1e7000 nid=0x396 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Surrogate Locker Thread (Concurrent GC)" daemon prio=10 tid=0x00007f5e3c1e5000 nid=0x395 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Finalizer" daemon prio=10 tid=0x00007f5e3c1bc000 nid=0x394 in Object.wait() [0x00007f5e386c2000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000b4cc4740> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) - locked <0x00000000b4cc4740> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) "Reference Handler" daemon prio=10 tid=0x00007f5e3c1ba000 nid=0x393 in Object.wait() [0x00007f5e387c3000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000b4cc4358> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) - locked <0x00000000b4cc4358> (a java.lang.ref.Reference$Lock) "main" prio=10 tid=0x00007f5e3c009800 nid=0x38c waiting on condition [0x00007f5e4339d000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b5669f08> (a java.util.concurrent.CountDownLatch$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236) at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:118) at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57) at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:150) at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57) at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.activemq.console.Main.runTaskClass(Main.java:262) at org.apache.activemq.console.Main.main(Main.java:115) "VM Thread" prio=10 tid=0x00007f5e3c1b6000 nid=0x392 runnable "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f5e3c01b000 nid=0x38d runnable "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f5e3c01c800 nid=0x38e runnable "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f5e3c01e800 nid=0x38f runnable "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f5e3c020800 nid=0x390 runnable "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007f5e3c0a1000 nid=0x391 runnable "VM Periodic Task Thread" prio=10 tid=0x00007f5e3c2ae800 nid=0x39b waiting on condition JNI global references: 171 [work@lx-dev00 tool]$ cat 21activemq.log | grep '0x00000000b4cc4358' - waiting on <0x00000000b4cc4358> (a java.lang.ref.Reference$Lock) - locked <0x00000000b4cc4358> (a java.lang.ref.Reference$Lock) [work@lx-dev00 tool]$ cat 21activemq.log | grep '0x00000000b4cc4358' -C 10 at java.lang.Object.wait(Native Method) - waiting on <0x00000000b4cc4740> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) - locked <0x00000000b4cc4740> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) "Reference Handler" daemon prio=10 tid=0x00007f5e3c1ba000 nid=0x393 in Object.wait() [0x00007f5e387c3000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000b4cc4358> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) - locked <0x00000000b4cc4358> (a java.lang.ref.Reference$Lock) "main" prio=10 tid=0x00007f5e3c009800 nid=0x38c waiting on condition [0x00007f5e4339d000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b5669f08> (a java.util.concurrent.CountDownLatch$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236) [work@lx-dev00 tool]$ sz sz: need at least one file to send Try `sz --help' for more information. [work@lx-dev00 tool]$ sz 21activemq.log [work@lx-dev00 tool]$ ^C [work@lx-dev00 tool]$ cat 21activemq.log 2015-11-11 14:18:00 Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.80-b07 mixed mode): "main-EventThread" daemon prio=10 tid=0x00007f5dbc067000 nid=0x4b83 waiting on condition [0x00007f5db21f7000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b0998d18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) "main-SendThread(172.30.0.21:2181)" daemon prio=10 tid=0x00007f5dbc0ab000 nid=0x4b82 runnable [0x00007f5daf4c1000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x00000000b0988558> (a sun.nio.ch.Util$2) - locked <0x00000000b0988548> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000b0988430> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) "ActiveMQ BrokerService[test] Task-456" daemon prio=10 tid=0x00007f5dd400c000 nid=0x4b7f waiting on condition [0x00007f5db12df000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b526d470> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) "Attach Listener" daemon prio=10 tid=0x00007f5e04003800 nid=0x4986 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "ActiveMQ BrokerService[test] Task-445" daemon prio=10 tid=0x00007f5dd402c800 nid=0x4901 waiting on condition [0x00007f5db0bd8000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b6849cf0> (a java.util.concurrent.CountDownLatch$Sync) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1033) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:282) at org.apache.activemq.broker.TransportConnection.stop(TransportConnection.java:1063) at org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41) at org.apache.activemq.broker.TransportConnector.stop(TransportConnector.java:287) at org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41) at org.apache.activemq.broker.BrokerService.stopAllConnectors(BrokerService.java:2116) at org.apache.activemq.broker.BrokerService.stop(BrokerService.java:771) at org.apache.activemq.xbean.XBeanBrokerService.stop(XBeanBrokerService.java:122) at org.apache.activemq.leveldb.replicated.ElectingLevelDBStore$$anonfun$stop_master$2.apply$mcV$sp(ElectingLevelDBStore.scala:258) at org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:330) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) "ActiveMQ BrokerService[test] Task-435" daemon prio=10 tid=0x00007f5dbc037000 nid=0x48f3 runnable [0x00007f5db05d1000] java.lang.Thread.State: RUNNABLE at javax.management.ObjectName.getSerializedNameString(ObjectName.java:1666) at javax.management.ObjectName.toString(ObjectName.java:1757) at org.apache.activemq.broker.jmx.BrokerMBeanSupport.createProducerName(BrokerMBeanSupport.java:90) at org.apache.activemq.broker.jmx.ManagedRegionBroker.removeProducer(ManagedRegionBroker.java:277) at org.apache.activemq.broker.BrokerFilter.removeProducer(BrokerFilter.java:142) at org.apache.activemq.advisory.AdvisoryBroker.removeProducer(AdvisoryBroker.java:332) at org.apache.activemq.broker.CompositeDestinationBroker.removeProducer(CompositeDestinationBroker.java:75) at org.apache.activemq.broker.BrokerFilter.removeProducer(BrokerFilter.java:142) at org.apache.activemq.broker.MutableBrokerFilter.removeProducer(MutableBrokerFilter.java:147) at org.apache.activemq.broker.TransportConnection.processRemoveProducer(TransportConnection.java:636) at org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:741) at org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:846) - locked <0x00000000b5661c80> (a org.apache.activemq.broker.jmx.ManagedTransportConnection) at org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1187) at org.apache.activemq.broker.TransportConnection$4.run(TransportConnection.java:1117) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) "ActiveMQ Transport: tcp:///172.30.13.76:38807@61616" daemon prio=10 tid=0x00007f5dd001e000 nid=0x1040 waiting on condition [0x00007f5db1be8000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b6890fc8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282) at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731) at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:181) at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113) at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270) at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83) at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214) at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196) at java.lang.Thread.run(Thread.java:745) "qtp2054301670-145" prio=10 tid=0x00007f5e3ccdc000 nid=0x1002 waiting on condition [0x00007f5db35f4000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b5626338> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) "qtp2054301670-144" prio=10 tid=0x00007f5e3ccda000 nid=0x1001 waiting on condition [0x00007f5db36f5000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b5626338> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) "qtp2054301670-143" prio=10 tid=0x00007f5e3ccd6800 nid=0x1000 waiting on condition [0x00007f5db37f6000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b5626338> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) "qtp2054301670-142" prio=10 tid=0x00007f5e3ccd4800 nid=0xfff waiting on condition [0x00007f5db38f7000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b5626338> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) "qtp2054301670-141" prio=10 tid=0x00007f5e3ccd2800 nid=0xffe waiting on condition [0x00007f5db39f8000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b5626338> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) "qtp2054301670-140 Acceptor0 SelectChannelConnector@0.0.0.0:8161" prio=10 tid=0x00007f5e3cccf000 nid=0xffd runnable [0x00007f5db3af9000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241) - locked <0x00000000b56285b8> (a java.lang.Object) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) "qtp2054301670-139" prio=10 tid=0x00007f5e3ccce000 nid=0xffc waiting on condition [0x00007f5db3bfa000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b5626338> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) "qtp2054301670-138 Selector0" prio=10 tid=0x00007f5e3cccd800 nid=0xffb runnable [0x00007f5db3cfb000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x00000000b5628fe0> (a sun.nio.ch.Util$2) - locked <0x00000000b5628fc8> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000b55fb490> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) "HashSessionScavenger-2" daemon prio=10 tid=0x00007f5e3cc5a800 nid=0xff7 in Object.wait() [0x00007f5e08156000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000b5621138> (a java.util.TaskQueue) at java.util.TimerThread.mainLoop(Timer.java:552) - locked <0x00000000b5621138> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:505) "HashSessionScavenger-1" daemon prio=10 tid=0x00007f5e3cc63000 nid=0xff6 in Object.wait() [0x00007f5e08257000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000b5621168> (a java.util.TaskQueue) at java.util.TimerThread.mainLoop(Timer.java:552) - locked <0x00000000b5621168> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:505) "HashSessionScavenger-0" daemon prio=10 tid=0x00007f5e3cbac000 nid=0xff5 in Object.wait() [0x00007f5e08358000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000b5621198> (a java.util.TaskQueue) at java.util.TimerThread.mainLoop(Timer.java:552) - locked <0x00000000b5621198> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:505) "qtp819231520-130" prio=10 tid=0x00007f5e3cb2a800 nid=0xff1 waiting on condition [0x00007f5e08459000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b563b640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) "qtp819231520-129" prio=10 tid=0x00007f5e3cb28800 nid=0xff0 waiting on condition [0x00007f5e0855a000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b563b640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) "qtp819231520-128" prio=10 tid=0x00007f5e3cb26800 nid=0xfef waiting on condition [0x00007f5e0865b000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b563b640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) "qtp819231520-127" prio=10 tid=0x00007f5e3cb24800 nid=0xfee waiting on condition [0x00007f5e088f3000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b563b640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) "qtp819231520-126" prio=10 tid=0x00007f5e3cb22000 nid=0xfed waiting on condition [0x00007f5e089f4000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b563b640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) "qtp819231520-125 Acceptor0 SelectChannelConnector@0.0.0.0:61614" prio=10 tid=0x00007f5e3cb20800 nid=0xfec runnable [0x00007f5e08af5000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241) - locked <0x00000000b563cdb0> (a java.lang.Object) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) "qtp819231520-124" prio=10 tid=0x00007f5e3cb1d000 nid=0xfeb waiting on condition [0x00007f5e08bf6000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b563b640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) "qtp819231520-123 Selector0" prio=10 tid=0x00007f5e3cb1c800 nid=0xfea runnable [0x00007f5e08cf7000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x00000000b563d6e0> (a sun.nio.ch.Util$2) - locked <0x00000000b563d4e8> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000b5621958> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) "ActiveMQ Transport Server: mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=1048576000" daemon prio=10 tid=0x00007f5e3c64c000 nid=0xfe9 runnable [0x00007f5e0900d000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:530) at java.net.ServerSocket.accept(ServerSocket.java:498) at org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:350) at java.lang.Thread.run(Thread.java:745) "ActiveMQ Transport Server Thread Handler: mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=1048576000" daemon prio=10 tid=0x00007f5e3c754000 nid=0xfe8 waiting on condition [0x00007f5e0910e000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b5651560> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.activemq.transport.tcp.TcpTransportServer$2.run(TcpTransportServer.java:425) at java.lang.Thread.run(Thread.java:745) "ActiveMQ Transport Server: stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=1048576000" daemon prio=10 tid=0x00007f5e3c745800 nid=0xfe7 runnable [0x00007f5e0920f000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:530) at java.net.ServerSocket.accept(ServerSocket.java:498) at org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:350) at java.lang.Thread.run(Thread.java:745) "ActiveMQ Transport Server Thread Handler: stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=1048576000" daemon prio=10 tid=0x00007f5e3c231800 nid=0xfe6 runnable [0x00007f5e09310000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.createEntry(HashMap.java:897) at java.util.HashMap.addEntry(HashMap.java:884) at java.util.HashMap.put(HashMap.java:505) at org.apache.activemq.transport.tcp.TcpTransportServer.handleSocket(TcpTransportServer.java:488) at org.apache.activemq.transport.tcp.TcpTransportServer$2.run(TcpTransportServer.java:428) at java.lang.Thread.run(Thread.java:745) "ActiveMQ Transport Server: amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=1048576000" daemon prio=10 tid=0x00007f5e3c230000 nid=0xfe5 runnable [0x00007f5e09411000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:530) at java.net.ServerSocket.accept(ServerSocket.java:498) at org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:350) at java.lang.Thread.run(Thread.java:745) "ActiveMQ Transport Server Thread Handler: amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=1048576000" daemon prio=10 tid=0x00007f5e3c76b000 nid=0xfe4 waiting on condition [0x00007f5e09512000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b5657bc0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.activemq.transport.tcp.TcpTransportServer$2.run(TcpTransportServer.java:425) at java.lang.Thread.run(Thread.java:745) "hawtdispatch timer" daemon prio=10 tid=0x00007f5dc80da800 nid=0x3ac in Object.wait() [0x00007f5e28382000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000b521a678> (a java.lang.Object) at java.lang.Object.wait(Object.java:503) at org.fusesource.hawtdispatch.internal.TimerThread.run(TimerThread.java:186) - locked <0x00000000b521a678> (a java.lang.Object) "hawtdispatch-DEFAULT-4" daemon prio=10 tid=0x00007f5dc8042800 nid=0x3ab runnable [0x00007f5e28483000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x00000000b5206048> (a sun.nio.ch.Util$2) - locked <0x00000000b5206030> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000b5205ef8> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) at org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52) at org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182) at org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130) at org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75) "hawtdispatch-DEFAULT-3" daemon prio=10 tid=0x00007f5dc8041800 nid=0x3aa runnable [0x00007f5e28584000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x00000000b51f58f0> (a sun.nio.ch.Util$2) - locked <0x00000000b51f58d8> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000b51f57a0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) at org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52) at org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182) at org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130) at org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75) "hawtdispatch-DEFAULT-2" daemon prio=10 tid=0x00007f5dc8049800 nid=0x3a9 runnable [0x00007f5e28685000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x00000000b51e5198> (a sun.nio.ch.Util$2) - locked <0x00000000b51e5180> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000b51e5048> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) at org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52) at org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182) at org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130) at org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75) "hawtdispatch-DEFAULT-1" daemon prio=10 tid=0x00007f5dc803f000 nid=0x3a8 runnable [0x00007f5e28786000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x00000000b51d2aa8> (a sun.nio.ch.Util$2) - locked <0x00000000b51d2a90> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000b51d2958> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) at org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52) at org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182) at org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130) at org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75) "ZooKeeper state change dispatcher thread" daemon prio=10 tid=0x00007f5e3ca77800 nid=0x3a5 waiting on condition [0x00007f5e29190000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b4fb04b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.activemq.leveldb.replicated.groups.ZKClient$StateChangeDispatcher.run(ZKClient.java:347) "RMI RenewClean-[172.30.0.21:57440]" daemon prio=10 tid=0x00007f5dd4003800 nid=0x3a3 in Object.wait() [0x00007f5e28b8a000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000b55a2858> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) - locked <0x00000000b55a2858> (a java.lang.ref.ReferenceQueue$Lock) at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:535) at java.lang.Thread.run(Thread.java:745) "RMI Scheduler(0)" daemon prio=10 tid=0x00007f5de4018000 nid=0x3a2 waiting on condition [0x00007f5e28c8b000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b4d5bc38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) "GC Daemon" daemon prio=10 tid=0x00007f5de4004800 nid=0x3a0 in Object.wait() [0x00007f5e28e8d000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000b4f9fdc8> (a sun.misc.GC$LatencyLock) at sun.misc.GC$Daemon.run(GC.java:117) - locked <0x00000000b4f9fdc8> (a sun.misc.GC$LatencyLock) "RMI Reaper" prio=10 tid=0x00007f5de4002800 nid=0x39f in Object.wait() [0x00007f5e28f8e000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000b4d5adc8> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) - locked <0x00000000b4d5adc8> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:351) at java.lang.Thread.run(Thread.java:745) "RMI TCP Accept-0" daemon prio=10 tid=0x00007f5de4001000 nid=0x39e runnable [0x00007f5e2908f000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:530) at java.net.ServerSocket.accept(ServerSocket.java:498) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360) at java.lang.Thread.run(Thread.java:745) "RMI TCP Accept-1099" daemon prio=10 tid=0x00007f5e3ca22800 nid=0x39c runnable [0x00007f5e29291000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:530) at java.net.ServerSocket.accept(ServerSocket.java:498) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360) at java.lang.Thread.run(Thread.java:745) "RMI TCP Accept-0" daemon prio=10 tid=0x00007f5e3c29e800 nid=0x39a runnable [0x00007f5e296ce000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:530) at java.net.ServerSocket.accept(ServerSocket.java:498) at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360) at java.lang.Thread.run(Thread.java:745) "Service Thread" daemon prio=10 tid=0x00007f5e3c1ee000 nid=0x399 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread1" daemon prio=10 tid=0x00007f5e3c1eb800 nid=0x398 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread0" daemon prio=10 tid=0x00007f5e3c1e9800 nid=0x397 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" daemon prio=10 tid=0x00007f5e3c1e7000 nid=0x396 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Surrogate Locker Thread (Concurrent GC)" daemon prio=10 tid=0x00007f5e3c1e5000 nid=0x395 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Finalizer" daemon prio=10 tid=0x00007f5e3c1bc000 nid=0x394 in Object.wait() [0x00007f5e386c2000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000b4cc4740> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) - locked <0x00000000b4cc4740> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) "Reference Handler" daemon prio=10 tid=0x00007f5e3c1ba000 nid=0x393 in Object.wait() [0x00007f5e387c3000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000b4cc4358> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) - locked <0x00000000b4cc4358> (a java.lang.ref.Reference$Lock) "main" prio=10 tid=0x00007f5e3c009800 nid=0x38c waiting on condition [0x00007f5e4339d000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b5669f08> (a java.util.concurrent.CountDownLatch$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236) at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:118) at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57) at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:150) at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57) at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.activemq.console.Main.runTaskClass(Main.java:262) at org.apache.activemq.console.Main.main(Main.java:115) "VM Thread" prio=10 tid=0x00007f5e3c1b6000 nid=0x392 runnable "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f5e3c01b000 nid=0x38d runnable "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f5e3c01c800 nid=0x38e runnable "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f5e3c01e800 nid=0x38f runnable "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f5e3c020800 nid=0x390 runnable "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007f5e3c0a1000 nid=0x391 runnable "VM Periodic Task Thread" prio=10 tid=0x00007f5e3c2ae800 nid=0x39b waiting on condition JNI global references: 171 -- This message was sent by Atlassian JIRA (v6.3.4#6332)