Hi,
please find attached the thread dump.
Regards,
Anna
-----Original Message-----
From: Ganesh Murthy [mailto:gmur...@redhat.com]
Sent: Dienstag, 29. September 2015 16:52
To: users@activemq.apache.org
Subject: Re: Problem shutting down activemq client with failover transport
Can you please provide a thread dump? That would throw some light on exactly
what is going on.
Thanks.
----- Original Message -----
From: "Anna Maier" <a.ma...@topdesk.com>
To: users@activemq.apache.org
Sent: Tuesday, September 29, 2015 9:31:13 AM
Subject: Problem shutting down activemq client with failover transport
Hi,
I am using Camel to connect to ActiveMQ and it fails to shutdown when the
ActiveMQComponent cannot make a connection to the broker with the failover
protocol (activemq version 5.12.0).
The problem seems to be that the failover transport tries to connect to the
broker in an endless loop on its own thread that cannot be interrupted. I think
the problem is in the PooledTaskRunner: there is a shutdown field which is set
to true when camel shuts down, but it appears as false to the thread that is
running the endless loop.
I am currently at a loss what to do. I want to keep the maximum retries setting
for the failover transport on indefinite, since I do not know when the the
broker is back online. At the same time, it has to be possible to shut down the
application even if there never was a connection to the ActiveMQ broker.
Has anybody seen this problem before and can give some advice? Is this a bug in
ActiveMQ?
Attached there is some code that reproduces the problem - it never terminates.
Regards,
Anna
The code:
public class ShutdownBug {
public static void main(String[] args) throws Exception {
DefaultCamelContext camelContext = new DefaultCamelContext();
ActiveMQComponent activeMQComponent = new ActiveMQComponent();
activeMQComponent.setBrokerURL("failover:(http://localhost:61616)");
camelContext.addComponent("activemq", activeMQComponent);
camelContext.addRoutes(new RouteBuilder() {
@Override
public void configure() {
from("activemq:queue:myqueue").to("stream:out");
}
});
Thread startThread = new Thread(new Runnable() {
@Override
public void run() {
try {
camelContext.start();
}
catch (Exception e) {
e.printStackTrace();
}
}
});
startThread.start();
startThread.interrupt();
startThread.join();
camelContext.stop();
}
}

2015-09-30 07:53:25
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.51-b03 mixed mode):
"RMI TCP Connection(2)-10.2.3.76" #21 daemon prio=5 os_prio=0
tid=0x000000001faac800 nid=0x18f0 runnable [0x000000002243e000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
- locked <0x000000076b594630> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:550)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$255(TCPTransport.java:683)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$2/934334256.run(Unknown
Source)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x000000076b309018> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"JMX server connection timeout 20" #20 daemon prio=5 os_prio=0
tid=0x00000000205bb800 nid=0x18d4 in Object.wait() [0x000000002225f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000076b38dd98> (a [I)
at
com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:168)
- locked <0x000000076b38dd98> (a [I)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"RMI Scheduler(0)" #19 daemon prio=5 os_prio=0 tid=0x00000000206ab800
nid=0x18cc waiting on condition [0x000000002210f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076b1ea6d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"RMI TCP Connection(1)-10.2.3.76" #18 daemon prio=5 os_prio=0
tid=0x00000000205ba800 nid=0x18c8 runnable [0x000000001f57e000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
- locked <0x000000076b35ac28> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:550)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$255(TCPTransport.java:683)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$2/934334256.run(Unknown
Source)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x000000076b306d08> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"RMI TCP Accept-0" #17 daemon prio=5 os_prio=0 tid=0x00000000205b8000
nid=0x18b8 runnable [0x0000000021eff000]
java.lang.Thread.State: RUNNABLE
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at
java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
- locked <0x000000076b1eb978> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at
sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Camel (camel-1) thread #1 - ShutdownTask" #16 daemon prio=5 os_prio=0
tid=0x0000000020581000 nid=0x630 in Object.wait() [0x0000000012d8e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000076ee08178> (a java.lang.Object)
at java.lang.Object.wait(Object.java:502)
at
org.springframework.jms.listener.DefaultMessageListenerContainer.doShutdown(DefaultMessageListenerContainer.java:566)
- locked <0x000000076ee08178> (a java.lang.Object)
at
org.springframework.jms.listener.AbstractJmsListeningContainer.shutdown(AbstractJmsListeningContainer.java:237)
at
org.springframework.jms.listener.AbstractJmsListeningContainer.destroy(AbstractJmsListeningContainer.java:177)
at
org.apache.camel.component.jms.DefaultJmsMessageListenerContainer.destroy(DefaultJmsMessageListenerContainer.java:133)
at
org.apache.camel.component.jms.JmsConsumer.stopAndDestroyListenerContainer(JmsConsumer.java:179)
at
org.apache.camel.component.jms.JmsConsumer.doStop(JmsConsumer.java:217)
at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102)
at
org.apache.camel.util.ServiceHelper.stopService(ServiceHelper.java:141)
at
org.apache.camel.impl.DefaultShutdownStrategy.shutdownNow(DefaultShutdownStrategy.java:368)
at
org.apache.camel.impl.DefaultShutdownStrategy$ShutdownTask.run(DefaultShutdownStrategy.java:646)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x000000076ee082b8> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"ActiveMQ InactivityMonitor ReadCheckTimer" #15 daemon prio=5 os_prio=0
tid=0x00000000203f9000 nid=0x868 in Object.wait() [0x000000002199f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000076ee00d30> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x000000076ee00d30> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"Camel (camel-1) thread #0 - JmsConsumer[myqueue]" #14 daemon prio=5 os_prio=0
tid=0x000000001fe2c800 nid=0x530 waiting for monitor entry [0x00000000217fe000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000076ee10178> (a java.lang.Object)
at
org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:623)
- locked <0x000000076ee10178> (a java.lang.Object)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at
org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1387)
at
org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1501)
- locked <0x000000076ee2aa30> (a java.lang.Object)
at
org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:323)
at
org.apache.activemq.jms.pool.ConnectionPool.makeSession(ConnectionPool.java:108)
at
org.apache.activemq.jms.pool.ConnectionPool$1.makeObject(ConnectionPool.java:78)
at
org.apache.activemq.jms.pool.ConnectionPool$1.makeObject(ConnectionPool.java:74)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1036)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:278)
at
org.apache.activemq.jms.pool.ConnectionPool.createSession(ConnectionPool.java:137)
at
org.apache.activemq.jms.pool.PooledConnection.createSession(PooledConnection.java:167)
at
org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:192)
at
org.springframework.jms.listener.DefaultMessageListenerContainer.access$1400(DefaultMessageListenerContainer.java:124)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.initResourcesIfNecessary(DefaultMessageListenerContainer.java:1164)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1143)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1136)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1033)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x000000076ee103b8> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
- <0x000000076ee2acd0> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
"ActiveMQ Task-1" #13 daemon prio=5 os_prio=0 tid=0x000000001fd3d800 nid=0x454
runnable [0x000000001ff9e000]
java.lang.Thread.State: RUNNABLE
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at
java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
- locked <0x000000076b9414f0> (a java.net.DualStackPlainSocketImpl)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at
org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
at
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:643)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:1138)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:1076)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:1050)
at
org.apache.activemq.transport.http.HttpClientTransport.doStart(HttpClientTransport.java:278)
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
at
org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:168)
at
org.apache.activemq.transport.InactivityMonitor.start(InactivityMonitor.java:50)
at
org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1030)
- locked <0x000000076ee10178> (a java.lang.Object)
at
org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:149)
- locked <0x000000076ee2af80> (a java.lang.Object)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x000000076ee2b248> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"Service Thread" #10 daemon prio=9 os_prio=0 tid=0x000000001d943800 nid=0xbb4
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread3" #9 daemon prio=9 os_prio=2 tid=0x000000001ca1f800
nid=0x1d4 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread2" #8 daemon prio=9 os_prio=2 tid=0x000000001ca1e800
nid=0x9c4 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread1" #7 daemon prio=9 os_prio=2 tid=0x000000001ca1d000
nid=0x8f0 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread0" #6 daemon prio=9 os_prio=2 tid=0x000000001d8ab000
nid=0x598 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x000000001d8a7800 nid=0x1068
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x000000001d8a6000 nid=0x3f0
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x000000001ca0e000 nid=0xa24 in
Object.wait() [0x000000001ee2f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000076ee20bc8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x000000076ee20bc8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
Locked ownable synchronizers:
- None
"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x000000001ca07800
nid=0x1428 in Object.wait() [0x000000001ec0f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000076ee20d80> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
- locked <0x000000076ee20d80> (a java.lang.ref.Reference$Lock)
Locked ownable synchronizers:
- None
"main" #1 prio=5 os_prio=0 tid=0x00000000021ef000 nid=0x508 waiting on
condition [0x00000000024be000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076ee20da0> (a
org.apache.camel.util.concurrent.RejectableFutureTask)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:426)
at java.util.concurrent.FutureTask.get(FutureTask.java:204)
at
org.apache.camel.impl.DefaultShutdownStrategy.doShutdown(DefaultShutdownStrategy.java:190)
at
org.apache.camel.impl.DefaultShutdownStrategy.shutdownForced(DefaultShutdownStrategy.java:142)
at
org.apache.camel.impl.DefaultCamelContext.doStop(DefaultCamelContext.java:2666)
- locked <0x000000076ee20fc8> (a
org.apache.camel.impl.DefaultCamelContext)
at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102)
at com.bug.ShutdownBug.main(ShutdownBug.java:34)
Locked ownable synchronizers:
- None
"VM Thread" os_prio=2 tid=0x000000001d852800 nid=0xf28 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00000000020ef800 nid=0xe3c
runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00000000020f1000 nid=0x1664
runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00000000020f2800 nid=0x1498
runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00000000020f4000 nid=0x1274
runnable
"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00000000020f7800 nid=0x62c
runnable
"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00000000020f8800 nid=0x8ec
runnable
"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00000000020fb800 nid=0xdbc
runnable
"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00000000020fd000 nid=0x26c
runnable
"VM Periodic Task Thread" os_prio=2 tid=0x000000001d91a000 nid=0x1fc waiting on
condition
JNI global references: 255
