[
https://issues.apache.org/jira/browse/AMQ-5510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhi updated AMQ-5510:
----------------------
Description:
I have multiple listeners sharing same connection and the broker url has
maxReconnectAttempts property set to 5.
If I restart the broker, the ActiveMQ layer tries 5 reconnect attempts then
gives up and the listener throws up connection refused exception and then tries
to refresh the jms connection and says its refreshed but there is not
connection made to ActiveMQ and the listener again throws up connection refused
exception and again refreshes the jms connection but fails. This process
continues indefinitely.
However, I use single listener and no sharing of connection, this works fine.
The ActiveMQ layers tries 5 reconnect attempts fails, listener throws up
exception and again triggers reconnect attempts which does not happens in case
of multiple listener.
This issue doesn't happen with ActiveMQ 5.8.0
Steps to reproduce this issue:
<bean id="connectionFactory"
class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL"
value="failover:(tcp://<hostname1>:61616?wireFormat.maxInactivityDuration=120000,tcp://<hostname2>:61616?wireFormat.maxInactivityDuration=120000)?randomize=false&maxReconnectAttempts=5"
/>
</bean>
<bean id="jmsConnectionFactory"
class="org.apache.activemq.pool.PooledConnectionFactory"
destroy-method="stop">
<property name="connectionFactory" ref="connectionFactory" />
</bean>
<amq:topic id="testTopic" physicalName="test.topic"/>
<!-- The message listener container to invoke the consumer -->
<bean id="jmsContainer1"
class="org.springframework.jms.listener.DefaultMessageListenerContainer"
p:connectionFactory-ref="jmsConnectionFactory"
p:destination-ref="testTopic"
p:pubSubDomain="true"
p:subscriptionDurable="false"
p:sessionTransacted="true"
p:messageListener-ref="consumer" />
<bean id="jmsContainer2"
class="org.springframework.jms.listener.DefaultMessageListenerContainer"
p:connectionFactory-ref="jmsConnectionFactory"
p:destination-ref="testTopic"
p:pubSubDomain="true"
p:subscriptionDurable="false"
p:sessionTransacted="true"
p:messageListener-ref="consumer" />
Use this configuration and start the application.
Now restart the ActiveMQ broker once the listeners successfully connect.
Here are some logs:
[20141223 01:21:39.277 EST (ActiveMQ Task-3)
org.apache.activemq.transport.failover.FailoverTransport#doReconnect 1089
ERROR] - Failed to connect to [tcp://a
rgon47.nyc:61616?wireFormat.maxInactivityDuration=120000,
tcp://argon36.nyc:61616?wireFormat.maxInactivityDuration=120000] after: 5
attempt(s)
[20141223 01:21:39.277 EST (ActiveMQ Task-3)
org.apache.activemq.ActiveMQConnection#onAsyncException 2010 DEBUG] - Async
exception with no exception listener
: java.net.ConnectException: Connection refused
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at
org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:501)
at
org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:464)
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
at
org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:138)
at
org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
at
org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
at
org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1017)
at
org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:148)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[20141223 01:21:39.279 EST (ActiveMQ Connection Executor: unconnected)
org.apache.activemq.transport.failover.FailoverTransport#stop 366 DEBUG] -
Stopped unc
onnected
[20141223 01:21:39.279 EST (ActiveMQ Connection Executor: unconnected)
org.apache.activemq.util.ThreadPoolUtils#shutdownNow 54 DEBUG] - Forcing
shutdown of E
xecutorService: java.util.concurrent.ThreadPoolExecutor@31736876[Running, pool
size = 2, active threads = 0, queued tasks = 0, completed tasks = 5]
[20141223 01:21:39.281 EST (ActiveMQ Connection Executor: unconnected)
org.apache.activemq.pool.PooledConnectionFactory$1$1#onException 110 INFO] -
Expiring connection ActiveMQConnection
{id=ID:cobalt14.nyc.deshaw.com-49500-1419315471791-1:1,clientId=ID:cobalt14.nyc.deshaw.com-49500-1419315471791-0:1,started=false}
on IOException: java.net.ConnectException: Connection refused
[20141223 01:21:39.281 EST (ActiveMQ Connection Executor: unconnected)
org.apache.activemq.pool.PooledConnectionFactory$1$1#onException 111 DEBUG] -
Expiring connection on IOException
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at
org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:501)
at
org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:464)
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
at
org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:138)
at
org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
at
org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
at
org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1017)
at
org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:148)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[20141223 01:21:39.282 EST (jmsContainer2-1)
org.apache.activemq.jms.pool.PooledSession#close 119 WARN] - Caught exception
trying rollback() when putting session back into the pool, will invalidate.
javax.jms.IllegalStateException: The Session is closed
javax.jms.IllegalStateException: The Session is closed
at
org.apache.activemq.ActiveMQSession.checkClosed(ActiveMQSession.java:767)
at
org.apache.activemq.ActiveMQSession.rollback(ActiveMQSession.java:588)
at
org.apache.activemq.jms.pool.PooledSession.close(PooledSession.java:116)
at
org.springframework.jms.support.JmsUtils.closeSession(JmsUtils.java:108)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.clearResources(DefaultMessageListenerContainer.java:1143)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1002)
at java.lang.Thread.run(Thread.java:745)
[20141223 01:21:39.282 EST (jmsContainer1-1)
org.apache.activemq.jms.pool.PooledSession#close 119 WARN] - Caught exception
trying rollback() when putting session back into the pool, will invalidate.
javax.jms.IllegalStateException: The Session is closed
javax.jms.IllegalStateException: The Session is closed
at
org.apache.activemq.ActiveMQSession.checkClosed(ActiveMQSession.java:767)
at
org.apache.activemq.ActiveMQSession.rollback(ActiveMQSession.java:588)
at
org.apache.activemq.jms.pool.PooledSession.close(PooledSession.java:116)
at
org.springframework.jms.support.JmsUtils.closeSession(JmsUtils.java:108)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.clearResources(DefaultMessageListenerContainer.java:1143)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1002)
at java.lang.Thread.run(Thread.java:745)
[20141223 01:21:39.283 EST (jmsContainer2-1)
org.springframework.jms.listener.DefaultMessageListenerContainer#handleListenerSetupFailure
839 WARN] - Setup of JMS message listener invoker failed for destination
'topic://cometd.seedTCAStatusPush' - trying to recover. Cause: Connection
refused
javax.jms.JMSException: Connection refused
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)
at
org.apache.activemq.ActiveMQMessageConsumer.dequeue(ActiveMQMessageConsumer.java:484)
at
org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:631)
at
org.apache.activemq.jms.pool.PooledMessageConsumer.receive(PooledMessageConsumer.java:67)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:430)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:310)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1102)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1094)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:991)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at
org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:501)
at
org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:464)
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
at
org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:138)
at
org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
at
org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
at
org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1017)
at
org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:148)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
[20141223 01:21:39.283 EST (jmsContainer1-1)
org.springframework.jms.listener.DefaultMessageListenerContainer#handleListenerSetupFailure
839 WARN] - Setup of JMS message listener invoker failed for destination
'topic://cometd.seedTCAStatusPush' - trying to recover. Cause: Connection
refused
javax.jms.JMSException: Connection refused
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)
at
org.apache.activemq.ActiveMQMessageConsumer.dequeue(ActiveMQMessageConsumer.java:484)
at
org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:631)
at
org.apache.activemq.jms.pool.PooledMessageConsumer.receive(PooledMessageConsumer.java:67)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:430)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:310)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1102)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1094)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:991)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at
org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:501)
at
org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:464)
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
at
org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:138)
at
org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
at
org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
at
org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1017)
at
org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:148)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
[20141223 01:21:39.284 EST (jmsContainer2-1)
org.springframework.jms.listener.DefaultMessageListenerContainer#refreshConnectionUntilSuccessful
891 INFO] - Successfully refreshed JMS Connection
[20141223 01:21:39.284 EST (jmsContainer1-1)
org.springframework.jms.listener.DefaultMessageListenerContainer#refreshConnectionUntilSuccessful
891 INFO] - Successfully refreshed JMS Connection
[20141223 01:21:44.286 EST (jmsContainer1-2)
org.springframework.jms.listener.DefaultMessageListenerContainer#handleListenerSetupFailure
839 WARN] - Setup of JMS message listener invoker failed for destination
'topic://cometd.seedTCAStatusPush' - trying to recover. Cause:
org.apache.activemq.ConnectionFailedException: The JMS connection has failed:
Connection refused
javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException:
The JMS connection has failed: Connection refused
at
org.apache.activemq.jms.pool.ConnectionPool.createSession(ConnectionPool.java:135)
at
org.apache.activemq.jms.pool.PooledConnection.createSession(PooledConnection.java:167)
at
org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:197)
at
org.springframework.jms.listener.DefaultMessageListenerContainer.access$1400(DefaultMessageListenerContainer.java:119)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.initResourcesIfNecessary(DefaultMessageListenerContainer.java:1122)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1101)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1094)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:991)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.activemq.ConnectionFailedException: The JMS connection
has failed: Connection refused
at
org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1492)
at
org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:327)
at
org.apache.activemq.jms.pool.ConnectionPool.makeSession(ConnectionPool.java:104)
at
org.apache.activemq.jms.pool.ConnectionPool$1.makeObject(ConnectionPool.java:81)
at
org.apache.activemq.jms.pool.ConnectionPool$1.makeObject(ConnectionPool.java:66)
at
org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
at
org.apache.activemq.jms.pool.ConnectionPool.createSession(ConnectionPool.java:133)
... 8 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at
org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:501)
at
org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:464)
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
at
org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:138)
at
org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
at
org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
at
org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1017)
at
org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:148)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
[20141223 01:21:44.286 EST (jmsContainer2-2)
org.springframework.jms.listener.DefaultMessageListenerContainer#handleListenerSetupFailure
839 WARN] - Setup of JMS message listener invoker failed for destination
'topic://cometd.seedTCAStatusPush' - trying to recover. Cause:
org.apache.activemq.ConnectionFailedException: The JMS connection has failed:
Connection refused
javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException:
The JMS connection has failed: Connection refused
at
org.apache.activemq.jms.pool.ConnectionPool.createSession(ConnectionPool.java:135)
at
org.apache.activemq.jms.pool.PooledConnection.createSession(PooledConnection.java:167)
at
org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:197)
at
org.springframework.jms.listener.DefaultMessageListenerContainer.access$1400(DefaultMessageListenerContainer.java:119)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.initResourcesIfNecessary(DefaultMessageListenerContainer.java:1122)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1101)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1094)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:991)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.activemq.ConnectionFailedException: The JMS connection
has failed: Connection refused
at
org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1492)
at
org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:327)
at
org.apache.activemq.jms.pool.ConnectionPool.makeSession(ConnectionPool.java:104)
at
org.apache.activemq.jms.pool.ConnectionPool$1.makeObject(ConnectionPool.java:81)
at
org.apache.activemq.jms.pool.ConnectionPool$1.makeObject(ConnectionPool.java:66)
at
org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
at
org.apache.activemq.jms.pool.ConnectionPool.createSession(ConnectionPool.java:133)
... 8 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at
org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:501)
at
org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:464)
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
at
org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:138)
at
org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
at
org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
at
org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1017)
at
org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:148)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
[20141223 01:21:44.288 EST (jmsContainer1-2)
org.springframework.jms.listener.DefaultMessageListenerContainer#refreshConnectionUntilSuccessful
891 INFO] - Successfully refreshed JMS Connection
[20141223 01:21:44.289 EST (jmsContainer2-2)
org.springframework.jms.listener.DefaultMessageListenerContainer#refreshConnectionUntilSuccessful
891 INFO] - Successfully refreshed JMS Connection
[20141223 01:21:49.291 EST (jmsContainer2-3)
org.springframework.jms.listener.DefaultMessageListenerContainer#handleListenerSetupFailure
839 WARN] - Setup of JMS message listener invoker failed for destination
'topic://cometd.seedTCAStatusPush' - trying to recover. Cause:
org.apache.activemq.ConnectionFailedException: The JMS connection has failed:
Connection refused
javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException:
The JMS connection has failed: Connection refused
at
org.apache.activemq.jms.pool.ConnectionPool.createSession(ConnectionPool.java:135)
at
org.apache.activemq.jms.pool.PooledConnection.createSession(PooledConnection.java:167)
at
org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:197)
at
org.springframework.jms.listener.DefaultMessageListenerContainer.access$1400(DefaultMessageListenerContainer.java:119)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.initResourcesIfNecessary(DefaultMessageListenerContainer.java:1122)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1101)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1094)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:991)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.activemq.ConnectionFailedException: The JMS connection
has failed: Connection refused
at
org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1492)
at
org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:327)
at
org.apache.activemq.jms.pool.ConnectionPool.makeSession(ConnectionPool.java:104)
at
org.apache.activemq.jms.pool.ConnectionPool$1.makeObject(ConnectionPool.java:81)
at
org.apache.activemq.jms.pool.ConnectionPool$1.makeObject(ConnectionPool.java:66)
at
org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
at
org.apache.activemq.jms.pool.ConnectionPool.createSession(ConnectionPool.java:133)
... 8 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at
org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:501)
at
org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:464)
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
at
org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:138)
at
org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
at
org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
at
org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1017)
at
org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:148)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
This continues indefinitely. I checked lsof output at this point and there was
no connection to ActiveMQ broker, so these connection attempts were not
reaching the broker itself.
was:
I have multiple listeners sharing same connection and the broker url has
maxReconnectAttempts property set to 5.
If I restart the broker, the ActiveMQ layer tries 5 reconnect attempts then
gives up and the listener throws up connection refused exception and then tries
to refresh the jms connection and says its refreshed but there is not
connection made to ActiveMQ and the listener again throws up connection refused
exception and again refreshes the jms connection but fails. This process
continues indefinitely.
However, I use single listener and no sharing of connection, this works fine.
The ActiveMQ layers tries 5 reconnect attempts fails, listener throws up
exception and again triggers reconnect attempts which does not happens in case
of multiple listener.
This issue doesn't happen with ActiveMQ 5.8.0
Steps to reproduce this issue:
<bean id="connectionFactory"
class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL"
value="failover:(tcp://<hostname1>:61616?wireFormat.maxInactivityDuration=120000,tcp://<hostname2>:61616?wireFormat.maxInactivityDuration=120000)?randomize=false&maxReconnectAttempts=5"
/>
</bean>
<bean id="jmsConnectionFactory"
class="org.apache.activemq.pool.PooledConnectionFactory"
destroy-method="stop">
<property name="connectionFactory" ref="connectionFactory" />
</bean>
<amq:topic id="testTopic" physicalName="test.topic"/>
<!-- The message listener container to invoke the consumer -->
<bean id="jmsContainer1"
class="org.springframework.jms.listener.DefaultMessageListenerContainer"
p:connectionFactory-ref="jmsConnectionFactory"
p:destination-ref="testTopic"
p:pubSubDomain="true"
p:subscriptionDurable="false"
p:sessionTransacted="true"
p:messageListener-ref="consumer" />
<bean id="jmsContainer2"
class="org.springframework.jms.listener.DefaultMessageListenerContainer"
p:connectionFactory-ref="jmsConnectionFactory"
p:destination-ref="testTopic"
p:pubSubDomain="true"
p:subscriptionDurable="false"
p:sessionTransacted="true"
p:messageListener-ref="consumer" />
Use this configuration and start the application.
Now restart the ActiveMQ broker once the listeners successfully connect.
I am attaching some debug logs for reference.
> Listener is not able to connect again once maxReconnectAttempts are exhausted
> by the ActiveMQ layers.
> -----------------------------------------------------------------------------------------------------
>
> Key: AMQ-5510
> URL: https://issues.apache.org/jira/browse/AMQ-5510
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.10.0
> Environment: ActiveMQ 5.10
> Reporter: Abhi
>
> I have multiple listeners sharing same connection and the broker url has
> maxReconnectAttempts property set to 5.
> If I restart the broker, the ActiveMQ layer tries 5 reconnect attempts then
> gives up and the listener throws up connection refused exception and then
> tries to refresh the jms connection and says its refreshed but there is not
> connection made to ActiveMQ and the listener again throws up connection
> refused exception and again refreshes the jms connection but fails. This
> process continues indefinitely.
> However, I use single listener and no sharing of connection, this works fine.
> The ActiveMQ layers tries 5 reconnect attempts fails, listener throws up
> exception and again triggers reconnect attempts which does not happens in
> case of multiple listener.
> This issue doesn't happen with ActiveMQ 5.8.0
> Steps to reproduce this issue:
> <bean id="connectionFactory"
> class="org.apache.activemq.ActiveMQConnectionFactory">
> <property name="brokerURL"
> value="failover:(tcp://<hostname1>:61616?wireFormat.maxInactivityDuration=120000,tcp://<hostname2>:61616?wireFormat.maxInactivityDuration=120000)?randomize=false&maxReconnectAttempts=5"
> />
> </bean>
> <bean id="jmsConnectionFactory"
> class="org.apache.activemq.pool.PooledConnectionFactory"
> destroy-method="stop">
> <property name="connectionFactory" ref="connectionFactory" />
> </bean>
> <amq:topic id="testTopic" physicalName="test.topic"/>
> <!-- The message listener container to invoke the consumer -->
> <bean id="jmsContainer1"
>
> class="org.springframework.jms.listener.DefaultMessageListenerContainer"
> p:connectionFactory-ref="jmsConnectionFactory"
> p:destination-ref="testTopic"
> p:pubSubDomain="true"
> p:subscriptionDurable="false"
> p:sessionTransacted="true"
> p:messageListener-ref="consumer" />
> <bean id="jmsContainer2"
>
> class="org.springframework.jms.listener.DefaultMessageListenerContainer"
> p:connectionFactory-ref="jmsConnectionFactory"
> p:destination-ref="testTopic"
> p:pubSubDomain="true"
> p:subscriptionDurable="false"
> p:sessionTransacted="true"
> p:messageListener-ref="consumer" />
> Use this configuration and start the application.
> Now restart the ActiveMQ broker once the listeners successfully connect.
> Here are some logs:
> [20141223 01:21:39.277 EST (ActiveMQ Task-3)
> org.apache.activemq.transport.failover.FailoverTransport#doReconnect 1089
> ERROR] - Failed to connect to [tcp://a
> rgon47.nyc:61616?wireFormat.maxInactivityDuration=120000,
> tcp://argon36.nyc:61616?wireFormat.maxInactivityDuration=120000] after: 5
> attempt(s)
> [20141223 01:21:39.277 EST (ActiveMQ Task-3)
> org.apache.activemq.ActiveMQConnection#onAsyncException 2010 DEBUG] - Async
> exception with no exception listener
> : java.net.ConnectException: Connection refused
> java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:589)
> at
> org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:501)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:464)
> at
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:138)
> at
> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
> at
> org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
> at
> org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1017)
> at
> org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:148)
> at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
> at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> [20141223 01:21:39.279 EST (ActiveMQ Connection Executor: unconnected)
> org.apache.activemq.transport.failover.FailoverTransport#stop 366 DEBUG] -
> Stopped unc
> onnected
> [20141223 01:21:39.279 EST (ActiveMQ Connection Executor: unconnected)
> org.apache.activemq.util.ThreadPoolUtils#shutdownNow 54 DEBUG] - Forcing
> shutdown of E
> xecutorService: java.util.concurrent.ThreadPoolExecutor@31736876[Running,
> pool size = 2, active threads = 0, queued tasks = 0, completed tasks = 5]
> [20141223 01:21:39.281 EST (ActiveMQ Connection Executor: unconnected)
> org.apache.activemq.pool.PooledConnectionFactory$1$1#onException 110 INFO] -
> Expiring connection ActiveMQConnection
> {id=ID:cobalt14.nyc.deshaw.com-49500-1419315471791-1:1,clientId=ID:cobalt14.nyc.deshaw.com-49500-1419315471791-0:1,started=false}
> on IOException: java.net.ConnectException: Connection refused
> [20141223 01:21:39.281 EST (ActiveMQ Connection Executor: unconnected)
> org.apache.activemq.pool.PooledConnectionFactory$1$1#onException 111 DEBUG] -
> Expiring connection on IOException
> java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:589)
> at
> org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:501)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:464)
> at
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:138)
> at
> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
> at
> org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
> at
> org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1017)
> at
> org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:148)
> at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
> at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> [20141223 01:21:39.282 EST (jmsContainer2-1)
> org.apache.activemq.jms.pool.PooledSession#close 119 WARN] - Caught exception
> trying rollback() when putting session back into the pool, will invalidate.
> javax.jms.IllegalStateException: The Session is closed
> javax.jms.IllegalStateException: The Session is closed
> at
> org.apache.activemq.ActiveMQSession.checkClosed(ActiveMQSession.java:767)
> at
> org.apache.activemq.ActiveMQSession.rollback(ActiveMQSession.java:588)
> at
> org.apache.activemq.jms.pool.PooledSession.close(PooledSession.java:116)
> at
> org.springframework.jms.support.JmsUtils.closeSession(JmsUtils.java:108)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.clearResources(DefaultMessageListenerContainer.java:1143)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1002)
> at java.lang.Thread.run(Thread.java:745)
> [20141223 01:21:39.282 EST (jmsContainer1-1)
> org.apache.activemq.jms.pool.PooledSession#close 119 WARN] - Caught exception
> trying rollback() when putting session back into the pool, will invalidate.
> javax.jms.IllegalStateException: The Session is closed
> javax.jms.IllegalStateException: The Session is closed
> at
> org.apache.activemq.ActiveMQSession.checkClosed(ActiveMQSession.java:767)
> at
> org.apache.activemq.ActiveMQSession.rollback(ActiveMQSession.java:588)
> at
> org.apache.activemq.jms.pool.PooledSession.close(PooledSession.java:116)
> at
> org.springframework.jms.support.JmsUtils.closeSession(JmsUtils.java:108)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.clearResources(DefaultMessageListenerContainer.java:1143)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1002)
> at java.lang.Thread.run(Thread.java:745)
> [20141223 01:21:39.283 EST (jmsContainer2-1)
> org.springframework.jms.listener.DefaultMessageListenerContainer#handleListenerSetupFailure
> 839 WARN] - Setup of JMS message listener invoker failed for destination
> 'topic://cometd.seedTCAStatusPush' - trying to recover. Cause: Connection
> refused
> javax.jms.JMSException: Connection refused
> at
> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)
> at
> org.apache.activemq.ActiveMQMessageConsumer.dequeue(ActiveMQMessageConsumer.java:484)
> at
> org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:631)
> at
> org.apache.activemq.jms.pool.PooledMessageConsumer.receive(PooledMessageConsumer.java:67)
> at
> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:430)
> at
> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:310)
> at
> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1102)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1094)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:991)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:589)
> at
> org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:501)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:464)
> at
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:138)
> at
> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
> at
> org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
> at
> org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1017)
> at
> org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:148)
> at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
> at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ... 1 more
> [20141223 01:21:39.283 EST (jmsContainer1-1)
> org.springframework.jms.listener.DefaultMessageListenerContainer#handleListenerSetupFailure
> 839 WARN] - Setup of JMS message listener invoker failed for destination
> 'topic://cometd.seedTCAStatusPush' - trying to recover. Cause: Connection
> refused
> javax.jms.JMSException: Connection refused
> at
> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)
> at
> org.apache.activemq.ActiveMQMessageConsumer.dequeue(ActiveMQMessageConsumer.java:484)
> at
> org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:631)
> at
> org.apache.activemq.jms.pool.PooledMessageConsumer.receive(PooledMessageConsumer.java:67)
> at
> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:430)
> at
> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:310)
> at
> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1102)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1094)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:991)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:589)
> at
> org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:501)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:464)
> at
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:138)
> at
> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
> at
> org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
> at
> org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1017)
> at
> org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:148)
> at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
> at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ... 1 more
> [20141223 01:21:39.284 EST (jmsContainer2-1)
> org.springframework.jms.listener.DefaultMessageListenerContainer#refreshConnectionUntilSuccessful
> 891 INFO] - Successfully refreshed JMS Connection
> [20141223 01:21:39.284 EST (jmsContainer1-1)
> org.springframework.jms.listener.DefaultMessageListenerContainer#refreshConnectionUntilSuccessful
> 891 INFO] - Successfully refreshed JMS Connection
> [20141223 01:21:44.286 EST (jmsContainer1-2)
> org.springframework.jms.listener.DefaultMessageListenerContainer#handleListenerSetupFailure
> 839 WARN] - Setup of JMS message listener invoker failed for destination
> 'topic://cometd.seedTCAStatusPush' - trying to recover. Cause:
> org.apache.activemq.ConnectionFailedException: The JMS connection has failed:
> Connection refused
> javax.jms.IllegalStateException:
> org.apache.activemq.ConnectionFailedException: The JMS connection has failed:
> Connection refused
> at
> org.apache.activemq.jms.pool.ConnectionPool.createSession(ConnectionPool.java:135)
> at
> org.apache.activemq.jms.pool.PooledConnection.createSession(PooledConnection.java:167)
> at
> org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:197)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer.access$1400(DefaultMessageListenerContainer.java:119)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.initResourcesIfNecessary(DefaultMessageListenerContainer.java:1122)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1101)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1094)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:991)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.activemq.ConnectionFailedException: The JMS connection
> has failed: Connection refused
> at
> org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1492)
> at
> org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:327)
> at
> org.apache.activemq.jms.pool.ConnectionPool.makeSession(ConnectionPool.java:104)
> at
> org.apache.activemq.jms.pool.ConnectionPool$1.makeObject(ConnectionPool.java:81)
> at
> org.apache.activemq.jms.pool.ConnectionPool$1.makeObject(ConnectionPool.java:66)
> at
> org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
> at
> org.apache.activemq.jms.pool.ConnectionPool.createSession(ConnectionPool.java:133)
> ... 8 more
> Caused by: java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:589)
> at
> org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:501)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:464)
> at
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:138)
> at
> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
> at
> org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
> at
> org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1017)
> at
> org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:148)
> at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
> at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ... 1 more
> [20141223 01:21:44.286 EST (jmsContainer2-2)
> org.springframework.jms.listener.DefaultMessageListenerContainer#handleListenerSetupFailure
> 839 WARN] - Setup of JMS message listener invoker failed for destination
> 'topic://cometd.seedTCAStatusPush' - trying to recover. Cause:
> org.apache.activemq.ConnectionFailedException: The JMS connection has failed:
> Connection refused
> javax.jms.IllegalStateException:
> org.apache.activemq.ConnectionFailedException: The JMS connection has failed:
> Connection refused
> at
> org.apache.activemq.jms.pool.ConnectionPool.createSession(ConnectionPool.java:135)
> at
> org.apache.activemq.jms.pool.PooledConnection.createSession(PooledConnection.java:167)
> at
> org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:197)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer.access$1400(DefaultMessageListenerContainer.java:119)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.initResourcesIfNecessary(DefaultMessageListenerContainer.java:1122)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1101)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1094)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:991)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.activemq.ConnectionFailedException: The JMS connection
> has failed: Connection refused
> at
> org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1492)
> at
> org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:327)
> at
> org.apache.activemq.jms.pool.ConnectionPool.makeSession(ConnectionPool.java:104)
> at
> org.apache.activemq.jms.pool.ConnectionPool$1.makeObject(ConnectionPool.java:81)
> at
> org.apache.activemq.jms.pool.ConnectionPool$1.makeObject(ConnectionPool.java:66)
> at
> org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
> at
> org.apache.activemq.jms.pool.ConnectionPool.createSession(ConnectionPool.java:133)
> ... 8 more
> Caused by: java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:589)
> at
> org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:501)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:464)
> at
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:138)
> at
> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
> at
> org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
> at
> org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1017)
> at
> org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:148)
> at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
> at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ... 1 more
> [20141223 01:21:44.288 EST (jmsContainer1-2)
> org.springframework.jms.listener.DefaultMessageListenerContainer#refreshConnectionUntilSuccessful
> 891 INFO] - Successfully refreshed JMS Connection
> [20141223 01:21:44.289 EST (jmsContainer2-2)
> org.springframework.jms.listener.DefaultMessageListenerContainer#refreshConnectionUntilSuccessful
> 891 INFO] - Successfully refreshed JMS Connection
> [20141223 01:21:49.291 EST (jmsContainer2-3)
> org.springframework.jms.listener.DefaultMessageListenerContainer#handleListenerSetupFailure
> 839 WARN] - Setup of JMS message listener invoker failed for destination
> 'topic://cometd.seedTCAStatusPush' - trying to recover. Cause:
> org.apache.activemq.ConnectionFailedException: The JMS connection has failed:
> Connection refused
> javax.jms.IllegalStateException:
> org.apache.activemq.ConnectionFailedException: The JMS connection has failed:
> Connection refused
> at
> org.apache.activemq.jms.pool.ConnectionPool.createSession(ConnectionPool.java:135)
> at
> org.apache.activemq.jms.pool.PooledConnection.createSession(PooledConnection.java:167)
> at
> org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:197)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer.access$1400(DefaultMessageListenerContainer.java:119)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.initResourcesIfNecessary(DefaultMessageListenerContainer.java:1122)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1101)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1094)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:991)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.activemq.ConnectionFailedException: The JMS connection
> has failed: Connection refused
> at
> org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1492)
> at
> org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:327)
> at
> org.apache.activemq.jms.pool.ConnectionPool.makeSession(ConnectionPool.java:104)
> at
> org.apache.activemq.jms.pool.ConnectionPool$1.makeObject(ConnectionPool.java:81)
> at
> org.apache.activemq.jms.pool.ConnectionPool$1.makeObject(ConnectionPool.java:66)
> at
> org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
> at
> org.apache.activemq.jms.pool.ConnectionPool.createSession(ConnectionPool.java:133)
> ... 8 more
> Caused by: java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:589)
> at
> org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:501)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:464)
> at
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:138)
> at
> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
> at
> org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
> at
> org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1017)
> at
> org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:148)
> at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
> at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ... 1 more
> This continues indefinitely. I checked lsof output at this point and there
> was no connection to ActiveMQ broker, so these connection attempts were not
> reaching the broker itself.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
