[
https://issues.apache.org/jira/browse/AMQ-6666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096095#comment-16096095
]
Jakub edited comment on AMQ-6666 at 7/21/17 10:17 AM:
------------------------------------------------------
In my case it also occurs in 5.14.5. I use below configuration:
* JDBC persistence storage
* master - slave
* client uses failover protocol
* AMQ is part of XA transaction (2PC)
It leads to hang of entire application, because one of thread owns forever a
lock:
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:623)
- locked <0x000000073ec21360> (a java.lang.Object)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at
org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1388)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1428)
at
org.apache.activemq.TransactionContext.setXid(TransactionContext.java:751)
at
org.apache.activemq.TransactionContext.invokeBeforeEnd(TransactionContext.java:424)
at
org.apache.activemq.TransactionContext.end(TransactionContext.java:408)
at
org.apache.geronimo.transaction.manager.WrapperNamedXAResource.end(WrapperNamedXAResource.java:61)
at
org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:588)
at
org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:567)
at
org.apache.geronimo.transaction.manager.TransactionImpl.beforePrepare(TransactionImpl.java:414)
at
org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:262)
at
org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:252)
at
org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1020)
at
org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:761)
at
org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:730)
at
org.apache.aries.transaction.internal.AriesPlatformTransactionManager.commit(AriesPlatformTransactionManager.java:75)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:245)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1164)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1156)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1053)
at java.lang.Thread.run(Thread.java:748)
was (Author: ragnor84):
In my case it also occurs in 5.14.5. I use below configuration:
* JDBC persistence storage
* master - slave
* client uses failover protocol
* AMQ is part of XA transaction (2PC)
It leads to hang of entire application, because one of thread owns forever a
lock:
{{
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:623)
- locked <0x000000073ec21360> (a java.lang.Object)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at
org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1388)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1428)
at
org.apache.activemq.TransactionContext.setXid(TransactionContext.java:751)
at
org.apache.activemq.TransactionContext.invokeBeforeEnd(TransactionContext.java:424)
at
org.apache.activemq.TransactionContext.end(TransactionContext.java:408)
at
org.apache.geronimo.transaction.manager.WrapperNamedXAResource.end(WrapperNamedXAResource.java:61)
at
org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:588)
at
org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:567)
at
org.apache.geronimo.transaction.manager.TransactionImpl.beforePrepare(TransactionImpl.java:414)
at
org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:262)
at
org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:252)
at
org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1020)
at
org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:761)
at
org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:730)
at
org.apache.aries.transaction.internal.AriesPlatformTransactionManager.commit(AriesPlatformTransactionManager.java:75)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:245)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1164)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1156)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1053)
at java.lang.Thread.run(Thread.java:748)
}}
> Failover Transport - send timeout not working
> ---------------------------------------------
>
> Key: AMQ-6666
> URL: https://issues.apache.org/jira/browse/AMQ-6666
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.14.0
> Reporter: Martin Lichtin
> Priority: Critical
>
> Running into a situation with the Failover Transport not respecting the
> timeout that's been set. The symptom is endless messages of this kind:
> {noformat}
> 2017-04-29 09:48:26,128 | TRACE | .engine.cfgengine.in]-11 |
> FailoverTransport | sport.failover.FailoverTransport 615 | 81
> - org.apache.activemq.activemq-osgi - 5.14.0 | Waiting for transport to
> reconnect..: TransactionInfo {commandId = 127798, responseRequired = true,
> type = 7, connectionId = ID:inucdev4-57330-1493370444659-3:3, transactionId =
> XID:[1096044365,globalId=6374726c6366672d656e67696e653130333530323030303034,branchId=6374726c6366672d656e67696e6531313036383134]}
> 2017-04-29 09:48:26,228 | TRACE | .engine.cfgengine.in]-11 |
> FailoverTransport | sport.failover.FailoverTransport 615 | 81
> - org.apache.activemq.activemq-osgi - 5.14.0 | Waiting for transport to
> reconnect..: TransactionInfo {commandId = 127798, responseRequired = true,
> type = 7, connectionId = ID:inucdev4-57330-1493370444659-3:3, transactionId =
> XID:[1096044365,globalId=6374726c6366672d656e67696e653130333530323030303034,branchId=6374726c6366672d656e67696e6531313036383134]}
> 2017-04-29 09:48:26,329 | TRACE | .engine.cfgengine.in]-11 |
> FailoverTransport | sport.failover.FailoverTransport 615 | 81
> - org.apache.activemq.activemq-osgi - 5.14.0 | Waiting for transport to
> reconnect..: TransactionInfo {commandId = 127798, responseRequired = true,
> type = 7, connectionId = ID:inucdev4-57330-1493370444659-3:3, transactionId =
> XID:[1096044365,globalId=6374726c6366672d656e67696e653130333530323030303034,branchId=6374726c6366672d656e67696e6531313036383134]}
> 2017-04-29 09:48:26,429 | TRACE | .engine.cfgengine.in]-11 |
> FailoverTransport | sport.failover.FailoverTransport 615 | 81
> - org.apache.activemq.activemq-osgi - 5.14.0 | Waiting for transport to
> reconnect..: TransactionInfo {commandId = 127798, responseRequired = true,
> type = 7, connectionId = ID:inucdev4-57330-1493370444659-3:3, transactionId =
> XID:[1096044365,globalId=6374726c6366672d656e67696e653130333530323030303034,branchId=6374726c6366672d656e67696e6531313036383134]}
> 2017-04-29 09:48:26,530 | TRACE | .engine.cfgengine.in]-11 |
> FailoverTransport | sport.failover.FailoverTransport 615 | 81
> - org.apache.activemq.activemq-osgi - 5.14.0 | Waiting for transport to
> reconnect..: TransactionInfo {commandId = 127798, responseRequired = true,
> type = 7, connectionId = ID:inucdev4-57330-1493370444659-3:3, transactionId =
> XID:[1096044365,globalId=6374726c6366672d656e67696e653130333530323030303034,branchId=6374726c6366672d656e67696e6531313036383134]}
> ...
> 2017-04-29 09:48:33,270 | TRACE | .engine.cfgengine.in]-11 |
> FailoverTransport | sport.failover.FailoverTransport 615 | 81
> - org.apache.activemq.activemq-osgi - 5.14.0 | Waiting for transport to
> reconnect..: TransactionInfo {commandId = 127798, responseRequired = true,
> type = 7, connectionId = ID:inucdev4-57330-1493370444659-3:3, transactionId =
> XID:[1096044365,globalId=6374726c6366672d656e67696e653130333530323030303034,branchId=6374726c6366672d656e67696e6531313036383134]}
> 2017-04-29 09:48:33,371 | TRACE | .engine.cfgengine.in]-11 |
> FailoverTransport | sport.failover.FailoverTransport 615 | 81
> - org.apache.activemq.activemq-osgi - 5.14.0 | Waiting for transport to
> reconnect..: TransactionInfo {commandId = 127798, responseRequired = true,
> type = 7, connectionId = ID:inucdev4-57330-1493370444659-3:3, transactionId =
> XID:[1096044365,globalId=6374726c6366672d656e67696e653130333530323030303034,branchId=6374726c6366672d656e67696e6531313036383134]}
> {noformat}
> The code seems to never get out of this loop:
> {noformat}
> while (transport == null && !disposed && connectionFailure == null
> && !Thread.currentThread().isInterrupted() && willReconnect()) {
> LOG.trace("Waiting for transport to reconnect..: {}", command);
> long end = System.currentTimeMillis();
> if (command.isMessage() && timeout > 0 && (end - start > timeout)) {
> timedout = true;
> LOG.info("Failover timed out after {} ms", (end - start));
> break;
> }
> try {
> reconnectMutex.wait(100);
> } catch (InterruptedException e) {
> Thread.currentThread().interrupt();
> LOG.debug("Interupted:", e);
> }
> transport = connectedTransport.get();
> }
> {noformat}
> The timeout is set to 5000ms and should have hit a long time ago, but as
> "command.isMessage()" returns false, it endlessly stays inside the loop.
> The "command.isMessage()" condition should likely be removed.
> Currently running tests with a patched ActiveMQ, and the situation has
> improved and fail-over seems to have worked (mostly). (Only seeing an issue
> with a topic consumer that has not reconnected.)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
