[
https://issues.apache.org/jira/browse/AMQ-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14483067#comment-14483067
]
Torsten Mielke commented on AMQ-5692:
-------------------------------------
Attached reproducer in AMQ-5692.pl.
To reproduce issue follow these steps:
- Run broker with stomp connector on port 61613 and this connector configuration
{code:xml}
<transportConnector name="stomp"
uri="stomp://0.0.0.0:61613?transport.defaultHeartBeat=10000,0&transport.useKeepAlive=true&trace=true"/>
{code}
- Run consumer
{code}
perl ./AMQ-5692.pl consumer
{code}
- Run producer
{code}
perl ./AMQ-5692.pl producer
{code}
After consuming around 8000 messages, the consumer hangs and won't report any
more messages consumed.
Take a broker thread dump and observe two stuck threads as in (1) below.
Despite the configured stomp heart-beat defaultHeartBeat=10000,0 the connection
does not time out after 10 seconds and no keep-alive is sent either.
Have not managed to reproduce the issue with the ActiveMQ stomp client library.
(1)
{code}
"ActiveMQ BrokerService[localhost] Task-4" daemon prio=5 tid=0x00007fe9ca087800
nid=0x650b runnable [0x00000001192d9000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
at
org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at
org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:176)
at
org.apache.activemq.transport.stomp.StompTransportFilter.sendToStomp(StompTransportFilter.java:98)
at
org.apache.activemq.transport.stomp.StompSubscription.onMessageDispatch(StompSubscription.java:103)
at
org.apache.activemq.transport.stomp.ProtocolConverter.onActiveMQCommand(ProtocolConverter.java:870)
at
org.apache.activemq.transport.stomp.StompTransportFilter.oneway(StompTransportFilter.java:62)
at
org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:304)
at
org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1419)
at
org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:938)
at
org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:984)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
"ActiveMQ Transport: tcp:///192.168.178.28:60902@61613" daemon prio=5
tid=0x00007fe9ca086800 nid=0x3b0f waiting on condition [0x0000000118f2d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007c0ecc2f0> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:43)
at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
at
org.apache.activemq.transport.stomp.StompTransportFilter.sendToActiveMQ(StompTransportFilter.java:87)
at
org.apache.activemq.transport.stomp.ProtocolConverter.sendToActiveMQ(ProtocolConverter.java:199)
at
org.apache.activemq.transport.stomp.ProtocolConverter.onStompAck(ProtocolConverter.java:433)
at
org.apache.activemq.transport.stomp.ProtocolConverter.onStompCommand(ProtocolConverter.java:247)
at
org.apache.activemq.transport.stomp.StompTransportFilter.onCommand(StompTransportFilter.java:75)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
at java.lang.Thread.run(Thread.java:744)
{code}
> Inactivity monitor does not time out on stuck socket writes
> ------------------------------------------------------------
>
> Key: AMQ-5692
> URL: https://issues.apache.org/jira/browse/AMQ-5692
> Project: ActiveMQ
> Issue Type: Improvement
> Components: Broker
> Affects Versions: 5.11.1
> Reporter: Torsten Mielke
> Labels: broker, inactivity
> Attachments: AMQ-5692.pl
>
>
> It is possible that a socket write is stuck but the inactivity monitor
> currently does not time out on a socketWrite.
> {code:title=AbstractInactivityMonitor.java}
> final void writeCheck() {
> if (inSend.get()) {
> LOG.trace("Send in progress. Skipping write check.");
> return;
> }
> {code}
> As a result a connection that is stuck in a tcp write will never be taken
> down due to inactivity. If a client misbehaves the broker will not be able to
> clear that connection as part of the inactivity monitoring.
> Now AMQ-2511 introduced a counter on the reachCheck() to detect it a socket
> read in progress really retrieves data or is stuck.
> I propose for a similar mechanism being applied on the writeCheck() operation
> so that a socket write that is stuck can be detected and the connection can
> be closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
