optimizeDispatch=true and UseDedicatedTaskRunner=false lead to deadlocked queues
--------------------------------------------------------------------------------
Key: AMQ-2356
URL: https://issues.apache.org/activemq/browse/AMQ-2356
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.2.0
Reporter: Mats Henrikson
We have an environment where we have a very large number of destinations. In an
effort to reduce the number of threads I have set the options
-Dorg.apache.activemq.UseDedicatedTaskRunner=false
and
<policyEntry queue=">" optimizedDispatch="true"/>
Unfortunately this very quickly leads to deadlocked queues.
My environment is:
ActiveMQ 5.2
Ubunty Jaunty kernel 2.6.28-14-generic #47-Ubuntu SMP (although only a single
core on my system)
TCP transportConnector
To reproduce the bug (which I can do 100% of the time) I connect 5 consumers
(AUTO_ACK) to 5 different queues. Then I start 5 producers and pair them up
with a consumer on a queue, and they start sending PERSISTENT messages. I've
set the producer to send 100 messages and disconnect, and the consumer to
receive 100 messages and disconnect. The first pair usually gets through their
100 messages and disconnect, at which point all the other pairs have deadlocked
at less than 30 messages each.
At this point I can connect jconsole and hit the Detect Deadlock button and it
finds the 4 deadlocks for me. The 4 deadlocks all have the same stacktraces,
one of which looks like this:
{panel}
Name: ActiveMQ Transport: tcp:///127.0.0.1:49307
State: BLOCKED on java.lang.obj...@f8828 owned by: ActiveMQ Transport:
tcp:///127.0.0.1:48031
Total blocked: 3 Total waited: 1
Stack trace:
org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:576)
org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:150)
org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:1266)
org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:1230)
org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1308)
org.apache.activemq.broker.region.Queue.iterate(Queue.java:1011)
org.apache.activemq.broker.region.Queue.wakeup(Queue.java:1146)
org.apache.activemq.broker.region.Queue.sendMessage(Queue.java:1141)
org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:474)
org.apache.activemq.broker.region.Queue.send(Queue.java:417)
org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:350)
org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:437)
org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
...SpecialAuthorizationBroker.send(DynamicAuthorizationBroker.java:156)
org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:450)
org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
java.lang.Thread.run(Thread.java:595)
----------------------------------------------------------------------------------
Name: ActiveMQ Transport: tcp:///127.0.0.1:48031
State: BLOCKED on org.apache.activemq.broker.region.queu...@102d82c owned by:
ActiveMQ Transport: tcp:///127.0.0.1:49307
Total blocked: 3 Total waited: 0
Stack trace:
org.apache.activemq.broker.region.Queue.iterate(Queue.java:951)
org.apache.activemq.broker.region.Queue.wakeup(Queue.java:1146)
org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1073)
org.apache.activemq.broker.region.QueueSubscription.acknowledge(QueueSubscription.java:51)
org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:238)
org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:373)
org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:462)
org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:85)
org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:456)
org.apache.activemq.command.MessageAck.visit(MessageAck.java:205)
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
java.lang.Thread.run(Thread.java:595)
{panel}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
