[
https://issues.apache.org/jira/browse/AMQ-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15607861#comment-15607861
]
Michal Kubricht commented on AMQ-5785:
--------------------------------------
Reviewing the commit I found that it introduces new mistake.
I believe that commit solves the deadlock problem, but...
It breaks original implementation of method {{expireOldMessages()}} which is:
# still used by other methods - {{tryAddMessageLast(MessageReference, long)}},
{{addMessageFirst(MessageReference)}}
# protected, so potentially used by extending classes
Problem caused by changing {{expireOldMessages()}} is that it breaks the
functionality of freeing space when adding new messages to the cursor.
Currently after this fix it only creates a an unnecessary load of expired
messages to the list which is never used.
Commenting on these lines in code (which are shared by both addMessageX
methods):
{code}
if (!hasSpace()) {
if (isDiskListEmpty()) {
expireOldMessages(); // this will not clear anything!
if (hasSpace()) { // this probably won't happen
memoryList.addMessageFirst(node);
node.incrementReferenceCount();
return;
} else { // will always fallback to flushing data to disk
flushToDisk();
}
}
}
{code}
> Deadlock between FilePendingMessageCursor usage change and incoming send
> operations.
> ------------------------------------------------------------------------------------
>
> Key: AMQ-5785
> URL: https://issues.apache.org/jira/browse/AMQ-5785
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.10.0
> Environment: Physical Machine (192 GB RAM, 24 VCPU), RHEL 5.9, Java
> 1.7
> ActiveMQ runs on 4 GB heap
> Reporter: Sree Panchajanyam D
> Assignee: Timothy Bish
> Priority: Critical
> Attachments: threaddump16214.txt
>
>
> During the peak loads we are encountering a recurring deadlock issue in
> ActiveMQ broker. The threads that are deadlocked are
> ActiveMQ NIO Worker - trying to add message to FilePendingCursor
> Broker.Service Worker - that is trying to expire message from
> FilePendingCursor.
> =============================
> Found one Java-level deadlock:
> =============================
> "ActiveMQ NIO Worker 1003":
> waiting to lock monitor 0x00002aeeb515a4f8 (object 0x00000007807da3e8, a
> org.apache.activemq.broker.region.cursors.FilePendingMessageCursor),
> which is held by "ActiveMQ BrokerService.worker.1"
> "ActiveMQ BrokerService.worker.1":
> waiting for ownable synchronizer 0x000000077ac84b40, (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
> which is held by "ActiveMQ NIO Worker 1003"
> Java stack information for the threads listed above:
> ===================================================
> "ActiveMQ NIO Worker 1003":
> at
> org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.addMessageLast(FilePendingMessageCursor.java:207)
> - waiting to lock <0x00000007807da3e8> (a
> org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
> at
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.addMessageLast(StoreQueueCursor.java:96)
> - locked <0x00000007784e8c88> (a
> org.apache.activemq.broker.region.cursors.StoreQueueCursor)
> at org.apache.activemq.broker.region.Queue.sendMessage(Queue.java:1855)
> at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:939)
> at org.apache.activemq.broker.region.Queue.send(Queue.java:733)
> at
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:424)
> at
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:445)
> at
> org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:297)
> at
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
> at
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:307)
> at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:147)
> at org.apache.activemq.broker.UserIDBroker.send(UserIDBroker.java:56)
> at
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:152)
> at
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:496)
> at
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:756)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:148)
> at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> at
> org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:138)
> at
> org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)
> at
> org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)
> at
> org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> "ActiveMQ BrokerService.worker.1":
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000000077ac84b40> (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
> at
> org.apache.activemq.broker.region.Queue.messageExpired(Queue.java:1841)
> at
> org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.discardExpiredMessage(FilePendingMessageCursor.java:474)
> at
> org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.expireOldMessages(FilePendingMessageCursor.java:420)
> - locked <0x00000007807da3e8> (a
> org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
> at
> org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.onUsageChanged(FilePendingMessageCursor.java:398)
> - locked <0x00000007807da3e8> (a
> org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
> at org.apache.activemq.usage.Usage$1.run(Usage.java:304)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
