[
https://issues.apache.org/jira/browse/AMQ-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18042004#comment-18042004
]
ASF subversion and git services commented on AMQ-5785:
------------------------------------------------------
Commit f80260774e40fffb1e9d32f2d0c3c6a84018797e in activemq's branch
refs/heads/activemq-6.1.x from Radek Kraus
[ https://gitbox.apache.org/repos/asf?p=activemq.git;h=f80260774e ]
AMQ-9813 - fix wrong QueueSize for non-persistent message with TTL
- add "missing" invocation of discardExpiredMessage() method
into tryAddMessageLast(), addMessageFirst(), probably caused
in context of AMQ-5785
- use same "postponed" strategy (outside of synchronization) like was
already done in original commit (see onUsageChanged() method)
(cherry picked from commit 5c5eb335eefc2e0422b17c512eec7796f8ab08df)
> Deadlock between FilePendingMessageCursor usage change and incoming send
> operations.
> ------------------------------------------------------------------------------------
>
> Key: AMQ-5785
> URL: https://issues.apache.org/jira/browse/AMQ-5785
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.10.0
> Environment: Physical Machine (192 GB RAM, 24 VCPU), RHEL 5.9, Java
> 1.7
> ActiveMQ runs on 4 GB heap
> Reporter: Sree Panchajanyam D
> Assignee: Timothy A. Bish
> Priority: Critical
> Attachments: threaddump16214.txt
>
>
> During the peak loads we are encountering a recurring deadlock issue in
> ActiveMQ broker. The threads that are deadlocked are
> ActiveMQ NIO Worker - trying to add message to FilePendingCursor
> Broker.Service Worker - that is trying to expire message from
> FilePendingCursor.
> =============================
> Found one Java-level deadlock:
> =============================
> "ActiveMQ NIO Worker 1003":
> waiting to lock monitor 0x00002aeeb515a4f8 (object 0x00000007807da3e8, a
> org.apache.activemq.broker.region.cursors.FilePendingMessageCursor),
> which is held by "ActiveMQ BrokerService.worker.1"
> "ActiveMQ BrokerService.worker.1":
> waiting for ownable synchronizer 0x000000077ac84b40, (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
> which is held by "ActiveMQ NIO Worker 1003"
> Java stack information for the threads listed above:
> ===================================================
> "ActiveMQ NIO Worker 1003":
> at
> org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.addMessageLast(FilePendingMessageCursor.java:207)
> - waiting to lock <0x00000007807da3e8> (a
> org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
> at
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.addMessageLast(StoreQueueCursor.java:96)
> - locked <0x00000007784e8c88> (a
> org.apache.activemq.broker.region.cursors.StoreQueueCursor)
> at org.apache.activemq.broker.region.Queue.sendMessage(Queue.java:1855)
> at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:939)
> at org.apache.activemq.broker.region.Queue.send(Queue.java:733)
> at
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:424)
> at
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:445)
> at
> org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:297)
> at
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
> at
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:307)
> at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:147)
> at org.apache.activemq.broker.UserIDBroker.send(UserIDBroker.java:56)
> at
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:152)
> at
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:496)
> at
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:756)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:148)
> at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> at
> org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:138)
> at
> org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)
> at
> org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)
> at
> org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> "ActiveMQ BrokerService.worker.1":
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000000077ac84b40> (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
> at
> org.apache.activemq.broker.region.Queue.messageExpired(Queue.java:1841)
> at
> org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.discardExpiredMessage(FilePendingMessageCursor.java:474)
> at
> org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.expireOldMessages(FilePendingMessageCursor.java:420)
> - locked <0x00000007807da3e8> (a
> org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
> at
> org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.onUsageChanged(FilePendingMessageCursor.java:398)
> - locked <0x00000007807da3e8> (a
> org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
> at org.apache.activemq.usage.Usage$1.run(Usage.java:304)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
For further information, visit: https://activemq.apache.org/contact
