[
https://issues.apache.org/jira/browse/AMQ-9448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17826956#comment-17826956
]
Albertas Vyšniauskas commented on AMQ-9448:
-------------------------------------------
After commit
[https://github.com/apache/activemq/commit/60859b0b7fa93b001ca2b0ac2f2385386cb2f47d|http://example.com/],
persistent scheduler fires scheduled jobs while holding read lock on store. If
fired job tries to add a new scheduled job then the attempt to acquire a write
lock on store is made and deadlock occurs.
I tried running existing scheduler tests in
activemq-unit-tests/src/test/java/org/apache/activemq/broker/scheduler/ and
they also deadlock, but it happens due to repeatable CRON jobs, because
schedule() method is also called while holding read lock and requires write
lock to modify schedule store.
This patch delays job firing and schedule calls, so that they happen once read
lock is released: [^doNotFireJobsWhileStoreLockIsHeld.patch]
> RedeliveryPolicy causing lock in dispatcher
> -------------------------------------------
>
> Key: AMQ-9448
> URL: https://issues.apache.org/jira/browse/AMQ-9448
> Project: ActiveMQ Classic
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.18.3
> Environment: ubuntu-22.04
> corretto-17
> Reporter: Oliver Holzmann
> Priority: Critical
> Attachments: doNotFireJobsWhileStoreLockIsHeld.patch, jstack1.out
>
>
> We are using ActiveMQ with redeliveryPolicy enabled:
> {code:java}
> <defaultEntry>
> <redeliveryPolicy maximumRedeliveries="60" initialRedeliveryDelay="1000"
> redeliveryDelay="1000" />
> </defaultEntry>{code}
> When a message exceeds maximumRedeliveries the queue is "frozen". This means
> the consumers can't read any messages and qrowing the queue in web console
> causes error 504.
> A thread dump shows a blocked JobScheduler:JMS thread:
> {code:java}
> priority : 5
> thread Id : x00007fb985664960
> native Id : xae
> state : BLOCKED
> StackTrace:
> "JobScheduler:JMS" #75 daemon prio=5 os_prio=0 cpu=478.63ms elapsed=757.29s
> tid=0x00007fb985664960 nid=0xae waiting for monitor entry [0x00007fb80def1000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:647)
> waiting to lock <0x000000077ebc62a0> (a java.lang.Object) locked
> <0x000000077ebc6290> (a java.lang.Object)
> at
> org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:159)
> at org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:2208)
> at org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:2156)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2307)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1728) locked
> <0x000000077ebc5178> (a java.lang.Object)
> at org.apache.activemq.broker.region.Queue.wakeup(Queue.java:1977)
> at org.apache.activemq.broker.region.Queue.messageSent(Queue.java:1971)
> at
> org.apache.activemq.broker.region.Queue.doPendingCursorAdditions(Queue.java:832)
> at org.apache.activemq.broker.region.Queue.tryOrderedCursorAdd(Queue.java:927)
> at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:906)
> at org.apache.activemq.broker.region.Queue.send(Queue.java:756)
> at
> org.apache.activemq.broker.region.DestinationFilter.send(DestinationFilter.java:138)
> at
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:511)
> at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:477)
> at
> org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:343)
> at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
> at
> org.apache.activemq.broker.scheduler.SchedulerBroker.scheduledJob(SchedulerBroker.java:429)
> at
> org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.fireJob(JobSchedulerImpl.java:823)
> at
> org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:753)
> at
> org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:699)
> at java.lang.Thread.run([email protected]/Thread.java:840)
> Locked ownable synchronizers: <0x000000077ebc3ed0> (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
