[
https://issues.apache.org/jira/browse/AMQ-9448?focusedWorklogId=921617&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-921617
]
ASF GitHub Bot logged work on AMQ-9448:
---------------------------------------
Author: ASF GitHub Bot
Created on: 01/Jun/24 06:55
Start Date: 01/Jun/24 06:55
Worklog Time Spent: 10m
Work Description: thezbyg commented on PR #1177:
URL: https://github.com/apache/activemq/pull/1177#issuecomment-2143330637
Yes. This is the stack trace of "JobScheduler:JMS" thread blocked by itself
due to acquiring read lock on store while iterating scheduled jobs in
mainLoop() method and then attempting to acquire write lock to write a new
scheduled job information:
`
"JobScheduler:JMS" #28 daemon prio=5 os_prio=0 cpu=11.62ms elapsed=117.73s
tid=0x00007f43d1344b50 nid=0x185b waiting on condition [0x00007f4356cfe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x000000008bd25150> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at
java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:211)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:715)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:938)
at
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock([email protected]/ReentrantReadWriteLock.java:959)
at
org.apache.activemq.store.kahadb.scheduler.JobSchedulerStoreImpl$8.visit(JobSchedulerStoreImpl.java:684)
at
org.apache.activemq.store.kahadb.data.KahaAddScheduledJobCommand.visit(KahaAddScheduledJobCommand.java:283)
at
org.apache.activemq.store.kahadb.scheduler.JobSchedulerStoreImpl.process(JobSchedulerStoreImpl.java:679)
at
org.apache.activemq.store.kahadb.AbstractKahaDBStore.store(AbstractKahaDBStore.java:495)
at
org.apache.activemq.store.kahadb.AbstractKahaDBStore.store(AbstractKahaDBStore.java:403)
at
org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.doSchedule(JobSchedulerImpl.java:252)
at
org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.schedule(JobSchedulerImpl.java:100)
at
org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:782)
at
org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:699)
at java.lang.Thread.run([email protected]/Thread.java:833)
`
Full dump:
[dump.txt](https://github.com/user-attachments/files/15520722/dump.txt)
Issue Time Tracking
-------------------
Worklog Id: (was: 921617)
Time Spent: 50m (was: 40m)
> RedeliveryPolicy causing lock in dispatcher
> -------------------------------------------
>
> Key: AMQ-9448
> URL: https://issues.apache.org/jira/browse/AMQ-9448
> Project: ActiveMQ Classic
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.18.3
> Environment: ubuntu-22.04
> corretto-17
> Reporter: Oliver Holzmann
> Priority: Critical
> Attachments: doNotFireJobsWhileStoreLockIsHeld.patch, jstack1.out
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> We are using ActiveMQ with redeliveryPolicy enabled:
> {code:java}
> <defaultEntry>
> <redeliveryPolicy maximumRedeliveries="60" initialRedeliveryDelay="1000"
> redeliveryDelay="1000" />
> </defaultEntry>{code}
> When a message exceeds maximumRedeliveries the queue is "frozen". This means
> the consumers can't read any messages and qrowing the queue in web console
> causes error 504.
> A thread dump shows a blocked JobScheduler:JMS thread:
> {code:java}
> priority : 5
> thread Id : x00007fb985664960
> native Id : xae
> state : BLOCKED
> StackTrace:
> "JobScheduler:JMS" #75 daemon prio=5 os_prio=0 cpu=478.63ms elapsed=757.29s
> tid=0x00007fb985664960 nid=0xae waiting for monitor entry [0x00007fb80def1000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:647)
> waiting to lock <0x000000077ebc62a0> (a java.lang.Object) locked
> <0x000000077ebc6290> (a java.lang.Object)
> at
> org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:159)
> at org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:2208)
> at org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:2156)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2307)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1728) locked
> <0x000000077ebc5178> (a java.lang.Object)
> at org.apache.activemq.broker.region.Queue.wakeup(Queue.java:1977)
> at org.apache.activemq.broker.region.Queue.messageSent(Queue.java:1971)
> at
> org.apache.activemq.broker.region.Queue.doPendingCursorAdditions(Queue.java:832)
> at org.apache.activemq.broker.region.Queue.tryOrderedCursorAdd(Queue.java:927)
> at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:906)
> at org.apache.activemq.broker.region.Queue.send(Queue.java:756)
> at
> org.apache.activemq.broker.region.DestinationFilter.send(DestinationFilter.java:138)
> at
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:511)
> at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:477)
> at
> org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:343)
> at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
> at
> org.apache.activemq.broker.scheduler.SchedulerBroker.scheduledJob(SchedulerBroker.java:429)
> at
> org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.fireJob(JobSchedulerImpl.java:823)
> at
> org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:753)
> at
> org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:699)
> at java.lang.Thread.run([email protected]/Thread.java:840)
> Locked ownable synchronizers: <0x000000077ebc3ed0> (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
For further information, visit: https://activemq.apache.org/contact
