### Motivation Found a deadlock in broker with below jstack. [Jstack](https://gist.github.com/rdhabalia/9cffa5005716d077a8a767a0cc6afd1b)
expireMessages thread is waiting on topic-map and ml-operation couldn't complete which created a deadlock at broker. ``` Thread 17133: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Compiled frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int) @bci=72, line=997 (Compiled frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int) @bci=24, line=1304 (Compiled frame) - java.util.concurrent.CountDownLatch.await() @bci=5, line=231 (Compiled frame) - org.apache.bookkeeper.mledger.impl.ManagedCursorImpl.getNthEntry(int, org.apache.bookkeeper.mledger.ManagedCursor$IndividualDeletedEntries) @bci=38, line=470 (Interpreted frame) - org.apache.pulsar.broker.service.persistent.PersistentTopic.isOldestMessageExpired(org.apache.bookkeeper.mledger.ManagedCursor, long) @bci=14, line=1646 (Interpreted frame) - org.apache.pulsar.broker.service.persistent.PersistentReplicator.expireMessages(int) @bci=40, line=571 (Compiled frame) - org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$32(org.apache.pulsar.common.policies.data.Policies, java.lang.String, org.apache.pulsar.broker.service.Replicator) @bci=8, line=963 (Compiled frame) - org.apache.pulsar.broker.service.persistent.PersistentTopic$$Lambda$407.accept(java.lang.Object, java.lang.Object) @bci=12 (Compiled frame) - org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(java.util.function.BiConsumer) @bci=121, line=386 (Compiled frame) - org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(java.util.function.BiConsumer) @bci=26, line=160 (Compiled frame) - org.apache.pulsar.broker.service.persistent.PersistentTopic.checkMessageExpiry() @bci=86, line=963 (Compiled frame) - org.apache.pulsar.broker.service.BrokerService$$Lambda$366.accept(java.lang.Object) @bci=4 (Compiled frame) - org.apache.pulsar.broker.service.BrokerService.lambda$20(java.util.function.Consumer, java.lang.String, java.util.concurrent.CompletableFuture) @bci=20, line=887 (Compiled frame) - org.apache.pulsar.broker.service.BrokerService$$Lambda$349.accept(java.lang.Object, java.lang.Object) @bci=12 (Compiled frame) - org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(java.util.function.BiConsumer) @bci=121, line=386 (Compiled frame) - org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(java.util.function.BiConsumer) @bci=26, line=160 (Compiled frame) - org.apache.pulsar.broker.service.BrokerService.forEachTopic(java.util.function.Consumer) @bci=10, line=884 (Compiled frame) - org.apache.pulsar.broker.service.BrokerService.checkMessageExpiry() @bci=6, line=861 (Interpreted frame) - org.apache.pulsar.broker.service.BrokerService$$Lambda$135.run() @bci=4 (Interpreted frame) - org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun() @bci=4, line=32 (Compiled frame) - org.apache.bookkeeper.common.util.SafeRunnable.run() @bci=1, line=36 (Compiled frame) - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511 (Compiled frame) - java.util.concurrent.FutureTask.runAndReset() @bci=47, line=308 (Compiled frame) - java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask) @bci=1, line=180 (Compiled frame) - java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run() @bci=37, line=294 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame) - io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=748 (Interpreted frame) ``` [ Full content available at: https://github.com/apache/pulsar/pull/2745 ] This message was relayed via gitbox.apache.org for [email protected]
