michaeljmarshall commented on issue #14438:
URL: https://github.com/apache/pulsar/issues/14438#issuecomment-1054722361
@leizhiyuan - after looking through your thread dump a bit more, I am
wondering if you have a separate issue in your delayed message implementation?
I see that thread is blocked on some other thread that is completing a lookup
call (that lookup call has a timeout on it, so that thread isn't indefinitely
blocked). Is there any chance that the `"DM-sync-delay-index"` thread being
blocked is causing an issue here?
```
"DM-sync-delay-index" #223 daemon prio=5 os_prio=0 tid=0x00007fcbd8023800
nid=0x7cb5 waiting for monitor entry [0x00007fcaa193f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1868)
- waiting to lock <0x0000000760a001d8> (a
java.util.concurrent.ConcurrentHashMap$ReservationNode)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2037)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2020)
at
com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:112)
at
com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:67)
at
com.tencent.tdmq.delaymessage.inner.MetadataCache.isDelayTenantOwner(MetadataCache.java:151)
at
com.tencent.tdmq.delaymessage.inner.SyncIndexService.syncTenant(SyncIndexService.java:65)
at
com.tencent.tdmq.delaymessage.inner.SyncIndexService.syncAllTenants(SyncIndexService.java:52)
at
com.tencent.tdmq.delaymessage.inner.SyncIndexService.lambda$new$1(SyncIndexService.java:41)
at
com.tencent.tdmq.delaymessage.inner.SyncIndexService$$Lambda$832/1549413052.run(Unknown
Source)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
```
"DM-clear-thread" #227 daemon prio=5 os_prio=0 tid=0x00007fcbd8035800
nid=0x7cb9 waiting on condition [0x00007fcaa153a000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000760a08240> (a
java.util.concurrent.CompletableFuture$Signaller)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1695)
at
java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at
java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1775)
at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
at
org.apache.pulsar.client.admin.internal.LookupImpl.lookupTopic(LookupImpl.java:56)
at
com.tencent.tdmq.delaymessage.inner.MetadataCache.lookup(MetadataCache.java:166)
at
com.tencent.tdmq.delaymessage.inner.MetadataCache$$Lambda$840/1429314972.load(Unknown
Source)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:3366)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache$$Lambda$839/2030879142.apply(Unknown
Source)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2039)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$88/947462790.apply(Unknown
Source)
at
java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
- locked <0x0000000760a001d8> (a
java.util.concurrent.ConcurrentHashMap$ReservationNode)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2037)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2020)
at
com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:112)
at
com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:67)
at
com.tencent.tdmq.delaymessage.inner.MetadataCache.isDelayTenantOwner(MetadataCache.java:151)
at
com.tencent.tdmq.delaymessage.inner.ClearTask.check(ClearTask.java:67)
at
com.tencent.tdmq.delaymessage.inner.ClearTask$$Lambda$846/1309049241.run(Unknown
Source)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
