leizhiyuan commented on issue #14438: URL: https://github.com/apache/pulsar/issues/14438#issuecomment-1054910918
> @leizhiyuan - after looking through your thread dump a bit more, I am wondering if you have a separate issue in your delayed message implementation? I see that thread is blocked on some other thread that is completing a lookup call (that lookup call has a timeout on it, so that thread isn't indefinitely blocked). Is there any chance that the `"DM-sync-delay-index"` thread being blocked is causing an issue here? > > ``` > "DM-sync-delay-index" #223 daemon prio=5 os_prio=0 tid=0x00007fcbd8023800 nid=0x7cb5 waiting for monitor entry [0x00007fcaa193f000] > java.lang.Thread.State: BLOCKED (on object monitor) > at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1868) > - waiting to lock <0x0000000760a001d8> (a java.util.concurrent.ConcurrentHashMap$ReservationNode) > at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2037) > at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2020) > at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:112) > at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:67) > at com.tencent.tdmq.delaymessage.inner.MetadataCache.isDelayTenantOwner(MetadataCache.java:151) > at com.tencent.tdmq.delaymessage.inner.SyncIndexService.syncTenant(SyncIndexService.java:65) > at com.tencent.tdmq.delaymessage.inner.SyncIndexService.syncAllTenants(SyncIndexService.java:52) > at com.tencent.tdmq.delaymessage.inner.SyncIndexService.lambda$new$1(SyncIndexService.java:41) > at com.tencent.tdmq.delaymessage.inner.SyncIndexService$$Lambda$832/1549413052.run(Unknown Source) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > ``` > > ``` > "DM-clear-thread" #227 daemon prio=5 os_prio=0 tid=0x00007fcbd8035800 nid=0x7cb9 waiting on condition [0x00007fcaa153a000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x0000000760a08240> (a java.util.concurrent.CompletableFuture$Signaller) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1695) > at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323) > at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1775) > at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) > at org.apache.pulsar.client.admin.internal.LookupImpl.lookupTopic(LookupImpl.java:56) > at com.tencent.tdmq.delaymessage.inner.MetadataCache.lookup(MetadataCache.java:166) > at com.tencent.tdmq.delaymessage.inner.MetadataCache$$Lambda$840/1429314972.load(Unknown Source) > at com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:3366) > at com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache$$Lambda$839/2030879142.apply(Unknown Source) > at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2039) > at com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$88/947462790.apply(Unknown Source) > at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853) > - locked <0x0000000760a001d8> (a java.util.concurrent.ConcurrentHashMap$ReservationNode) > at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2037) > at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2020) > at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:112) > at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:67) > at com.tencent.tdmq.delaymessage.inner.MetadataCache.isDelayTenantOwner(MetadataCache.java:151) > at com.tencent.tdmq.delaymessage.inner.ClearTask.check(ClearTask.java:67) > at com.tencent.tdmq.delaymessage.inner.ClearTask$$Lambda$846/1309049241.run(Unknown Source) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > ``` Thanks for your help, but we analyzed this thread, `DM-sync-delay-index` will not affect the lookup process. It only uses cache to lookup if a topic exists, if cache missed ,It will lookup use `PulsarService` <img width="922" alt="image" src="https://user-images.githubusercontent.com/2684384/156091563-b10c79f2-e1ed-44de-a247-0f104147c9ca.png";> when cache missed, we do this. <img width="1006" alt="image" src="https://user-images.githubusercontent.com/2684384/156091601-311364f2-4dbc-4ae7-9dce-47806c39711b.png";> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
![https://user-images.githubusercontent.com/2684384/156091563-b10c79f2-e1ed-44de-a247-0f104147c9ca.png"](https://user-images.githubusercontent.com/2684384/156091563-b10c79f2-e1ed-44de-a247-0f104147c9ca.png"){kind=link}
![https://user-images.githubusercontent.com/2684384/156091601-311364f2-4dbc-4ae7-9dce-47806c39711b.png"](https://user-images.githubusercontent.com/2684384/156091601-311364f2-4dbc-4ae7-9dce-47806c39711b.png"){kind=link}