mattisonchao opened a new pull request, #23786:
URL: https://github.com/apache/pulsar/pull/23786
### Motivation
```
Found one Java-level deadlock:
=============================
"metadata-store-10-1":
waiting to lock monitor 0x00007fb363ef6a60 (object 0x00000000bf8b3b88, a
java.util.concurrent.ConcurrentHashMap$Node),
which is held by "broker-client-shared-internal-executor-5-1"
"broker-client-shared-internal-executor-5-1":
waiting to lock monitor 0x00007fb36617a5b0 (object 0x00000000bdf00940, a
java.util.concurrent.ConcurrentHashMap$ReservationNode),
which is held by "pulsar-io-3-6"
"pulsar-io-3-6":
waiting to lock monitor 0x00007fb363ef6a60 (object 0x00000000bf8b3b88, a
java.util.concurrent.ConcurrentHashMap$Node),
which is held by "broker-client-shared-internal-executor-5-1"
Java stack information for the threads listed above:
===================================================
"metadata-store-10-1":
at
java.util.concurrent.ConcurrentHashMap.compute([email protected]/Unknown Source)
- waiting to lock <0x00000000bf8b3b88> (a
java.util.concurrent.ConcurrentHashMap$Node)
at
org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService.lambda$getTopicPoliciesAsync$10(SystemTopicBasedTopicPoliciesService.java:271)
at
org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService$$Lambda/0x0000000100bbdac0.accept(Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniAcceptNow([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniAcceptStage([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture.thenAccept([email protected]/Unknown
Source)
at
org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService.getTopicPoliciesAsync(SystemTopicBasedTopicPoliciesService.java:271)
at
org.apache.pulsar.broker.service.BrokerService.getTopicPoliciesBypassSystemTopic(BrokerService.java:1186)
at
org.apache.pulsar.broker.service.BrokerService.lambda$getTopic$37(BrokerService.java:1080)
at
org.apache.pulsar.broker.service.BrokerService$$Lambda/0x00000001007af710.apply(Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniComposeStage([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture.thenCompose([email protected]/Unknown
Source)
at
org.apache.pulsar.broker.service.BrokerService.getTopic(BrokerService.java:1076)
at
org.apache.pulsar.broker.service.BrokerService.getTopic(BrokerService.java:1040)
at
org.apache.pulsar.broker.service.BrokerService.getTopic(BrokerService.java:1035)
at
org.apache.pulsar.broker.service.ServerCnx.lambda$handleSubscribe$18(ServerCnx.java:1331)
at
org.apache.pulsar.broker.service.ServerCnx$$Lambda/0x0000000100ba4230.apply(Unknown
Source)
at
java.util.concurrent.CompletableFuture$UniCompose.tryFire([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture.postComplete([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture.complete([email protected]/Unknown Source)
at
org.apache.pulsar.metadata.impl.ZKMetadataStore.handleGetResult(ZKMetadataStore.java:289)
at
org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$7(ZKMetadataStore.java:233)
at
org.apache.pulsar.metadata.impl.ZKMetadataStore$$Lambda/0x00000001004c1740.run(Unknown
Source)
at
java.util.concurrent.Executors$RunnableAdapter.call([email protected]/Unknown
Source)
at java.util.concurrent.FutureTask.run([email protected]/Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown
Source)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.runWith([email protected]/Unknown Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"broker-client-shared-internal-executor-5-1":
at
java.util.concurrent.ConcurrentHashMap.transfer([email protected]/Unknown Source)
- waiting to lock <0x00000000bdf00940> (a
java.util.concurrent.ConcurrentHashMap$ReservationNode)
at
java.util.concurrent.ConcurrentHashMap.addCount([email protected]/Unknown Source)
at
java.util.concurrent.ConcurrentHashMap.computeIfAbsent([email protected]/Unknown
Source)
at
org.apache.pulsar.broker.service.BrokerService.lambda$getTopic$36(BrokerService.java:1110)
at
org.apache.pulsar.broker.service.BrokerService$$Lambda/0x00000001007ada38.apply(Unknown
Source)
at
java.util.concurrent.CompletableFuture$UniCompose.tryFire([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture.postComplete([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture.complete([email protected]/Unknown Source)
at
org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService.lambda$getTopicPoliciesAsync$9(SystemTopicBasedTopicPoliciesService.java:280)
at
org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService$$Lambda/0x0000000100c36818.apply(Unknown
Source)
at
java.util.concurrent.ConcurrentHashMap.compute([email protected]/Unknown Source)
- locked <0x00000000bf8b3b88> (a
java.util.concurrent.ConcurrentHashMap$Node)
at
org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService.lambda$getTopicPoliciesAsync$10(SystemTopicBasedTopicPoliciesService.java:271)
at
org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService$$Lambda/0x0000000100bbdac0.accept(Unknown
Source)
at
java.util.concurrent.CompletableFuture$UniAccept.tryFire([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture.postComplete([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture.complete([email protected]/Unknown Source)
at
org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService.lambda$initPolicesCache$21(SystemTopicBasedTopicPoliciesService.java:463)
at
org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService$$Lambda/0x0000000100c2be30.accept(Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniWhenComplete([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture.postComplete([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture.complete([email protected]/Unknown Source)
at
org.apache.pulsar.client.impl.MultiTopicsConsumerImpl.lambda$hasMessageAvailableAsync$37(MultiTopicsConsumerImpl.java:879)
at
org.apache.pulsar.client.impl.MultiTopicsConsumerImpl$$Lambda/0x0000000100c2b980.accept(Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniWhenComplete([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture.postComplete([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture.complete([email protected]/Unknown Source)
at
org.apache.pulsar.client.impl.ConsumerImpl.lambda$completehasMessageAvailableWithValue$65(ConsumerImpl.java:2574)
at
org.apache.pulsar.client.impl.ConsumerImpl$$Lambda/0x0000000100c35920.run(Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown
Source)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.runWith([email protected]/Unknown Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"pulsar-io-3-6":
at
java.util.concurrent.ConcurrentHashMap.compute([email protected]/Unknown Source)
- waiting to lock <0x00000000bf8b3b88> (a
java.util.concurrent.ConcurrentHashMap$Node)
at
org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService.lambda$getTopicPoliciesAsync$10(SystemTopicBasedTopicPoliciesService.java:271)
at
org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService$$Lambda/0x0000000100bbdac0.accept(Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniAcceptNow([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniAcceptStage([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture.thenAccept([email protected]/Unknown
Source)
at
org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService.getTopicPoliciesAsync(SystemTopicBasedTopicPoliciesService.java:271)
at
org.apache.pulsar.broker.service.BrokerService.getTopicPoliciesBypassSystemTopic(BrokerService.java:1186)
at
org.apache.pulsar.broker.service.BrokerService.getManagedLedgerConfig(BrokerService.java:1906)
at
org.apache.pulsar.broker.service.BrokerService.getManagedLedgerFactoryForTopic(BrokerService.java:1273)
at
org.apache.pulsar.broker.service.BrokerService.fetchTopicPropertiesAsync(BrokerService.java:1658)
at
org.apache.pulsar.broker.service.BrokerService.lambda$checkOwnershipAndCreatePersistentTopic$76(BrokerService.java:1694)
at
org.apache.pulsar.broker.service.BrokerService$$Lambda/0x00000001007b3220.accept(Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniAcceptNow([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniAcceptStage([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture.thenAccept([email protected]/Unknown
Source)
at
org.apache.pulsar.broker.service.BrokerService.checkOwnershipAndCreatePersistentTopic(BrokerService.java:1689)
at
org.apache.pulsar.broker.service.BrokerService.lambda$loadOrCreatePersistentTopic$66(BrokerService.java:1624)
at
org.apache.pulsar.broker.service.BrokerService$$Lambda/0x00000001007b2ba0.run(Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniRunNow([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniRunStage([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture.thenRun([email protected]/Unknown Source)
at
org.apache.pulsar.broker.service.BrokerService.loadOrCreatePersistentTopic(BrokerService.java:1620)
at
org.apache.pulsar.broker.service.BrokerService.lambda$getTopic$35(BrokerService.java:1111)
at
org.apache.pulsar.broker.service.BrokerService$$Lambda/0x0000000100c36e50.apply(Unknown
Source)
at
java.util.concurrent.ConcurrentHashMap.computeIfAbsent([email protected]/Unknown
Source)
- locked <0x00000000bdf00940> (a
java.util.concurrent.ConcurrentHashMap$ReservationNode)
at
org.apache.pulsar.broker.service.BrokerService.lambda$getTopic$36(BrokerService.java:1110)
at
org.apache.pulsar.broker.service.BrokerService$$Lambda/0x00000001007ada38.apply(Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniComposeStage([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture.thenCompose([email protected]/Unknown
Source)
at
org.apache.pulsar.broker.service.BrokerService.lambda$getTopic$37(BrokerService.java:1087)
at
org.apache.pulsar.broker.service.BrokerService$$Lambda/0x00000001007af710.apply(Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniComposeStage([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture.thenCompose([email protected]/Unknown
Source)
at
org.apache.pulsar.broker.service.BrokerService.getTopic(BrokerService.java:1076)
at
org.apache.pulsar.broker.service.BrokerService.getTopic(BrokerService.java:1040)
at
org.apache.pulsar.broker.service.BrokerService.getTopic(BrokerService.java:1035)
at
org.apache.pulsar.broker.service.ServerCnx.lambda$handleSubscribe$18(ServerCnx.java:1331)
at
org.apache.pulsar.broker.service.ServerCnx$$Lambda/0x0000000100ba4230.apply(Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniComposeStage([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture.thenCompose([email protected]/Unknown
Source)
at
org.apache.pulsar.broker.service.ServerCnx.lambda$handleSubscribe$24(ServerCnx.java:1330)
at
org.apache.pulsar.broker.service.ServerCnx$$Lambda/0x0000000100b9ec00.apply(Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniApplyNow([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniApplyStage([email protected]/Unknown
Source)
at
java.util.concurrent.CompletableFuture.thenApply([email protected]/Unknown
Source)
at
org.apache.pulsar.broker.service.ServerCnx.handleSubscribe(ServerCnx.java:1271)
at
org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:243)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at
io.netty.handler.flow.FlowControlHandler.dequeue(FlowControlHandler.java:202)
at
io.netty.handler.flow.FlowControlHandler.channelRead(FlowControlHandler.java:164)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:333)
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:455)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at
io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.runWith([email protected]/Unknown Source)
at java.lang.Thread.run([email protected]/Unknown Source)
```
### Modifications
- Move the callback logic out of lock scope.
### Verifying this change
- [x] Make sure that the change passes the CI checks.
### Documentation
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
