Shawyeok opened a new issue, #15131:
URL: https://github.com/apache/pulsar/issues/15131
**To Reproduce**
Steps to reproduce the behavior:
1. setup a remote cluster for a namespace
2. delete the remote cluster from cluster list and keep namespace unchanged
3. start a consumer or producer for a new topic which under the namespace
above
4. See log `Failed to subscribe to topic on xxx` and the deadlock showed up
**Describe the bug**
I reproduced with current master HEAD version, it may also reproduce with
other code branch e.g. checkDeduplicationStatus fail
When create a persistent topic, it first acquire a w lock under the hood
`topics.computeIfAbsent`, and then try acquire another w lock to remove the
corresponding entry from the `topics` map if create topic fails.
Full threaddump
```java
"pulsar-io-18-11" #209 prio=5 os_prio=31 cpu=14.73ms elapsed=5841.59s
tid=0x00007fed20af8000 nid=0x33f03 waiting on condition [0x0000700017bfd000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000007818e4cd8> (a
org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section)
at
java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:194)
at
java.util.concurrent.locks.StampedLock.acquireWrite([email protected]/StampedLock.java:1315)
at
java.util.concurrent.locks.StampedLock.writeLock([email protected]/StampedLock.java:461)
// wait to acquire another w lock here
at
org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.remove(ConcurrentOpenHashMap.java:436)
at
org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.access$300(ConcurrentOpenHashMap.java:293)
at
org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.remove(ConcurrentOpenHashMap.java:251)
at
org.apache.pulsar.broker.service.BrokerService$2.lambda$openLedgerComplete$5(BrokerService.java:1447)
at
org.apache.pulsar.broker.service.BrokerService$2$$Lambda$1415/0x000000080095d040.accept(Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniWhenComplete([email protected]/CompletableFuture.java:859)
at
java.util.concurrent.CompletableFuture.uniWhenCompleteStage([email protected]/CompletableFuture.java:883)
at
java.util.concurrent.CompletableFuture.whenComplete([email protected]/CompletableFuture.java:2251)
at
org.apache.pulsar.broker.service.BrokerService$2.lambda$openLedgerComplete$6(BrokerService.java:1446)
at
org.apache.pulsar.broker.service.BrokerService$2$$Lambda$1068/0x000000080087cc40.apply(Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniExceptionally([email protected]/CompletableFuture.java:986)
at
java.util.concurrent.CompletableFuture.uniExceptionallyStage([email protected]/CompletableFuture.java:1004)
at
java.util.concurrent.CompletableFuture.exceptionally([email protected]/CompletableFuture.java:2307)
at
org.apache.pulsar.broker.service.BrokerService$2.openLedgerComplete(BrokerService.java:1443)
at
org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.lambda$asyncOpen$7(ManagedLedgerFactoryImpl.java:426)
at
org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl$$Lambda$909/0x0000000800818040.accept(Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniAcceptNow([email protected]/CompletableFuture.java:753)
at
java.util.concurrent.CompletableFuture.uniAcceptStage([email protected]/CompletableFuture.java:731)
at
java.util.concurrent.CompletableFuture.thenAccept([email protected]/CompletableFuture.java:2108)
at
org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.asyncOpen(ManagedLedgerFactoryImpl.java:426)
at
org.apache.pulsar.broker.service.BrokerService.lambda$createPersistentTopic$55(BrokerService.java:1410)
at
org.apache.pulsar.broker.service.BrokerService$$Lambda$890/0x0000000800801840.accept(Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniAcceptNow([email protected]/CompletableFuture.java:753)
at
java.util.concurrent.CompletableFuture.uniAcceptStage([email protected]/CompletableFuture.java:731)
at
java.util.concurrent.CompletableFuture.thenAccept([email protected]/CompletableFuture.java:2108)
at
org.apache.pulsar.broker.service.BrokerService.createPersistentTopic(BrokerService.java:1391)
at
org.apache.pulsar.broker.service.BrokerService.lambda$loadOrCreatePersistentTopic$49(BrokerService.java:1343)
at
org.apache.pulsar.broker.service.BrokerService$$Lambda$882/0x00000008007ff440.run(Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniRunNow([email protected]/CompletableFuture.java:815)
at
java.util.concurrent.CompletableFuture.uniRunStage([email protected]/CompletableFuture.java:799)
at
java.util.concurrent.CompletableFuture.thenRun([email protected]/CompletableFuture.java:2121)
at
org.apache.pulsar.broker.service.BrokerService.loadOrCreatePersistentTopic(BrokerService.java:1339)
at
org.apache.pulsar.broker.service.BrokerService.lambda$getTopic$23(BrokerService.java:982)
at
org.apache.pulsar.broker.service.BrokerService$$Lambda$877/0x00000008007fdc40.apply(Unknown
Source)
at
org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.put(ConcurrentOpenHashMap.java:404)
// acquired a w lock here
at
org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.computeIfAbsent(ConcurrentOpenHashMap.java:238)
at
org.apache.pulsar.broker.service.BrokerService.getTopic(BrokerService.java:981)
at
org.apache.pulsar.broker.service.BrokerService.getTopic(BrokerService.java:946)
at
org.apache.pulsar.broker.service.ServerCnx.lambda$handleSubscribe$15(ServerCnx.java:1026)
at
org.apache.pulsar.broker.service.ServerCnx$$Lambda$893/0x0000000800807840.apply(Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniApplyNow([email protected]/CompletableFuture.java:680)
at
java.util.concurrent.CompletableFuture.uniApplyStage([email protected]/CompletableFuture.java:658)
at
java.util.concurrent.CompletableFuture.thenApply([email protected]/CompletableFuture.java:2094)
at
org.apache.pulsar.broker.service.ServerCnx.handleSubscribe(ServerCnx.java:977)
at
org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:229)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.handler.flow.FlowControlHandler.dequeue(FlowControlHandler.java:200)
at
io.netty.handler.flow.FlowControlHandler.channelRead(FlowControlHandler.java:162)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run([email protected]/Thread.java:834)
Locked ownable synchronizers:
- None
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
