keyboardbobo opened a new issue, #20185: URL: https://github.com/apache/pulsar/issues/20185
### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Version pulsar 2.9.2 ### Minimal reproduce step Start three consumer processes with the following commands: `nohup bin/pulsar-perf consume -n 50 --receiver-queue-size 100000 -ioThreads 10 -u pulsar://10.190.70.226:6650,10.190.70.227:6650,10.190.71.35:6650 -ss sub1 -st Shared persistent://chenjianbo-tenant/chenjianbo-ns/input-test1 & ` The broker stack shows that there are many pulsar-io threads blocked on the `ManagedLedgerImpl.asyncOpenCursor` method, The whole process will be stuck and cannot be stopped normally, and the service can only be closed through kill: ``` "pulsar-io-69-126" #959 prio=5 os_prio=0 tid=0x00007f7c8800a000 nid=0x5bbab waiting for monitor entry [0x00007f7a4b23d000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.asyncOpenCursor(ManagedLedgerImpl.java:886) - waiting to lock <0x00000003cbaad110> (a org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl) at org.apache.pulsar.broker.service.persistent.PersistentTopic.getDurableSubscription(PersistentTopic.java:867) at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$subscribe$19(PersistentTopic.java:753) at org.apache.pulsar.broker.service.persistent.PersistentTopic$$Lambda$1827/845333798.apply(Unknown Source) at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981) at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124) at org.apache.pulsar.broker.service.persistent.PersistentTopic.subscribe(PersistentTopic.java:680) at org.apache.pulsar.broker.service.ServerCnx.lambda$null$12(ServerCnx.java:1051) at org.apache.pulsar.broker.service.ServerCnx$$Lambda$1096/227211373.apply(Unknown Source) at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981) at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124) at org.apache.pulsar.broker.service.ServerCnx.lambda$handleSubscribe$15(ServerCnx.java:1021) at org.apache.pulsar.broker.service.ServerCnx$$Lambda$1094/1366070297.apply(Unknown Source) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:614) at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1983) at org.apache.pulsar.broker.service.ServerCnx.handleSubscribe(ServerCnx.java:969) at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:228) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.flow.FlowControlHandler.dequeue(FlowControlHandler.java:200) at io.netty.handler.flow.FlowControlHandler.channelRead(FlowControlHandler.java:162) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) ``` ### What did you expect to see? Regardless of how the client uses it, it should not affect service stability ### What did you see instead? Limiting the number of consumers, limiting consumption traffic and the number of unacks cannot effectively solve the problem ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
