ThomasTaketurns opened a new issue, #19784: URL: https://github.com/apache/pulsar/issues/19784
### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Version The broker version is https://pulsar.apache.org/release-notes/versioned/pulsar-2.10.1/ And the client is : https://javadoc.io/doc/org.apache.pulsar/pulsar-client-api/2.10.3/index.html ### Minimal reproduce step We are currently facing what seems to be a deadlock issue on our testing environment. We are working with the Java SDK. In my thread dump I can see 24 threads stuck on this line : final Transaction txn = this.getClient().newTransaction().withTransactionTimeout(1, TimeUnit.MINUTES).build().get(); Where getClient returns an instance of PulsarClient. at jdk.internal.misc.Unsafe.park([email protected]/Native Method) - parking to wait for <0x00000000ad73c838> (a java.util.concurrent.CompletableFuture$Signaller) at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:211) at java.util.concurrent.CompletableFuture$Signaller.block([email protected]/CompletableFuture.java:1864) at java.util.concurrent.ForkJoinPool.unmanagedBlock([email protected]/ForkJoinPool.java:3463) at java.util.concurrent.ForkJoinPool.managedBlock([email protected]/ForkJoinPool.java:3434) at java.util.concurrent.CompletableFuture.waitingGet([email protected]/CompletableFuture.java:1898) at java.util.concurrent.CompletableFuture.get([email protected]/CompletableFuture.java:2072) at [com.taketurns.es](http://com.taketurns.es/).pulsar.controller.EntityPulsarController.publishEvents(EntityPulsarController.java:75) Those threads are waiting for a CompletableFuture that never complete. I can also see that pulsar-timer-9-1 thread is blocked. java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.pulsar.client.impl.ProducerImpl.run(ProducerImpl.java:1876) - waiting to lock <0x00000000ad6fd530> (a org.apache.pulsar.client.impl.ProducerImpl) at [org.apache.pulsar.shade.io](http://org.apache.pulsar.shade.io/).netty.util.HashedWheelTimer$HashedWheelTimeout.run(HashedWheelTimer.java:715) at [org.apache.pulsar.shade.io](http://org.apache.pulsar.shade.io/).netty.util.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:34) at [org.apache.pulsar.shade.io](http://org.apache.pulsar.shade.io/).netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:703) at [org.apache.pulsar.shade.io](http://org.apache.pulsar.shade.io/).netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:790) at [org.apache.pulsar.shade.io](http://org.apache.pulsar.shade.io/).netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:503) at [org.apache.pulsar.shade.io](http://org.apache.pulsar.shade.io/).netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run([email protected]/Thread.java:833) Locked ownable synchronizers: - None And the thread holding the lock on <0x00000000ad6fd530> is the following. pulsar-client-io-1-1 Stack Trace is: java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park([email protected]/Native Method) - parking to wait for <0x00000000ad6fd400> (a java.util.concurrent.CompletableFuture$Signaller) at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:211) at java.util.concurrent.CompletableFuture$Signaller.block([email protected]/CompletableFuture.java:1864) at java.util.concurrent.ForkJoinPool.unmanagedBlock([email protected]/ForkJoinPool.java:3463) at java.util.concurrent.ForkJoinPool.managedBlock([email protected]/ForkJoinPool.java:3434) at java.util.concurrent.CompletableFuture.waitingGet([email protected]/CompletableFuture.java:1898) at java.util.concurrent.CompletableFuture.get([email protected]/CompletableFuture.java:2072) at org.apache.pulsar.client.impl.schema.AbstractStructSchema.atSchemaVersion(AbstractStructSchema.java:100) at org.apache.pulsar.client.impl.MessageImpl.getReaderSchema(MessageImpl.java:400) at org.apache.pulsar.client.impl.ConsumerImpl.lambda$doReconsumeLater$11(ConsumerImpl.java:657) at org.apache.pulsar.client.impl.ConsumerImpl$$Lambda$4185/0x0000000801ee43e0.accept(Unknown Source) at java.util.concurrent.CompletableFuture$UniAccept.tryFire([email protected]/CompletableFuture.java:718) at java.util.concurrent.CompletableFuture.postComplete([email protected]/CompletableFuture.java:510) at java.util.concurrent.CompletableFuture.complete([email protected]/CompletableFuture.java:2147) at org.apache.pulsar.client.impl.ProducerImpl.lambda$resendMessages$18(ProducerImpl.java:1795) - locked <0x00000000ad6fd530> (a org.apache.pulsar.client.impl.ProducerImpl) at org.apache.pulsar.client.impl.ProducerImpl$$Lambda$2248/0x0000000801b24440.run(Unknown Source) at [org.apache.pulsar.shade.io](http://org.apache.pulsar.shade.io/).netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) at [org.apache.pulsar.shade.io](http://org.apache.pulsar.shade.io/).netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) at [org.apache.pulsar.shade.io](http://org.apache.pulsar.shade.io/).netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) at [org.apache.pulsar.shade.io](http://org.apache.pulsar.shade.io/).netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:403) at [org.apache.pulsar.shade.io](http://org.apache.pulsar.shade.io/).netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at [org.apache.pulsar.shade.io](http://org.apache.pulsar.shade.io/).netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at [org.apache.pulsar.shade.io](http://org.apache.pulsar.shade.io/).netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run([email protected]/Thread.java:833) Locked ownable synchronizers: Which is also waiting for a CompletableFuture that never completes. ### What did you expect to see? No deadlock ### What did you see instead? A deadlock ### Anything else? Here is the thread dump. Please tell me if you would need anything else. [dump-2023-03-08.txt](https://github.com/apache/pulsar/files/10942445/dump-2023-03-08.txt) Thanks for your help, Thomas ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
