pkumar-singh opened a new pull request #8351: URL: https://github.com/apache/pulsar/pull/8351
Motivation Current version of gRPC is deadlocks as explained below. This deadlock is currently only appearing in state store but might affect pulsar in general as well. Found one Java-level deadlock: ============================= "io-read-scheduler-OrderedScheduler-0-0": waiting to lock monitor 0x00007f10e804e100 (object 0x00000000e3b8e0a8, a io.grpc.inprocess.InProcessTransport$InProcessStream$InProcessClientStream), which is held by "grpc-default-executor-17" "grpc-default-executor-17": waiting to lock monitor 0x00007f107000ca00 (object 0x00000000e3b8e1d8, a io.grpc.inprocess.InProcessTransport$InProcessStream$InProcessServerStream), which is held by "io-read-scheduler-OrderedScheduler-0-0" Java stack information for the threads listed above: =================================================== "io-read-scheduler-OrderedScheduler-0-0": at io.grpc.inprocess.InProcessTransport$InProcessStream$InProcessClientStream.request(InProcessTransport.java:639) waiting to lock <0x00000000e3b8e0a8> (a io.grpc.inprocess.InProcessTransport$InProcessStream$InProcessClientStream) at io.grpc.internal.ForwardingClientStream.request(ForwardingClientStream.java:32) at io.grpc.internal.ClientCallImpl.request(ClientCallImpl.java:369) at io.grpc.PartialForwardingClientCall.request(PartialForwardingClientCall.java:34) at io.grpc.ForwardingClientCall.request(ForwardingClientCall.java:22) at io.grpc.ForwardingClientCall$SimpleForwardingClientCall.request(ForwardingClientCall.java:44) at io.grpc.PartialForwardingClientCall.request(PartialForwardingClientCall.java:34) at io.grpc.ForwardingClientCall.request(ForwardingClientCall.java:22) at io.grpc.ForwardingClientCall$SimpleForwardingClientCall.request(ForwardingClientCall.java:44) at io.grpc.PartialForwardingClientCall.request(PartialForwardingClientCall.java:34) at io.grpc.ForwardingClientCall.request(ForwardingClientCall.java:22) at io.grpc.ForwardingClientCall$SimpleForwardingClientCall.request(ForwardingClientCall.java:44) at org.apache.bookkeeper.common.grpc.proxy.ProxyCall$ResponseProxy.onMessage(ProxyCall.java:112) locked <0x00000000e3b8dfd8> (a org.apache.bookkeeper.common.grpc.proxy.ProxyCall$ResponseProxy) at io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33) at io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:519) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializeReentrantCallsDirectExecutor.execute(SerializeReentrantCallsDirectExecutor.java:49) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.messagesAvailable(ClientCallImpl.java:536) at io.grpc.internal.ForwardingClientStreamListener.messagesAvailable(ForwardingClientStreamListener.java:44) at io.grpc.inprocess.InProcessTransport$InProcessStream$InProcessServerStream.writeMessage(InProcessTransport.java:455) locked <0x00000000e3b8e1d8> (a io.grpc.inprocess.InProcessTransport$InProcessStream$InProcessServerStream) at io.grpc.internal.ServerCallImpl.sendMessage(ServerCallImpl.java:139) at io.grpc.ForwardingServerCall.sendMessage(ForwardingServerCall.java:32) at org.apache.bookkeeper.common.grpc.stats.MonitoringServerCall.sendMessage(MonitoringServerCall.java:47) at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onNext(ServerCalls.java:344) at org.apache.bookkeeper.stream.storage.impl.grpc.handler.ResponseHandler.accept(ResponseHandler.java:49) at org.apache.bookkeeper.stream.storage.impl.grpc.handler.ResponseHandler.accept(ResponseHandler.java:29) at java.util.concurrent.CompletableFuture.uniWhenComplete([email protected]/CompletableFuture.java:859) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire([email protected]/CompletableFuture.java:837) at java.util.concurrent.CompletableFuture.postComplete([email protected]/CompletableFuture.java:506) at java.util.concurrent.CompletableFuture.complete([email protected]/CompletableFuture.java:2073) at org.apache.bookkeeper.statelib.impl.journal.AbstractStateStoreWithJournal.lambda$executeIO$16(AbstractStateStoreWithJournal.java:472) at org.apache.bookkeeper.statelib.impl.journal.AbstractStateStoreWithJournal$$Lambda$294/0x00000008404b7040.run(Unknown Source) at java.util.concurrent.Executors$RunnableAdapter.call([email protected]/Executors.java:515) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.Executors$RunnableAdapter.call([email protected]/Executors.java:515) at java.util.concurrent.FutureTask.run([email protected]/FutureTask.java:264) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run([email protected]/ScheduledThreadPoolExecutor.java:304) at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run([email protected]/Thread.java:834) "grpc-default-executor-17": at io.grpc.inprocess.InProcessTransport$InProcessStream$InProcessServerStream.isReady(InProcessTransport.java:466) waiting to lock <0x00000000e3b8e1d8> (a io.grpc.inprocess.InProcessTransport$InProcessStream$InProcessServerStream) at io.grpc.internal.ServerCallImpl.isReady(ServerCallImpl.java:167) at io.grpc.PartialForwardingServerCall.isReady(PartialForwardingServerCall.java:43) at io.grpc.ForwardingServerCall.isReady(ForwardingServerCall.java:22) at io.grpc.ForwardingServerCall$SimpleForwardingServerCall.isReady(ForwardingServerCall.java:39) at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:173) at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35) at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23) at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializeReentrantCallsDirectExecutor.execute(SerializeReentrantCallsDirectExecutor.java:49) at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener.halfClosed(ServerImpl.java:722) at io.grpc.inprocess.InProcessTransport$InProcessStream$InProcessClientStream.halfClose(InProcessTransport.java:745) locked <0x00000000e3b8e0a8> (a io.grpc.inprocess.InProcessTransport$InProcessStream$InProcessClientStream) at io.grpc.internal.ForwardingClientStream.halfClose(ForwardingClientStream.java:67) at io.grpc.internal.ClientCallImpl.halfClose(ClientCallImpl.java:408) at io.grpc.PartialForwardingClientCall.halfClose(PartialForwardingClientCall.java:44) at io.grpc.ForwardingClientCall.halfClose(ForwardingClientCall.java:22) at io.grpc.ForwardingClientCall$SimpleForwardingClientCall.halfClose(ForwardingClientCall.java:44) at io.grpc.PartialForwardingClientCall.halfClose(PartialForwardingClientCall.java:44) at io.grpc.ForwardingClientCall.halfClose(ForwardingClientCall.java:22) at io.grpc.ForwardingClientCall$SimpleForwardingClientCall.halfClose(ForwardingClientCall.java:44) at io.grpc.PartialForwardingClientCall.halfClose(PartialForwardingClientCall.java:44) at io.grpc.ForwardingClientCall.halfClose(ForwardingClientCall.java:22) at io.grpc.ForwardingClientCall$SimpleForwardingClientCall.halfClose(ForwardingClientCall.java:44) at org.apache.bookkeeper.common.grpc.proxy.ProxyCall$RequestProxy.onHalfClose(ProxyCall.java:68) at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628) at java.lang.Thread.run([email protected]/Thread.java:834) Found 1 deadlock. Solution Verified that with gRPC-1.31 issue is fixed. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
