yes, disabling the retry eliminates the problem. On Mon, Feb 26, 2024 at 10:57 AM tuk <subharaj.ma...@gmail.com> wrote:
> Thanks for replying. > > Just confirming for now the workaround is to disable the retry as > suggested in the linked GitHub issue. > > On Monday 26 February 2024 at 23:43:24 UTC+5:30 Larry Safran wrote: > >> There was an issue with a test in Google's internal repository, so this >> fix was temporarily backed out and is now targeted for 1.63. >> >> On Mon, Feb 26, 2024 at 9:45 AM 'Yuri Golobokov' via grpc.io < >> grp...@googlegroups.com> wrote: >> >>> Maybe you are hitting this bug: >>> https://github.com/grpc/grpc-java/issues/10838 >>> It is fixed and backported to 1.59 and later versions. >>> >>> On Monday, February 26, 2024 at 7:00:57 AM UTC-8 tuk wrote: >>> >>>> I am using gRPC 1.55.1 and observing an issue similar to the one >>>> discussed in the below GitHub issue >>>> >>>> - https://github.com/grpc/grpc-java/issues/9069 >>>> >>>> I have set the KeepAlive on both the client and server-side as >>>> suggested in the above issue >>>> >>>> I am creating the stub like below >>>> >>>> var channel = ManagedChannelBuilder.forAddress(network.getIp(), >>>> network.getPort()) >>>> .keepAliveTime(130, TimeUnit.SECONDS) >>>> .maxInboundMessageSize(maxInboundMessageSize) >>>> .maxInboundMetadataSize(maxInboundMetadataSize) >>>> .enableRetry() >>>> .build(); >>>> >>>> var stub = >>>> HelloServiceGrpc.newBlockingStub(channel).withDeadline(Deadline.after(115, >>>> TimeUnit.SECONDS)); >>>> stub.sayHello(); >>>> stub.sayHello(); >>>> >>>> In server side also keepAliveTime is set as suggested in the above >>>> github issue. >>>> >>>> Grpc.newServerBuilderForPort(port, InsecureServerCredentials.create()) >>>> .addService(new GreeterImpl()) >>>> .keepAliveTime(130, TimeUnit.SECONDS) >>>> .build() >>>> .start(); >>>> >>>> In my case, client calls server 1 then server 1 acts as a client to >>>> server 2. >>>> >>>> I am observing that when the deadline is exceeded in server2 the same >>>> deadline error is coming to server 1 like below which is also coming to >>>> client. It is working as expected. >>>> >>>> io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: context timed out >>>> at >>>> io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) >>>> at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) >>>> at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) >>>> >>>> >>>> But on some rare occasions I am observing the client thread hanging >>>> like the below and not getting the deadline exceeded error >>>> >>>> at jdk.internal.misc.Unsafe.park(java...@17.0.9/Native Method) >>>> - parking to wait for <0x0000000767a53a00> (a >>>> io.grpc.stub.ClientCalls$ThreadlessExecutor) >>>> at java.util.concurrent.locks.LockSupport.park(java...@17.0.9 >>>> /LockSupport.java:211) >>>> at >>>> io.grpc.stub.ClientCalls$ThreadlessExecutor.waitAndDrain(ClientCalls.java:748) >>>> at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:157) >>>> >>>> I waited for about 2 hours & it did not recover. The only way to >>>> recover from this is to restart the client application. It happened 3-4 >>>> times in the last couple of months for us. >>>> >>>> Can someone let me know >>>> >>>> 1. What I am doing wrong or is there any known issue in the >>>> grpc-java 1.55.1 that I am using? >>>> 2. Is there any timeout config I can set on the gRPC client side so >>>> that the client threads do not hang indefinitely? >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "grpc.io" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to grpc-io+u...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/grpc-io/d2700afd-762e-49ee-a15b-7f7901a6d10cn%40googlegroups.com >>> <https://groups.google.com/d/msgid/grpc-io/d2700afd-762e-49ee-a15b-7f7901a6d10cn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups " > grpc.io" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to grpc-io+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/grpc-io/80d5231c-d509-4c20-824b-34d71ee97f18n%40googlegroups.com > <https://groups.google.com/d/msgid/grpc-io/80d5231c-d509-4c20-824b-34d71ee97f18n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CANuT3qX_W8ou3wPw4MFyzdyegrAM8XVV3qXiOnb8ATgiZbW0cQ%40mail.gmail.com.
