All the methods should be overriden. The idle connection time is likely that timeout.
On Monday, October 30, 2017 at 2:00:59 PM UTC-7, [email protected] wrote: > > Some findings to share, not sure if related though: > 1. we use custom grpc name resolver, and refresh() method is not > overridden (default is no-op) > 2. in the past occurrences, before DEADLINE_EXCEEDED starts happening, > there is ALWAYS a 30 minutes gap between the last succeeded and the second > last succeeded grpc from the broken client. > > 30 minutes seems to be the default connection idle timeout, which is what > we use. Not sure if it is related though. > > On Thursday, October 19, 2017 at 11:09:26 AM UTC-7, Bi Ran wrote: >> >> The client automatically recovered after a few hours. >> This is the only thread that is relevant. >> I did not explicitly set waitForReady, and I think by default it is >> disabled. >> I can provide more thread dumps when it happen again next time. Let me >> know if any information other than thread dump is needed. >> >> >> On Thu, Oct 19, 2017 at 10:31 AM 'Carl Mastrangelo' via grpc.io < >> [email protected] <javascript:>> wrote: >> >>> Is that the only thread? Also, are you using waitForReady? >>> >>> >>> On Wednesday, October 18, 2017 at 3:40:19 PM UTC-7, [email protected] >>> wrote: >>>> >>>> The problem happened to me again even with keep-alive. >>>> netstats suggests that underlying tcp connection is established. >>>> Client thread dump follows: >>>> java.lang.Thread.State: WAITING (parking) >>>> at sun.misc.Unsafe.park(Native Method) >>>> - parking to wait for <0x00007f1624000038> (a >>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) >>>> at >>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) >>>> at >>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) >>>> at >>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) >>>> at >>>> io.grpc.stub.ClientCalls$ThreadlessExecutor.waitAndDrain(ClientCalls.java:572) >>>> at >>>> io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:120) >>>> >>>> All server threads are idle. It is likely server receives no request. >>>> >>>> >>>> On Wednesday, October 11, 2017 at 7:57:28 AM UTC-7, Taehyun Park wrote: >>>>> >>>>> This is what I did to avoid this problem in production. I wrapped all >>>>> grpc calls with RxJava and used Retry to re-initialize channel when >>>>> DEADLINE_EXCEEDED is thrown. >>>>> >>>>> >>>>> >>>>> On Tuesday, September 26, 2017 at 7:19:14 AM UTC+9, [email protected] >>>>> wrote: >>>>>> >>>>>> Language: java >>>>>> Version: 1.5 >>>>>> >>>>>> I ran into weird issue multiple times recently: all RPCs from one >>>>>> client failed with DEADLINE_EXCEEDED. From server log, it looks like >>>>>> these >>>>>> failed requests didn't arrive at server at all. Other GRPC clients >>>>>> worked >>>>>> fine during that time. The issue was fixed by restarting client >>>>>> application. >>>>>> Keep-alive feature is not used in client. From my understanding, >>>>>> client channel should manage the underlying connection properly even >>>>>> keep-alive is off. >>>>>> This issue happens occasionally and I haven't find a reliable way to >>>>>> reproduce. >>>>>> >>>>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "grpc.io" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/grpc-io/0t9n1-2GTms/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected] <javascript:>. >>> To post to this group, send email to [email protected] >>> <javascript:>. >>> Visit this group at https://groups.google.com/group/grpc-io. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/grpc-io/178d7d15-fc27-4c0a-b53d-39f48fc6f9f9%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/grpc-io/178d7d15-fc27-4c0a-b53d-39f48fc6f9f9%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/6608a54d-9292-45e6-8ec0-a4dd98fc57cd%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
