Another thing I forgot to mention is, the issue starts happening after periodic health-check grpc call is removed, which makes me feel it is related to internal connection management.
On Wednesday, September 27, 2017 at 4:20:03 AM UTC-7, Jorg Heymans wrote: > > Happy to see this being posted. We are running into this as well, it > happens once every couple of weeks we have no idea why. Indeed server side > we don't see anything in the logs it's as if nothing is going on. When it > happens we restart the services though, not the client application and it > makes the problem go away. The client does not seem to be able to recover > from it. It seems to be an issue only in production, presumably because the > client applications and services do not get restarted as much as in test ? > > We have a zookeeper based NameResolver, client deadlines are set to 10 > seconds. Are you suggesting we should up this > 1 minute to get around any > resolution failures ? > > Thanks, > Jorg > > On Tuesday, September 26, 2017 at 12:19:14 AM UTC+2, [email protected] > wrote: >> >> Language: java >> Version: 1.5 >> >> I ran into weird issue multiple times recently: all RPCs from one client >> failed with DEADLINE_EXCEEDED. From server log, it looks like these failed >> requests didn't arrive at server at all. Other GRPC clients worked fine >> during that time. The issue was fixed by restarting client application. >> Keep-alive feature is not used in client. From my understanding, client >> channel should manage the underlying connection properly even keep-alive is >> off. >> This issue happens occasionally and I haven't find a reliable way to >> reproduce. >> > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/cc1fc25b-012d-446a-9513-c8bec5284bd0%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
