oh.. I didn't realize you were doing a fork() call. grpc actually does not support fork and is known to create strange issues like the one you reported. We did some work to mitigate some specific uses of fork ( https://github.com/grpc/grpc/pull/13025) but by and large it is not it is not supported.
Moving "import grpc" after forking seems like a great idea. thanks, Sree On Tue, Jan 9, 2018 at 4:45 PM, Giang Nguyen <[email protected]> wrote: > > > On Tue, Jan 9, 2018 at 2:33 PM, Sree Kuchibhotla <[email protected]> wrote: > >> Hi, >> >> This looks like a bug in the code. >> >> - Does this failure happen consistently ? >> > - Could you give more details on your test that is causing this? (would >> be ideal if I can create something similar) >> >> Also, would it be possible for you to rerun your tests by replacing the >> assert with the following log line ? >> (I am interested in seeing what is the state the root and root->next >> workers are) >> >> if (next_worker->state != KICKED) { >> gpr_log(GPR_ERROR, "root: %p (%d), next: %p (%d)", root_worker, >> root_worker->state, next_worker, next_worker->state); } >> > > Hi Sree, > > Yea, kind of 100% failure rate. We believe it's related to > https://github.com/grpc/grpc/issues/13873. Moving "import grpc" after > fork'ing seems to resolve the issue for us. > > Unfortunately it's non-trivial to run the tests against a modified grpc > because our environment is bazelized and we have not been able to build > grpc in our bazel environment. > > Thanks, > > > >> >> thanks, >> Sree >> >> >> On Mon, Jan 8, 2018 at 4:51 PM, <[email protected]> wrote: >> >>> Hi, >>> >>> We are trying to upgrade to grpc 1.8.3 from 1.6 and seeing an issue with >>> one of our internal tests that works in 1.6 but fails in 1.8.3, where we're >>> seeing a bunch of these message: >>> >>> E0108 23:10:24.009205427 68294 ev_epoll1_linux.cc:1098] assertion >>> failed: next_worker->state == KICKED >>> E0108 23:10:24.011880853 68330 ev_epoll1_linux.cc:1098] assertion >>> failed: next_worker->state == KICKED >>> E0108 23:10:24.011978910 68369 ev_epoll1_linux.cc:1098] assertion >>> failed: next_worker->state == KICKED >>> >>> >>> Looking at the code in branch 1.8.x [1], I see these two lines: >>> >>> >>> 1098: GPR_ASSERT(next_worker->state == KICKED); >>> >>> 1099: SET_KICK_STATE(next_worker, KICKED); >>> >>> >>> I'm curious why the code is asserting X right before it's making X true? It >>> makes me look further and I see: >>> >>> 1044 } else if (next_worker->state == KICKED) { >>> >>> ... >>> >>> goto done; >>> ... >>> 1063 } else if (next_worker->state == UNKICKED) { >>> >>> ... >>> >>> goto done; >>> >>> ... >>> >>> 1072 } else if (next_worker->state == DESIGNATED_POLLER) { >>> >>> ... >>> >>> goto done; >>> >>> } else { >>> >>> 1098 GPR_ASSERT(next_worker->state == KICKED); >>> >>> >>> that seems to be a bug somewhere, since if we reach line 1098, then >>> next_worker->state cannot be KICKED? >>> >>> Ideas? >>> >>> Thanks! >>> >>> [1] https://github.com/grpc/grpc/blob/v1.8.x/src/core/lib/iomgr/ >>> ev_epoll1_linux.cc >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "grpc.io" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/grpc-io. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/grpc-io/11b1743c-1f45-4f6c-b983-64559b36d040%40googlegroups.com >>> <https://groups.google.com/d/msgid/grpc-io/11b1743c-1f45-4f6c-b983-64559b36d040%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CALRi9Qd5MvzeLOOt7oiCj12mh-zDBPL_1zZo%2B2_TNue7hsgDrQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
