Hi Alex, I'm glad to see the problem is finally going away. Sorry that I can't track down this issue in Windows. And there are many possibilities to cause that behavior, might related to underlying file descriptors, the garbage collections, fork+exec issue. I don't have an answer.
In addition, the details you posted here is more than enough to file a bug report. It would be great if you can submit an issue in Github where we can revisit this issue when we got resources. https://github.com/grpc/grpc/issues/new Thanks, Lidi Zheng On Tue, Apr 2, 2019 at 3:38 AM Alejandro Villagrán <negral...@gmail.com> wrote: > Hi, > > The subprocess is finishing OK, almost immediately as it's not doing > anything apart from printing a message to the console. "subprocess.call" > returns a status code 0. So everything should be fine on that side. > Not 100% sure what you meant with tracing the lifecycle of the channel. I > called grpc.Channel.subscribe to print the different states of the channel > used: > > *def stateChange(conn):* > * print conn* > > *class GRPCStub(object):* > * def __init__(self):* > * self.channel = grpc.insecure_channel('127.0.0.1:50051 > <http://127.0.0.1:50051>')* > * self.channel.subscribe(stateChange)* > * self.stub = Test_pb2_grpc.GRPCTestStub(self.channel)* > > With that call to subscribe, the exception is no longer thrown. The > channel starts IDLE and then changes to READY as soon as the first grpc > call is made and stays like that for the rest of the program. > > It'd be interesting if you could reproduce it on your end to see what's > really happening under the bonnet. Why would that call to > grpc.Channel.subscribe will prevent the exception from being raised? > > Thanks, > Alex. > > On Mon, 1 Apr 2019 at 22:32, Lidi Zheng <li...@google.com> wrote: > >> Hi Alex, >> >> From the log, I found the error is located at client-side. The gRPC >> client sends reset stream frame with error code 2 indicating internal >> error, and its channel state somehow becomes SHUTDOWN. >> Also, I tried your code in Linux, it works fine without any error. I had >> another failed attempt to build the VS solution with 400+ errors... >> >> Although the root cause still remained unclear, the scope is reduced to >> the gRPC client behavior on Windows. To dig deeper into the bug, I think >> one have to trace if the "subprocess" finishes and the life cycle of >> channel. >> >> Lidi Zheng >> >> On Mon, Apr 1, 2019 at 7:55 AM Alejandro Villagrán <negral...@gmail.com> >> wrote: >> >>> Hi Lidi, >>> >>> I've set those environment variables you mentioned and I've attached the >>> log files (serverlog.txt and clientlog.txt). >>> Since I reported this issue, I've upgrade the Python version of gRPC, so >>> the attached log files were created with grpcio 1.19.0 >>> >>> Do they give you any hint as to what could be wrong? >>> >>> Thanks, >>> Alex. >>> >>> On Thu, 28 Mar 2019 at 17:07, Alejandro Villagrán <negral...@gmail.com> >>> wrote: >>> >>>> Hi Lidi, >>>> >>>> In the last version of the code I attached, I don't use CherryPy >>>> anymore so the issue cannot come from there. Eric said that fork handlers >>>> are not registered on Windows so he ruled out that option too. >>>> Unfortunately I haven't got access to a Linux machine and I'm not >>>> familiar enough with Linux to be able to recreate the issue there. >>>> >>>> I did use turn on the debug trace a few weeks ago but I couldn't see >>>> anything obvious. I'll find time to enable them again and share the logs >>>> with you. >>>> >>>> Thanks, >>>> Alex. >>>> >>>> On Thu, 28 Mar 2019 at 16:55, Lidi Zheng <li...@google.com> wrote: >>>> >>>>> My apologies Alex. There are other stuff keep come up and consumed my >>>>> time. I'm not a Windows expert, it will take me a long time to setup the >>>>> compilation environment for gRPC in Windows with debugger. And I failed to >>>>> find a Windows expert to debug your issue. >>>>> Eric has mentioned that the breakage can be caused by either fork >>>>> handlers registration or CherryPy. Do you think you can migrate the >>>>> reproduce case to Linux, if the root cause is the software? It would be >>>>> much easier to debug. >>>>> Also, have you tried to turn on the debug trace in gRPC by setting >>>>> environmental >>>>> variables >>>>> <https://github.com/grpc/grpc/blob/master/doc/environment_variables.md> >>>>> "*GRPC_VERBOSITY*" to "*DEBUG*", and "*GRPC_TRACE*" to " >>>>> *api,channel,connectivity_state*"? They might produce useful >>>>> information for us to identify the problem. >>>>> >>>>> Thanks, >>>>> Lidi Zheng >>>>> >>>>> On Thu, Mar 28, 2019 at 3:47 AM Alejandro Villagrán < >>>>> negral...@gmail.com> wrote: >>>>> >>>>>> Hi Lidi, >>>>>> >>>>>> Did you manage to reproduce the issue? >>>>>> >>>>>> Thanks, >>>>>> Alex. >>>>>> >>>>>> On Tue, 19 Mar 2019 at 01:06, Lidi Zheng <li...@google.com> wrote: >>>>>> >>>>>>> Hi Alex, >>>>>>> >>>>>>> Thank you for providing the reproduce code. I will spin up a Windows >>>>>>> machine to investigate this error. >>>>>>> If I'm able to find something useful, I'll let you know. >>>>>>> >>>>>>> Lidi Zheng >>>>>>> >>>>>>> On Mon, Mar 18, 2019 at 4:09 AM Alejandro Villagrán < >>>>>>> negral...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi Eric/Lidi, >>>>>>>> >>>>>>>> Yes, I'm running on Windows. I have now removed the CherryPy code >>>>>>>> and I still get the exception. >>>>>>>> >>>>>>>> Please follow these steps to reproduce the issue: >>>>>>>> - Unzip ReproduceGRPCIssue.zip >>>>>>>> - Go to the BreakGRPC folder and compile BreakGRPC.sln. Make sure >>>>>>>> BreakGRPC.exe is saved in BreakGRPC/x64/Release. >>>>>>>> - Go to the Services/src folder and open two command prompts there. >>>>>>>> - Run "python GRPCserver.py" in one command prompt. >>>>>>>> - Run "python GRPCclient.py" in the other command prompt. >>>>>>>> >>>>>>>> You should see the exception on the client command prompt. >>>>>>>> >>>>>>>> Please let me know if you are still unable to reproduce the issue >>>>>>>> with this version of the code. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Alex. >>>>>>>> >>>>>>>> On Mon, 4 Mar 2019 at 17:04, Eric Gribkoff <ericgribk...@google.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> +Lidi Zheng <li...@google.com>, who will be available for any >>>>>>>>> follow-up questions (it will be easier for him to notice your >>>>>>>>> questions if >>>>>>>>> you include his email address on the "to:" line) >>>>>>>>> >>>>>>>>> Hi Alex, >>>>>>>>> >>>>>>>>> Sorry for the delay. I was not able to reproduce the problem; it >>>>>>>>> looks like you are running on Windows, in which case gRPC's fork >>>>>>>>> handlers >>>>>>>>> are not registered/run, so those shouldn't be the cause here . Since >>>>>>>>> the >>>>>>>>> reproduction example also uses CherryPy websockets, it's quite >>>>>>>>> possible the >>>>>>>>> issue stems from that software rather than the gRPC stack - we'd >>>>>>>>> likely >>>>>>>>> need a reproduction case that only uses gRPC, without the websockets, >>>>>>>>> to be >>>>>>>>> able to help debug this further. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Eric >>>>>>>>> >>>>>>>>> On Mon, Mar 4, 2019 at 2:36 AM Alex <negral...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Eric, >>>>>>>>>> >>>>>>>>>> Just wondering if you had time to run my attached example and >>>>>>>>>> managed to reproduce the problem? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Alex. >>>>>>>>>> >>>>>>>>>> On Wednesday, February 20, 2019 at 7:04:51 PM UTC, Eric Gribkoff >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Can you post the code you're using to reproduce this error? If >>>>>>>>>>> you're using subprocess.Popen (or otherwise using fork+exec) to >>>>>>>>>>> start the >>>>>>>>>>> C++ grpc client process, the C++ client itself cannot be >>>>>>>>>>> interfering with >>>>>>>>>>> the Python process. Something could be going wrong in the gRPC core >>>>>>>>>>> fork >>>>>>>>>>> handlers, however - you can try running with the environment >>>>>>>>>>> variable >>>>>>>>>>> `GRPC_ENABLE_FORK_SUPPORT=0` to disable this feature and see if it >>>>>>>>>>> fixes >>>>>>>>>>> the issue. >>>>>>>>>>> >>>>>>>>>>> Also, in your step 5 you note that the C++ client isn't >>>>>>>>>>> communicating with the server. If you remove the fork+exec of a C++ >>>>>>>>>>> subprocess altogether, do you still see this intermittent exception >>>>>>>>>>> in the >>>>>>>>>>> Python client? >>>>>>>>>>> >>>>>>>>>>> Eric >>>>>>>>>>> >>>>>>>>>>> On Wed, Feb 20, 2019 at 6:57 AM Alex <negr...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> I should add that the Python client application which owns the >>>>>>>>>>>> Python grpc client is the one that runs the C++ grpc client as a >>>>>>>>>>>> subprocess >>>>>>>>>>>> in case that makes a difference. >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>> Google Groups "grpc.io" group. >>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>> it, send an email to grpc-io+u...@googlegroups.com. >>>>>>>>>>>> To post to this group, send email to grp...@googlegroups.com. >>>>>>>>>>>> Visit this group at https://groups.google.com/group/grpc-io. >>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>> https://groups.google.com/d/msgid/grpc-io/b323fac3-978b-47c1-b1fa-555c2f62b544%40googlegroups.com >>>>>>>>>>>> <https://groups.google.com/d/msgid/grpc-io/b323fac3-978b-47c1-b1fa-555c2f62b544%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>> . >>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "grpc.io" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to grpc-io+unsubscr...@googlegroups.com. >>>>>>>>>> To post to this group, send email to grpc-io@googlegroups.com. >>>>>>>>>> Visit this group at https://groups.google.com/group/grpc-io. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/grpc-io/af4d55fa-a5e0-4e3f-a5ad-9cb62378703d%40googlegroups.com >>>>>>>>>> <https://groups.google.com/d/msgid/grpc-io/af4d55fa-a5e0-4e3f-a5ad-9cb62378703d%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>> . >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To post to this group, send email to grpc-io@googlegroups.com. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAMC1%3DjeSdkRk_wW5Y%2BnjK3jjUt9-xi%2BxtkdE028kYggPO_M59A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
