And you are sure there is no proxy in between the client and server? One thing that can cause this is data integrity issues, caused by networking hardware. To protect against this, we always suggest using TLS (SSL).
I know this is more effort, but it would help indicate if there is errant networking hardware. Can you try using TLS client and server side (using the certs from our HelloWorldTls example), and see if the problem persists? On Tuesday, September 18, 2018 at 5:51:55 PM UTC-7, Anthony Corbacho wrote: > > Yes, both are written in Java and I am using plaintext. > > On Tuesday, September 18, 2018 at 7:58:16 PM UTC-4, Carl Mastrangelo wrote: >> >> Are both your client and server written using Java? Also, are you using >> TLS or plaintext? >> >> On Monday, September 17, 2018 at 8:21:30 PM UTC-7, Anthony Corbacho wrote: >>> >>> Hi, >>> >>> This is strange I have enabled the same log but I only see RST_STREAM. >>> Do I need to do something else? >>> >>> # GRPC debugging >>> log4j.logger.io.grpc.netty.NettyServerHandler=ALL >>> log4j.logger.io.grpc.netty.NettyClientHandler =ALL >>> >>> >>> >>> On Monday, September 17, 2018 at 4:52:02 PM UTC-4, Carl Mastrangelo >>> wrote: >>>> >>>> Here's what i use to turn it on: >>>> https://gist.github.com/carl-mastrangelo/49f6d6a8ff29200fcb7d9e25e473b2d0 >>>> >>>> On Monday, September 17, 2018 at 11:39:47 AM UTC-7, Anthony Corbacho >>>> wrote: >>>>> >>>>> Hi Carl, >>>>> >>>>> Thanks for the fast answer. >>>>> How can I enable `netty debug log frame that's for DATA`? >>>>> >>>>> thanks~. >>>>> >>>>> On Monday, September 17, 2018 at 1:38:20 PM UTC-4, Carl Mastrangelo >>>>> wrote: >>>>>> >>>>>> You should look for a netty debuglog frame that's for DATA, not >>>>>> RST_STREAM. That should show you the corrupted message. >>>>>> >>>>>> There are also some hooks into the core gRPC library that (while more >>>>>> complicated) will let you examine the message bytes. By using a custom >>>>>> Marshaller, you can peak at the bytes and then delegate the remaining >>>>>> message to the protobuf Marshaller. You can see how to wire up a >>>>>> Marshaller by looking in the generated code for the MethodDescriptor. >>>>>> >>>>>> On Sunday, September 16, 2018 at 2:38:26 PM UTC-7, Anthony Corbacho >>>>>> wrote: >>>>>>> >>>>>>> Hello, >>>>>>> I am new to Grpc and so far like it very much. >>>>>>> >>>>>>> I am using a bidirectional stream and from time to time I get an >>>>>>> exception like this one: >>>>>>> >>>>>>> io.grpc.StatusRuntimeException: CANCELLED: Failed to read message. >>>>>>> at io.grpc.Status.asRuntimeException(Status.java:526) at >>>>>>> io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:418) >>>>>>> >>>>>>> at >>>>>>> io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:41) >>>>>>> >>>>>>> at >>>>>>> io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:663) >>>>>>> >>>>>>> at >>>>>>> io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:41) >>>>>>> >>>>>>> at >>>>>>> io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:392) >>>>>>> >>>>>>> at >>>>>>> io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:443) >>>>>>> at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63) >>>>>>> at >>>>>>> io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:525) >>>>>>> >>>>>>> at >>>>>>> io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:446) >>>>>>> >>>>>>> at >>>>>>> io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:510) >>>>>>> >>>>>>> at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at >>>>>>> io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) >>>>>>> at >>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>>>>> >>>>>>> at >>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>>>>> >>>>>>> at java.lang.Thread.run(Thread.java:748) Caused by: >>>>>>> io.grpc.StatusRuntimeException: INTERNAL: Invalid protobuf byte >>>>>>> sequence at >>>>>>> io.grpc.Status.asRuntimeException(Status.java:517) at >>>>>>> io.grpc.protobuf.lite.ProtoLiteUtils$2.parse(ProtoLiteUtils.java:168) >>>>>>> at >>>>>>> io.grpc.protobuf.lite.ProtoLiteUtils$2.parse(ProtoLiteUtils.java:82) at >>>>>>> io.grpc.MethodDescriptor.parseResponse(MethodDescriptor.java:265) at >>>>>>> io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:498) >>>>>>> >>>>>>> ... 5 more Caused by: >>>>>>> com.google.protobuf.InvalidProtocolBufferException: >>>>>>> Protocol message contained an invalid tag (zero). at >>>>>>> com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:105) >>>>>>> >>>>>>> at >>>>>>> com.google.protobuf.CodedInputStream$ArrayDecoder.readTag(CodedInputStream.java:646) >>>>>>> >>>>>>> at >>>>>>> com.zepl.notebook.service.grpc.NotebookResponse.<init>(NotebookResponse.java:46) >>>>>>> >>>>>>> at >>>>>>> com.zepl.notebook.service.grpc.NotebookResponse.<init>(NotebookResponse.java:13) >>>>>>> >>>>>>> at >>>>>>> com.zepl.notebook.service.grpc.NotebookResponse$1.parsePartialFrom(NotebookResponse.java:2851) >>>>>>> >>>>>>> at >>>>>>> com.zepl.notebook.service.grpc.NotebookResponse$1.parsePartialFrom(NotebookResponse.java:2846) >>>>>>> >>>>>>> at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:91) >>>>>>> at >>>>>>> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49) at >>>>>>> io.grpc.protobuf.lite.ProtoLiteUtils$2.parseFrom(ProtoLiteUtils.java:173) >>>>>>> >>>>>>> at >>>>>>> io.grpc.protobuf.lite.ProtoLiteUtils$2.parse(ProtoLiteUtils.java:165) >>>>>>> ... 8 more >>>>>>> >>>>>>> I enabled netty debug logs, and I have this line: [id: 0xfc5978c0, >>>>>>> L:/100.119.42.167:39090 - R:--/--] OUTBOUND RST_STREAM: >>>>>>> streamId=1539 errorCode=8. >>>>>>> I dont really get what is wrong, I get this error once in a while >>>>>>> and I am stuck. >>>>>>> I am calling the observers from different threads in the server >>>>>>> side, do I need to synchronize the method that calls the observers? >>>>>>> >>>>>>> Thank you for your help and time. >>>>>>> >>>>>> -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/0cedb419-4bbd-43e0-a072-722fc0ce5b6a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
