Hi Steven, It seems the original post has been deleted. Was the issue resolved?
On Fri, Dec 23, 2016 at 12:52 PM, Steven Parkes <[email protected]> wrote: > We're getting a lot of "Invalid protobuf byte sequence" errors over a bidi > streaming service call. > > The server is grpc-java and the client go. > > The message that seems to be causing the issue is a relatively large > message of repeated maps. One example is around 400kB. > > It doesn't look like it's deterministically repeatable. > > We seem to be getting the "invalid" message on both sides, though it's a > little hard (for me) to tell if, on the go side, it's just reiterating what > the java side told it. > > The java side is definitely failing trying to parse a buffer. > > The cause of the invalid error isn't consistent: sometimes its a zero tag, > sometimes bad UTF-8, etc., so it appears to be a bad buffer. > > I added some very hacky debugging and in the bad UTF-8 case, which > confirms it looks like a bad buffer. There's the prefix of a normal string > and then a couple of nulls. > > Trying to figure out what in the middle of all of this to try to observe > to narrow things down. I did turn on the netty logging though I don't have > the background to get much out of it at this point. > > grpc-java is 1.0.3. Go grpc is > up-to-date/63bd55dfbf781b183216d2dd4433a659c947648a > according to glide. > > Annecdotally, it seems to happen only a little at the start of the service > and then increase in frequency (not sure if this is load based ... a bit > hard to tell that right now.) > > Also, just saw this for the first time: > > 2016-12-23T20:43:49.213Z [grpc-default-worker-ELG-3-3] ERROR > io.netty.util.ResourceLeakDetector > - LEAK: ByteBuf.release() was not called before it's garbage-collected. > Enable advanced leak reporting to find out where the leak occurred. To > enable advanced leak reporting, specify the JVM option > '-Dio.netty.leakDetection.level=adv > > Typical error message: > > 2016-12-23T19:52:23.597Z [grpc-default-executor-4] ERROR > io.grpc.internal.SerializingExecutor > - Exception while executing runnable io.grpc.internal.ServerImpl$ > JumpToApplicationThreadServerStreamListener$1@6e71d62c > io.grpc.StatusRuntimeException: INTERNAL: Invalid protobuf byte sequence > at io.grpc.Status.asRuntimeException(Status.java:532) > at io.grpc.protobuf.lite.ProtoLiteUtils$2.parse( > ProtoLiteUtils.java:175) > at io.grpc.protobuf.lite.ProtoLiteUtils$2.parse( > ProtoLiteUtils.java:96) > at io.grpc.MethodDescriptor.parseRequest(MethodDescriptor. > java:267) > at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl. > messageRead(ServerCallImpl.java:241) > at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerS > treamListener$1.runInContext(ServerImpl.java:557) > at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:52) > at io.grpc.internal.SerializingExecutor$TaskRunner.run( > SerializingExecutor.java:154) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol > message had invalid UTF-8. > at com.google.protobuf.InvalidProtocolBufferException.invalidUtf8( > InvalidProtocolBufferException.java:144) > at com.google.protobuf.CodedInputStream$ArrayDecoder. > readStringRequireUtf8(CodedInputStream.java:734) > at io.ascend.protos.core.Value.<init>(Value.java:65) > at io.ascend.protos.core.Value.<init>(Value.java:17) > at io.ascend.protos.core.Value$1.parsePartialFrom(Value.java:1553) > at io.ascend.protos.core.Value$1.parsePartialFrom(Value.java:1548) > at io.ascend.protos.core.Value$Builder.mergeFrom(Value.java:844) > at io.ascend.protos.core.Value$Builder.mergeFrom(Value.java:660) > at com.google.protobuf.CodedInputStream$ArrayDecoder. > readMessage(CodedInputStream.java:799) > at com.google.protobuf.MapEntryLite.parseField( > MapEntryLite.java:128) > at com.google.protobuf.MapEntryLite.parseEntry( > MapEntryLite.java:184) > at com.google.protobuf.MapEntry.<init>(MapEntry.java:106) > at com.google.protobuf.MapEntry.<init>(MapEntry.java:51) > at com.google.protobuf.MapEntry$Metadata$1.parsePartialFrom( > MapEntry.java:71) > at com.google.protobuf.MapEntry$Metadata$1.parsePartialFrom( > MapEntry.java:65) > at com.google.protobuf.CodedInputStream$ArrayDecoder. > readMessage(CodedInputStream.java:815) > at io.ascend.protos.core.Struct.<init>(Struct.java:57) > at io.ascend.protos.core.Struct.<init>(Struct.java:14) > at io.ascend.protos.core.Struct$1.parsePartialFrom(Struct. > java:673) > at io.ascend.protos.core.Struct$1.parsePartialFrom(Struct. > java:668) > at com.google.protobuf.CodedInputStream$ArrayDecoder. > readMessage(CodedInputStream.java:815) > at io.ascend.protos.worker.RecordsRead.<init>(RecordsRead.java:52) > at io.ascend.protos.worker.RecordsRead.<init>(RecordsRead.java:9) > at io.ascend.protos.worker.RecordsRead$1. > parsePartialFrom(RecordsRead.java:694) > at io.ascend.protos.worker.RecordsRead$1. > parsePartialFrom(RecordsRead.java:689) > at com.google.protobuf.CodedInputStream$ArrayDecoder. > readMessage(CodedInputStream.java:815) > at io.ascend.protos.worker.Update.<init>(Update.java:106) > at io.ascend.protos.worker.Update.<init>(Update.java:9) > at io.ascend.protos.worker.Update$1.parsePartialFrom( > Update.java:1942) > at io.ascend.protos.worker.Update$1.parsePartialFrom( > Update.java:1937) > at com.google.protobuf.AbstractParser.parseFrom( > AbstractParser.java:91) > at com.google.protobuf.AbstractParser.parseFrom( > AbstractParser.java:49) > at io.grpc.protobuf.lite.ProtoLiteUtils$2.parseFrom( > ProtoLiteUtils.java:180) > at io.grpc.protobuf.lite.ProtoLiteUtils$2.parse( > ProtoLiteUtils.java:172) > ... 9 common frames omitted > > -- > You received this message because you are subscribed to the Google Groups " > grpc.io" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/grpc-io. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/grpc-io/0b6a274a-ae50-4962-9dcd-ca9e6a94e1ed%40googlegroups.com > <https://groups.google.com/d/msgid/grpc-io/0b6a274a-ae50-4962-9dcd-ca9e6a94e1ed%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- - Kun -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAAyQuKxFhupvc4Bsr%2BHR_yEQwVwuRvZ1NnEOhOtmB6_vOaff4g%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
smime.p7s
Description: S/MIME Cryptographic Signature
