LI123456mo commented on issue #16414: URL: https://github.com/apache/dubbo/issues/16414#issuecomment-5226023236
> [@1536191986zh-cloud](https://github.com/1536191986zh-cloud) I investigated this issue and found that it is caused by decoder state loss in the no-stub gRPC provider path. > > ### Root cause > This only affects the no-stub provider path, where `GrpcHttp2ServerTransportListener` first uses `LazyFindMethodListener` to parse the initial client-streaming message and then switches to the real method listener. The problematic case is when the first HTTP/2 DATA frame contains more than one gRPC message fragment, for example: > > ``` > [complete message #1][beginning of message #2] > ``` > > In the old implementation, `LazyFindMethodListener` created a temporary `GrpcStreamingDecoder` to decode the first message. If the same DATA frame also contained the beginning of the second message, that partial second message was buffered inside the temporary decoder. > > After the listener switched to the real method listener, the real streaming decoder did not have that buffered state. As a result, decoding continued from an incorrect message boundary, which could lead to protobuf parse errors, invalid compression/reserved-bit errors, or Netty buffer reference-count errors. > > This is why the issue is hard to reproduce locally: it depends on HTTP/2 DATA frame splitting/coalescing timing. It is more likely with: > > * no-stub provider > * client streaming > * many fast consecutive `onNext` calls > * a very small first message > * larger following messages, for example `128KB` > * no artificial delay or logging inside the sending loop > > ### Fix > The fix is to make `LazyFindMethodListener` use the transport's actual streaming decoder instead of creating a temporary decoder. > > So instead of decoding the first message with a separate `new GrpcStreamingDecoder()`, it now reuses `getStreamingDecoder()`. > > This preserves any buffered partial gRPC message across the transition from `LazyFindMethodListener` to the real method listener. > > ### Verification > I added a regression test that simulates this exact shape: > > ``` > DATA frame = [full first gRPC message][partial second gRPC message] > ``` > > The test verifies that after the listener switch, the remaining bytes of the second message are still decoded correctly. > > I also reproduced the issue against Dubbo `3.3.6` using a no-stub provider and client-streaming calls. With `20` messages of `128KB` sent quickly, one run failed at iteration 17 with server-side decode errors including `LengthFieldStreamingDecoder`, `InvalidProtocolBufferException`, and `IllegalReferenceCountException: refCnt: 0`. > > provider logļ¼ > > ``` > 2026-08-08 02:34:09.244 ERROR 10324 --- [0051-thread-200] .r.p.t.h.AbstractServerTransportListener : [DUBBO] An error occurred while processing the http request with GrpcHttp2ServerTransportListener, Http2MetadataFrame{method='POST', path='/repro.api.UploadFacade/upload', contentType='application/grpc+proto', streamId=3, endStream=false}, method=repro.api.UploadFacade.upload(StreamObserver), dubbo version: 3.3.6, current host: 192.168.0.115, error code: 99-0. This may be caused by , go to https://dubbo.apache.org/faq/99/0 to find instructions. > > org.apache.dubbo.remoting.http12.exception.DecodeException: Internal Server Error > at org.apache.dubbo.rpc.protocol.tri.h12.grpc.GrpcCompositeCodec.decode(GrpcCompositeCodec.java:105) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.rpc.protocol.tri.h12.grpc.GrpcCompositeCodec.decode(GrpcCompositeCodec.java:111) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.remoting.http12.message.HttpMessageDecoder.decode(HttpMessageDecoder.java:56) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.remoting.http12.message.DefaultListeningDecoder.decode(DefaultListeningDecoder.java:41) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.remoting.http12.message.StreamingDecoder$DefaultFragmentListener.onFragmentMessage(StreamingDecoder.java:55) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.invokeListener(LengthFieldStreamingDecoder.java:180) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.processBody(LengthFieldStreamingDecoder.java:172) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.deliver(LengthFieldStreamingDecoder.java:121) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.decode(LengthFieldStreamingDecoder.java:69) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.rpc.protocol.tri.h12.http2.GenericHttp2ServerTransportListener$StreamingHttpMessageListener.onMessage(GenericHttp2ServerTransportListener.java:220) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.rpc.protocol.tri.h12.AbstractServerTransportListener.doOnData(AbstractServerTransportListener.java:183) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.rpc.protocol.tri.h12.AbstractServerTransportListener.lambda$onData$1(AbstractServerTransportListener.java:168) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.common.threadpool.serial.SerializingExecutor.run(SerializingExecutor.java:111) ~[dubbo-3.3.6.jar:3.3.6] > at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na] > at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na] > at org.apache.dubbo.common.threadlocal.InternalRunnable.run(InternalRunnable.java:39) ~[dubbo-3.3.6.jar:3.3.6] > at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na] > Caused by: java.io.IOException: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either that the input has been truncated or that an embedded message misreported its own length. > at org.apache.dubbo.rpc.protocol.tri.SingleProtobufUtils.deserialize(SingleProtobufUtils.java:104) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.rpc.protocol.tri.PbUnpack.unpack(PbUnpack.java:35) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.rpc.model.PackableMethod.parseRequest(PackableMethod.java:26) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.rpc.protocol.tri.h12.grpc.GrpcCompositeCodec.decode(GrpcCompositeCodec.java:101) ~[dubbo-3.3.6.jar:3.3.6] > ... 16 common frames omitted > Caused by: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either that the input has been truncated or that an embedded message misreported its own length. > at com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:92) ~[protobuf-java-3.25.5.jar:na] > at com.google.protobuf.CodedInputStream$StreamDecoder.readRawBytesSlowPathRemainingChunks(CodedInputStream.java:2950) ~[protobuf-java-3.25.5.jar:na] > at com.google.protobuf.CodedInputStream$StreamDecoder.readBytesSlowPath(CodedInputStream.java:2987) ~[protobuf-java-3.25.5.jar:na] > at com.google.protobuf.CodedInputStream$StreamDecoder.readBytes(CodedInputStream.java:2373) ~[protobuf-java-3.25.5.jar:na] > at repro.upload.UploadFrame$Builder.mergeFrom(UploadFrame.java:452) ~[classes/:na] > at repro.upload.UploadFrame$1.parsePartialFrom(UploadFrame.java:648) ~[classes/:na] > at repro.upload.UploadFrame$1.parsePartialFrom(UploadFrame.java:640) ~[classes/:na] > at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:192) ~[protobuf-java-3.25.5.jar:na] > at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:209) ~[protobuf-java-3.25.5.jar:na] > at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:25) ~[protobuf-java-3.25.5.jar:na] > at org.apache.dubbo.rpc.protocol.tri.SingleProtobufUtils$SingleMessageMarshaller.parse(SingleProtobufUtils.java:132) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.rpc.protocol.tri.SingleProtobufUtils.deserialize(SingleProtobufUtils.java:102) ~[dubbo-3.3.6.jar:3.3.6] > ... 19 common frames omitted > > 2026-08-08 02:34:09.253 ERROR 10324 --- [0051-thread-200] .r.p.t.h.AbstractServerTransportListener : [DUBBO] An error occurred while processing the http request with GrpcHttp2ServerTransportListener, Http2MetadataFrame{method='POST', path='/repro.api.UploadFacade/upload', contentType='application/grpc+proto', streamId=3, endStream=false}, method=repro.api.UploadFacade.upload(StreamObserver), dubbo version: 3.3.6, current host: 192.168.0.115, error code: 99-0. This may be caused by , go to https://dubbo.apache.org/faq/99/0 to find instructions. > > io.netty.util.IllegalReferenceCountException: refCnt: 0 > at io.netty.buffer.AbstractByteBuf.ensureAccessible(AbstractByteBuf.java:1454) ~[netty-buffer-4.1.101.Final.jar:4.1.101.Final] > at io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1440) ~[netty-buffer-4.1.101.Final.jar:4.1.101.Final] > at io.netty.buffer.AbstractByteBuf.checkReadableBytes(AbstractByteBuf.java:1428) ~[netty-buffer-4.1.101.Final.jar:4.1.101.Final] > at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:895) ~[netty-buffer-4.1.101.Final.jar:4.1.101.Final] > at io.netty.buffer.ByteBufInputStream.read(ByteBufInputStream.java:183) ~[netty-buffer-4.1.101.Final.jar:4.1.101.Final] > at org.apache.dubbo.remoting.http12.CompositeInputStream.read(CompositeInputStream.java:82) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.readRawMessage(LengthFieldStreamingDecoder.java:185) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.rpc.protocol.tri.h12.grpc.GrpcStreamingDecoder.readRawMessage(GrpcStreamingDecoder.java:54) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.processBody(LengthFieldStreamingDecoder.java:170) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.deliver(LengthFieldStreamingDecoder.java:121) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.decode(LengthFieldStreamingDecoder.java:69) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.rpc.protocol.tri.h12.http2.GenericHttp2ServerTransportListener$StreamingHttpMessageListener.onMessage(GenericHttp2ServerTransportListener.java:220) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.rpc.protocol.tri.h12.AbstractServerTransportListener.doOnData(AbstractServerTransportListener.java:183) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.rpc.protocol.tri.h12.AbstractServerTransportListener.lambda$onData$1(AbstractServerTransportListener.java:168) ~[dubbo-3.3.6.jar:3.3.6] > at org.apache.dubbo.common.threadpool.serial.SerializingExecutor.run(SerializingExecutor.java:111) ~[dubbo-3.3.6.jar:3.3.6] > at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na] > at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na] > at org.apache.dubbo.common.threadlocal.InternalRunnable.run(InternalRunnable.java:39) ~[dubbo-3.3.6.jar:3.3.6] > at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na] > > 2026-08-08 02:34:09.281 INFO 10324 --- [rverWorker-3-21] o.a.d.r.t.netty4.NettyServerHandler : [DUBBO] The connection [id: 0xf714b008, L:/192.168.0.115:50051 ! R:/192.168.0.115:56989] of 192.168.0.115:56989 -> 192.168.0.115:50051 is disconnected., dubbo version: 3.3.6, current host: 192.168.0.115 > 2026-08-08 02:34:09.282 INFO 10324 --- [rverWorker-3-21] o.a.d.r.t.n.NettyPortUnificationServer : [DUBBO] All clients has disconnected from /192.168.0.115:50051. You can graceful shutdown now., dubbo version: 3.3.6, current host: 192.168.0.115 > 2026-08-08 02:34:09.282 INFO 10324 --- [rverWorker-3-21] o.a.d.r.t.netty4.NettyChannelHandler : [DUBBO] The connection [id: 0xf714b008, L:/192.168.0.115:50051 ! R:/192.168.0.115:56989] of 192.168.0.115:56989 -> 192.168.0.115:50051 is disconnected., dubbo version: 3.3.6, current host: 192.168.0.115 > ``` Hello @Anon2Tokyo , can you give the steps to run this to also affirm the same failure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
