Anon2Tokyo commented on issue #16414:
URL: https://github.com/apache/dubbo/issues/16414#issuecomment-5221616509
@1536191986zh-cloud
I investigated this issue and found that it is caused by decoder state loss
in the no-stub gRPC provider path.
### Root cause This only affects the no-stub provider path, where
`GrpcHttp2ServerTransportListener` first uses `LazyFindMethodListener` to parse
the initial client-streaming message and then switches to the real method
listener. The problematic case is when the first HTTP/2 DATA frame contains
more than one gRPC message fragment, for example: ```text [complete message
#1][beginning of message #2]
In the old implementation, `LazyFindMethodListener` created a temporary
`GrpcStreamingDecoder` to decode the first message. If the same DATA frame also
contained the beginning of the second message, that partial second message was
buffered inside the temporary decoder.
After the listener switched to the real method listener, the real streaming
decoder did not have that buffered state. As a result, decoding continued from
an incorrect message boundary, which could lead to protobuf parse errors,
invalid compression/reserved-bit errors, or Netty buffer reference-count errors.
This is why the issue is hard to reproduce locally: it depends on HTTP/2
DATA frame splitting/coalescing timing. It is more likely with:
- no-stub provider
- client streaming
- many fast consecutive `onNext` calls
- a very small first message
- larger following messages, for example `128KB`
- no artificial delay or logging inside the sending loop
### Fix
The fix is to make `LazyFindMethodListener` use the transport's actual
streaming decoder instead of creating a temporary decoder.
So instead of decoding the first message with a separate `new
GrpcStreamingDecoder()`, it now reuses `getStreamingDecoder()`.
This preserves any buffered partial gRPC message across the transition from
`LazyFindMethodListener` to the real method listener.
### Verification
I added a regression test that simulates this exact shape:
```text
DATA frame = [full first gRPC message][partial second gRPC message]
```
The test verifies that after the listener switch, the remaining bytes of the
second message are still decoded correctly.
I also reproduced the issue against Dubbo `3.3.6` using a no-stub provider
and client-streaming calls. With `20` messages of `128KB` sent quickly, one run
failed at iteration 17 with server-side decode errors including
`LengthFieldStreamingDecoder`, `InvalidProtocolBufferException`, and
`IllegalReferenceCountException: refCnt: 0`.
provider logļ¼
````
2026-08-08 02:34:09.244 ERROR 10324 --- [0051-thread-200]
.r.p.t.h.AbstractServerTransportListener : [DUBBO] An error occurred while
processing the http request with GrpcHttp2ServerTransportListener,
Http2MetadataFrame{method='POST', path='/repro.api.UploadFacade/upload',
contentType='application/grpc+proto', streamId=3, endStream=false},
method=repro.api.UploadFacade.upload(StreamObserver), dubbo version: 3.3.6,
current host: 192.168.0.115, error code: 99-0. This may be caused by , go to
https://dubbo.apache.org/faq/99/0 to find instructions.
org.apache.dubbo.remoting.http12.exception.DecodeException: Internal Server
Error
at
org.apache.dubbo.rpc.protocol.tri.h12.grpc.GrpcCompositeCodec.decode(GrpcCompositeCodec.java:105)
~[dubbo-3.3.6.jar:3.3.6]
at
org.apache.dubbo.rpc.protocol.tri.h12.grpc.GrpcCompositeCodec.decode(GrpcCompositeCodec.java:111)
~[dubbo-3.3.6.jar:3.3.6]
at
org.apache.dubbo.remoting.http12.message.HttpMessageDecoder.decode(HttpMessageDecoder.java:56)
~[dubbo-3.3.6.jar:3.3.6]
at
org.apache.dubbo.remoting.http12.message.DefaultListeningDecoder.decode(DefaultListeningDecoder.java:41)
~[dubbo-3.3.6.jar:3.3.6]
at
org.apache.dubbo.remoting.http12.message.StreamingDecoder$DefaultFragmentListener.onFragmentMessage(StreamingDecoder.java:55)
~[dubbo-3.3.6.jar:3.3.6]
at
org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.invokeListener(LengthFieldStreamingDecoder.java:180)
~[dubbo-3.3.6.jar:3.3.6]
at
org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.processBody(LengthFieldStreamingDecoder.java:172)
~[dubbo-3.3.6.jar:3.3.6]
at
org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.deliver(LengthFieldStreamingDecoder.java:121)
~[dubbo-3.3.6.jar:3.3.6]
at
org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.decode(LengthFieldStreamingDecoder.java:69)
~[dubbo-3.3.6.jar:3.3.6]
at
org.apache.dubbo.rpc.protocol.tri.h12.http2.GenericHttp2ServerTransportListener$StreamingHttpMessageListener.onMessage(GenericHttp2ServerTransportListener.java:220)
~[dubbo-3.3.6.jar:3.3.6]
at
org.apache.dubbo.rpc.protocol.tri.h12.AbstractServerTransportListener.doOnData(AbstractServerTransportListener.java:183)
~[dubbo-3.3.6.jar:3.3.6]
at
org.apache.dubbo.rpc.protocol.tri.h12.AbstractServerTransportListener.lambda$onData$1(AbstractServerTransportListener.java:168)
~[dubbo-3.3.6.jar:3.3.6]
at
org.apache.dubbo.common.threadpool.serial.SerializingExecutor.run(SerializingExecutor.java:111)
~[dubbo-3.3.6.jar:3.3.6]
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
~[na:na]
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
~[na:na]
at
org.apache.dubbo.common.threadlocal.InternalRunnable.run(InternalRunnable.java:39)
~[dubbo-3.3.6.jar:3.3.6]
at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
Caused by: java.io.IOException:
com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol
message, the input ended unexpectedly in the middle of a field. This could
mean either that the input has been truncated or that an embedded message
misreported its own length.
at
org.apache.dubbo.rpc.protocol.tri.SingleProtobufUtils.deserialize(SingleProtobufUtils.java:104)
~[dubbo-3.3.6.jar:3.3.6]
at org.apache.dubbo.rpc.protocol.tri.PbUnpack.unpack(PbUnpack.java:35)
~[dubbo-3.3.6.jar:3.3.6]
at
org.apache.dubbo.rpc.model.PackableMethod.parseRequest(PackableMethod.java:26)
~[dubbo-3.3.6.jar:3.3.6]
at
org.apache.dubbo.rpc.protocol.tri.h12.grpc.GrpcCompositeCodec.decode(GrpcCompositeCodec.java:101)
~[dubbo-3.3.6.jar:3.3.6]
... 16 common frames omitted
Caused by: com.google.protobuf.InvalidProtocolBufferException: While parsing
a protocol message, the input ended unexpectedly in the middle of a field.
This could mean either that the input has been truncated or that an embedded
message misreported its own length.
at
com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:92)
~[protobuf-java-3.25.5.jar:na]
at
com.google.protobuf.CodedInputStream$StreamDecoder.readRawBytesSlowPathRemainingChunks(CodedInputStream.java:2950)
~[protobuf-java-3.25.5.jar:na]
at
com.google.protobuf.CodedInputStream$StreamDecoder.readBytesSlowPath(CodedInputStream.java:2987)
~[protobuf-java-3.25.5.jar:na]
at
com.google.protobuf.CodedInputStream$StreamDecoder.readBytes(CodedInputStream.java:2373)
~[protobuf-java-3.25.5.jar:na]
at repro.upload.UploadFrame$Builder.mergeFrom(UploadFrame.java:452)
~[classes/:na]
at repro.upload.UploadFrame$1.parsePartialFrom(UploadFrame.java:648)
~[classes/:na]
at repro.upload.UploadFrame$1.parsePartialFrom(UploadFrame.java:640)
~[classes/:na]
at
com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:192)
~[protobuf-java-3.25.5.jar:na]
at
com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:209)
~[protobuf-java-3.25.5.jar:na]
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:25)
~[protobuf-java-3.25.5.jar:na]
at
org.apache.dubbo.rpc.protocol.tri.SingleProtobufUtils$SingleMessageMarshaller.parse(SingleProtobufUtils.java:132)
~[dubbo-3.3.6.jar:3.3.6]
at
org.apache.dubbo.rpc.protocol.tri.SingleProtobufUtils.deserialize(SingleProtobufUtils.java:102)
~[dubbo-3.3.6.jar:3.3.6]
... 19 common frames omitted
2026-08-08 02:34:09.253 ERROR 10324 --- [0051-thread-200]
.r.p.t.h.AbstractServerTransportListener : [DUBBO] An error occurred while
processing the http request with GrpcHttp2ServerTransportListener,
Http2MetadataFrame{method='POST', path='/repro.api.UploadFacade/upload',
contentType='application/grpc+proto', streamId=3, endStream=false},
method=repro.api.UploadFacade.upload(StreamObserver), dubbo version: 3.3.6,
current host: 192.168.0.115, error code: 99-0. This may be caused by , go to
https://dubbo.apache.org/faq/99/0 to find instructions.
io.netty.util.IllegalReferenceCountException: refCnt: 0
at
io.netty.buffer.AbstractByteBuf.ensureAccessible(AbstractByteBuf.java:1454)
~[netty-buffer-4.1.101.Final.jar:4.1.101.Final]
at
io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1440)
~[netty-buffer-4.1.101.Final.jar:4.1.101.Final]
at
io.netty.buffer.AbstractByteBuf.checkReadableBytes(AbstractByteBuf.java:1428)
~[netty-buffer-4.1.101.Final.jar:4.1.101.Final]
at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:895)
~[netty-buffer-4.1.101.Final.jar:4.1.101.Final]
at io.netty.buffer.ByteBufInputStream.read(ByteBufInputStream.java:183)
~[netty-buffer-4.1.101.Final.jar:4.1.101.Final]
at
org.apache.dubbo.remoting.http12.CompositeInputStream.read(CompositeInputStream.java:82)
~[dubbo-3.3.6.jar:3.3.6]
at
org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.readRawMessage(LengthFieldStreamingDecoder.java:185)
~[dubbo-3.3.6.jar:3.3.6]
at
org.apache.dubbo.rpc.protocol.tri.h12.grpc.GrpcStreamingDecoder.readRawMessage(GrpcStreamingDecoder.java:54)
~[dubbo-3.3.6.jar:3.3.6]
at
org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.processBody(LengthFieldStreamingDecoder.java:170)
~[dubbo-3.3.6.jar:3.3.6]
at
org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.deliver(LengthFieldStreamingDecoder.java:121)
~[dubbo-3.3.6.jar:3.3.6]
at
org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.decode(LengthFieldStreamingDecoder.java:69)
~[dubbo-3.3.6.jar:3.3.6]
at
org.apache.dubbo.rpc.protocol.tri.h12.http2.GenericHttp2ServerTransportListener$StreamingHttpMessageListener.onMessage(GenericHttp2ServerTransportListener.java:220)
~[dubbo-3.3.6.jar:3.3.6]
at
org.apache.dubbo.rpc.protocol.tri.h12.AbstractServerTransportListener.doOnData(AbstractServerTransportListener.java:183)
~[dubbo-3.3.6.jar:3.3.6]
at
org.apache.dubbo.rpc.protocol.tri.h12.AbstractServerTransportListener.lambda$onData$1(AbstractServerTransportListener.java:168)
~[dubbo-3.3.6.jar:3.3.6]
at
org.apache.dubbo.common.threadpool.serial.SerializingExecutor.run(SerializingExecutor.java:111)
~[dubbo-3.3.6.jar:3.3.6]
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
~[na:na]
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
~[na:na]
at
org.apache.dubbo.common.threadlocal.InternalRunnable.run(InternalRunnable.java:39)
~[dubbo-3.3.6.jar:3.3.6]
at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
2026-08-08 02:34:09.281 INFO 10324 --- [rverWorker-3-21]
o.a.d.r.t.netty4.NettyServerHandler : [DUBBO] The connection [id:
0xf714b008, L:/192.168.0.115:50051 ! R:/192.168.0.115:56989] of
192.168.0.115:56989 -> 192.168.0.115:50051 is disconnected., dubbo version:
3.3.6, current host: 192.168.0.115
2026-08-08 02:34:09.282 INFO 10324 --- [rverWorker-3-21]
o.a.d.r.t.n.NettyPortUnificationServer : [DUBBO] All clients has
disconnected from /192.168.0.115:50051. You can graceful shutdown now., dubbo
version: 3.3.6, current host: 192.168.0.115
2026-08-08 02:34:09.282 INFO 10324 --- [rverWorker-3-21]
o.a.d.r.t.netty4.NettyChannelHandler : [DUBBO] The connection [id:
0xf714b008, L:/192.168.0.115:50051 ! R:/192.168.0.115:56989] of
192.168.0.115:56989 -> 192.168.0.115:50051 is disconnected., dubbo version:
3.3.6, current host: 192.168.0.115
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
