cuihaohao opened a new issue, #16427: URL: https://github.com/apache/dubbo/issues/16427
### Pre-check - [x] I am sure that all the content I provide is in English. ### Search before asking - [x] I had searched in the [issues](https://github.com/apache/dubbo/issues?q=is%3Aissue) and found no similar issues. ### Apache Dubbo Component Java SDK (apache/dubbo) ### Dubbo Version Environment 1: - Apache Dubbo Java: 3.3.1 - Netty: 4.1.107.Final and 4.1.113.Final - JDK: 1.8.0_192 - OS: Linux - Consumer: Java application using the Triple protocol - Provider: PHP application using standard gRPC over HTTP/2 Environment 2: - Apache Dubbo Java: 3.3.7-ygf-SNAPSHOT - Based on the Dubbo 3.3 branch and including the Triple backpressure changes from https://github.com/apache/dubbo/pull/15957 - Netty: 4.2.15.Final - JDK: 1.8.0_192 - OS: Linux - Consumer: Java application using the Triple protocol - Provider: PHP application using standard gRPC over HTTP/2 ### Steps to reproduce this issue ## Topology ```text Dubbo Java Triple consumer (10.16.8.5) | | HTTP/2 + gRPC v PHP gRPC provider (10.16.8.8:50051) ``` The invoked unary method is: ```text UserDataBusiness/ExportUserData ``` The serialized response is approximately 92 KiB: ```text gRPC payload: approximately 94,486 bytes gRPC record including the 5-byte prefix: approximately 94,491 bytes ``` The issue is reproducible when the PHP gRPC server returns this response. ## Case 1: 64 KiB initial stream window Configure or advertise `SETTINGS_INITIAL_WINDOW_SIZE = 65535`. The PHP server sends exactly the available stream-window data and then waits for a stream-level `WINDOW_UPDATE`. The Java consumer does not complete the approximately 92 KiB gRPC message and the call eventually times out. Observed sequence: ```text Java -> PHP: SETTINGS_INITIAL_WINDOW_SIZE = 65535 PHP -> Java: response DATA until the stream window is exhausted PHP: waits for stream-level WINDOW_UPDATE Java: waits for the remaining bytes of the complete gRPC message Java -> PHP: RST_STREAM CANCEL after the RPC timeout ``` This looks like a flow-control progress deadlock: the decoder needs more DATA to complete the message, while the sender needs WINDOW_UPDATE to send the remaining DATA. ## Case 2: advertised window increased to 8 MiB When the restriction is removed, the Java client advertises: ```text SETTINGS_INITIAL_WINDOW_SIZE = 8,388,608 SETTINGS_MAX_FRAME_SIZE = 8,388,608 ``` The packet capture confirms these client SETTINGS values: ```text SETTINGS_HEADER_TABLE_SIZE = 1,048,576 SETTINGS_ENABLE_PUSH = 0 SETTINGS_MAX_CONCURRENT_STREAMS = 2,147,483,647 SETTINGS_INITIAL_WINDOW_SIZE = 8,388,608 SETTINGS_MAX_FRAME_SIZE = 8,388,608 SETTINGS_MAX_HEADER_LIST_SIZE = 32,768 ``` The PHP server then sends the complete approximately 92 KiB response immediately, split into multiple HTTP/2 DATA frames (mostly 16 KiB each). The full response reaches the Java host in less than 100 ms. However, the Java client fails while Netty processes the inbound DATA frames: ```text io.netty.handler.codec.http2.Http2Exception$StreamException: Flow control window exceeded for stream: 5 at io.netty.handler.codec.http2.Http2Exception.streamError(...) at io.netty.handler.codec.http2.DefaultHttp2LocalFlowController$DefaultState.receiveFlowControlledFrame(...) at io.netty.handler.codec.http2.DefaultHttp2LocalFlowController.receiveFlowControlledFrame(...) at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(...) ``` The application finally receives: ```text org.apache.dubbo.rpc.StatusRpcException: CANCELLED : Canceled by remote peer, errorCode=2 ``` According to the packet capture, the Java side sends the reset after the local Netty flow-control exception; the PHP provider does not cancel the RPC first. The same exception is reproducible with official Dubbo 3.3.1: ```text dubbo version: 3.3.1 netty-codec-http2: 4.1.113.Final Flow control window exceeded for stream: 3 ``` It was also observed with Netty 4.1.107.Final, so this does not appear to be exclusively introduced by PR #15957. ### What you expected to happen The Java Triple consumer should successfully receive and decode a unary gRPC response of approximately 92 KiB. The response is smaller than Dubbo's defaults: ```text DEFAULT_INITIAL_WINDOW_SIZE = 8 MiB DEFAULT_MAX_MESSAGE_SIZE = 50 MiB ``` The receive-window value advertised to the peer should be consistent with the window enforced by the Java client's local HTTP/2 flow controller. If the client advertises `SETTINGS_INITIAL_WINDOW_SIZE = 8 MiB`, receiving approximately 92 KiB of DATA on one stream should not result in `Flow control window exceeded for stream`. With a smaller window, the client should return flow-control credit as DATA is consumed: ```text PHP sends part of the response Java consumes the received DATA Java sends stream-level WINDOW_UPDATE PHP sends the remaining response Java completes the unary RPC successfully ``` ### Anything else ## Initial analysis We currently see two related failure modes. ### 1. Progress problem with a 64 KiB window If consumed bytes are returned only after a complete gRPC message has been assembled, a message larger than the stream window cannot make progress: ```text Decoder needs more DATA to complete the message Sender needs WINDOW_UPDATE to send more DATA WINDOW_UPDATE depends on the decoder consuming the message ``` The backpressure work in https://github.com/apache/dubbo/pull/15957 appears related to this part. ### 2. Advertised and locally enforced windows may be inconsistent When Java advertises an 8 MiB stream window, the PHP server legally sends the complete 92 KiB response. Netty nevertheless reports that the stream window is exceeded. Possible explanations include: - `Http2Settings.initialWindowSize` is sent to the peer but not consistently applied to the local `Http2LocalFlowController`. - A newly created stream is initialized with 65,535 bytes instead of the configured value. - Flow-controlled bytes arrive faster than the local consumption path updates the window. - Stream-channel automatic/manual flow-control accounting is not synchronized. - Configuration is applied to the outbound SETTINGS frame but not to every internal receive-window state. These are hypotheses based on packet captures and stack traces, not a confirmed root cause. ## Why PHP gRPC exposes it consistently The PHP gRPC implementation trusts the Java client's SETTINGS. If Java advertises 8 MiB, PHP may immediately send the complete response. Java-to-Java Triple calls may use different write scheduling or backpressure behavior and may interleave transmission and consumption, hiding the same receive-window problem. ## Clarification about our usage pattern We understand that our current usage pattern may not be ideal: - A standard PHP gRPC provider is called by a Dubbo Java Triple consumer. - The method returns a relatively large unary response. - We experimented with HTTP/2 window settings as a workaround. - Server streaming may be more appropriate if the response continues to grow. However, the current response is only approximately 92 KiB and remains below Dubbo's default message/window limits. We would appreciate guidance on: 1. Is a standard PHP gRPC provider officially supported as a Dubbo Java Triple interoperability scenario? 2. Should unary responses larger than 64 KiB work without application-level streaming? 3. Must users configure both stream and connection windows manually? 4. Should `TripleConfig.initialWindowSize` automatically initialize both the outbound SETTINGS value and the local `Http2LocalFlowController`? 5. Is PR #15957 expected to resolve this case completely, or only provide application-level backpressure APIs? 6. Would explicitly initializing the local controller be appropriate? For example: ```java int initialWindowSize = tripleConfig.getInitialWindowSizeOrDefault(); TripleHttp2LocalFlowController flowController = new TripleHttp2LocalFlowController(connection, windowUpdateRatio); flowController.initialWindowSize(initialWindowSize); connection.local().flowController(flowController); ``` We can provide packet captures and prepare a minimal Java/PHP reproduction if needed. ### Do you have a (mini) reproduction demo? - [ ] Yes, I have a minimal reproduction demo to help resolve this issue more effectively! ### Are you willing to submit a pull request to fix on your own? - [ ] Yes I am willing to submit a pull request on my own! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
