qianye1001 opened a new issue, #1309: URL: https://github.com/apache/rocketmq-clients/issues/1309
### Before Creating the Bug Report - [x] I found a bug, not just asking a question, which should be created in GitHub Discussions. - [x] I have searched the GitHub Issues and GitHub Discussions of this repository and believe that this is not a duplicate. - [x] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ. ### Programming Language of the Client Java ### Runtime Platform Environment Any ### RocketMQ Version of the Client/Server Java client `master` / 5.2.1. The problem also affects older versions with the same RPC and channel handling. ### Run or Compiler Version Any supported JDK ### Describe the Bug The Java gRPC client currently has two related transport-resilience gaps. #### 1. gRPC `RESOURCE_EXHAUSTED` is not recognized as throttling The client recognizes the RocketMQ response-body status `Code.TOO_MANY_REQUESTS` in `StatusChecker` and converts it to `TooManyRequestsException`. However, if a server or an intermediary returns the canonical gRPC transport status `RESOURCE_EXHAUSTED`, the RPC future fails with `StatusRuntimeException`. The response transformation and `StatusChecker` are never invoked. As a result, `ProducerImpl` treats the failure as a non-throttling error and retries immediately instead of applying the configured throttling backoff. Push-consumer receive logic also uses the normal failure delay instead of its flow-control delay. #### 2. A half-open TCP connection may be reused long after heartbeats start failing `RpcClientImpl` currently configures: ```java keepAliveTime(300, TimeUnit.SECONDS) keepAliveTimeout(30, TimeUnit.SECONDS) keepAliveWithoutCalls(true) ``` The client sends heartbeats every 10 seconds with the configured RPC deadline (3 seconds by default), but `ClientImpl#doHeartbeat` only logs heartbeat failures. It does not ask the cached `ManagedChannel` to create a new transport. If packets are silently dropped after a TCP connection is established (no FIN/RST), send RPCs and heartbeat RPCs repeatedly time out while subsequent requests continue using the same cached channel. Recovery may wait for the 5-minute gRPC keepalive plus its timeout, or for the operating-system TCP timeout. ### Steps to Reproduce #### Transport throttling 1. Make a gRPC server/proxy return `io.grpc.Status.RESOURCE_EXHAUSTED` for `SendMessage`. 2. Publish a message with the Java client. 3. Observe that the failure remains a `StatusRuntimeException` and Producer retries it as a non-throttling failure. #### Half-open connection 1. Establish a Java client connection and successfully publish messages. 2. Drop packets for the established connection without sending FIN/RST, for example with a firewall `DROP` rule. 3. Continue publishing messages. 4. Observe heartbeat deadline failures and repeated send timeouts while the same channel remains cached. ### What Did You Expect to See? - gRPC `RESOURCE_EXHAUSTED` should be normalized to the same client semantic as RocketMQ `Code.TOO_MANY_REQUESTS`, so existing throttling backoff logic is reused. - Consecutive heartbeat transport failures should cause new RPCs to use a fresh transport within the heartbeat detection window, rather than waiting several minutes for TCP/keepalive timeout. - Throttling and ordinary application errors must not trigger channel recreation. ### What Did You See Instead? - `RESOURCE_EXHAUSTED` bypasses `StatusChecker`, so it is not recognized as throttling. - Heartbeat failures only produce log messages and do not accelerate transport recovery. - Producer retries may continue on the half-open channel until the underlying transport is eventually declared dead. ### Suggested Direction #### Normalize transport throttling at the RPC boundary At a common RPC future/client-manager boundary, convert gRPC `RESOURCE_EXHAUSTED` into `TooManyRequestsException` using RocketMQ response code `42900`, the current RPC request ID, and the gRPC description. Preserve the original `StatusRuntimeException` as the cause. Other gRPC status codes should remain unchanged. #### Use heartbeat as the primary channel-health signal Avoid rebuilding the channel for every send failure because send failures include throttling, application errors, server latency, and ambiguous `DEADLINE_EXCEEDED` results. Concurrent producers could also cause a channel-recreation storm. A possible policy per endpoint is: - Any heartbeat response proves the transport is alive and resets the consecutive-failure count, even if the response contains a non-OK RocketMQ application status. - A definite heartbeat transport failure such as connection-closed/`UNAVAILABLE` may trigger recovery immediately. - Two consecutive heartbeat `DEADLINE_EXCEEDED` failures trigger `ManagedChannel#enterIdle()` so new RPCs create a new transport. - Add a cooldown to avoid repeated reconnect attempts during a server-wide incident. - `RESOURCE_EXHAUSTED` must be treated as throttling and must not trigger channel recovery. `enterIdle()` is preferable to replacing the entire `ManagedChannel`: existing RPCs can finish or reach their deadlines, while new RPCs establish a new transport without replacing cached stubs and channel ownership structures. ### Additional Context Related issue: #872 PR #870 added default Java gRPC keepalive for #872. That prevents an infinite wait, but the current 5-minute interval still leaves a long outage window, and heartbeat failures do not currently accelerate recovery. This issue describes the remaining Java recovery gap and the separate missing transport-status normalization. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
