caojiaqiang opened a new issue, #10911: URL: https://github.com/apache/rocketmq/issues/10911
### Before Creating the Bug Report - [x] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions). - [x] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions) of this repository and believe that this is not a duplicate. - [x] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ. ### Runtime platform environment - OS: Linux - Multiple independent producers and consumers connect directly to the Broker. There is no proxy or load balancer on this path. - `serverPooledByteBufAllocatorEnable=true` - The incident occurred during normal concurrent `SEND_MESSAGE` and long-poll `PULL_MESSAGE` traffic. ### RocketMQ version - Branch: 4.9.x - Version: customized distribution `4.9.8.2-ly`, based on Apache RocketMQ 4.9.8 - Netty: 4.1.65.Final This has not yet been reproduced on an unmodified official RocketMQ release. We are also auditing distribution-specific changes. ### JDK Version - Compiler/runtime: JDK 8 ### Describe the Bug A rare Broker-side data corruption was observed. A persisted business message body starts with the expected JSON, but from byte offset 457 it contains multiple RocketMQ RemotingCommand frames, including request code 11 (`PULL_MESSAGE`), request code 38, and binary total-length/header-length prefixes. The embedded requests contain metadata from unrelated consumers running in different processes on different hosts. The producer and those consumers only share the same Broker connection endpoint; they do not share a client process or sending path. The CommitLog record was read directly and its BODYCRC was independently verified: ```text total message record size: 2310 body length: 2063 stored BODYCRC: 348336256 (0x14c33080) CRC32 recalculated from the stored body: 348336256 BODYCRC_MATCH=true ``` Therefore, the unexpected bytes were already part of the message body when the Broker calculated BODYCRC. This excludes query-tool decoding and post-write disk corruption. During the exact incident second, Broker remoting logs also show a burst of decoder errors across independent remote connections, mainly `TooLongFrameException` and JSON header parse failures. A sanitized consumer request found in one decoder failure matches the request metadata embedded in the persisted business body. A separate producer connection also shows a `SEND_MESSAGE` header interrupted by a binary frame prefix and the start of a `PULL_MESSAGE` header. This suggests corruption in the Broker remoting receive/decode path, but the exact root cause is not yet proven. Possible areas include pooled `ByteBuf` ownership/lifetime, interaction with the codec `DefaultEventExecutorGroup`, or a distribution-specific modification. ### Steps to Reproduce There is no deterministic reproduction yet because the event is rare. The following steps reproduce the forensic verification of the observed record: 1. Run a Broker with the pooled server allocator enabled and multiple independent clients concurrently sending messages and issuing long-poll pull requests. 2. When a corrupted message is detected, resolve its physical CommitLog offset and read the message record directly from the mapped CommitLog file. 3. Parse `TOTALSIZE`, `BODYCRC`, `SYSFLAG`, and `BODYLEN`, then extract exactly `BODYLEN` bytes from the record. 4. Recalculate CRC32 over those body bytes and compare it with the stored `BODYCRC`. 5. Search the extracted body for RocketMQ RemotingCommand headers and binary frame prefixes. 6. Correlate the embedded request metadata with Broker decoder errors from other remote connections at the same timestamp. For the observed 2063-byte body, foreign request markers occurred at these offsets: ```text code 11: 457, 859, 905, 1663, 1801 code 38: 1353 ``` All production IP addresses, message IDs, topics, consumer groups, and raw logs are omitted from this public report. ### What Did You Expect to See? Each channel decoder should only observe bytes received from that channel. The body persisted for a `SEND_MESSAGE` request should match the producer's request body. A malformed frame on one connection should be rejected or close only that connection and must not affect another connection's request. ### What Did You See Instead? Remoting request bytes associated with independent consumer connections appeared inside another producer's persisted business message body. Because the body was still structurally acceptable to the decoded `SEND_MESSAGE` request, the Broker stored it with a matching BODYCRC instead of rejecting it. Some mixed frames caused decoder exceptions and connection closures, while this body-only corruption passed decoding and was persisted silently. ### Additional Context The closest issues found were: - #8086 and #6784: gRPC clients connecting to classic Remoting ports, causing protocol mismatch. - #3292: a TLS and `transferMsgByHeap=false` issue on the client response path. None of them reports frames from independent connections appearing in a persisted message body with a matching BODYCRC. Relevant 4.9.x code path: - `NettyRemotingServer` uses `DefaultEventExecutorGroup` for the pipeline and optionally configures `PooledByteBufAllocator.DEFAULT`. - Each channel receives a new `NettyDecoder` extending `LengthFieldBasedFrameDecoder`. - `NettyDecoder` calls `RemotingCommand.decode(frame)` and releases the extracted frame in `finally`. - `RemotingCommand.decode(ByteBuf)` copies the remaining body into a new `byte[]`. Questions for maintainers: 1. Is there a known issue in this 4.9.x pipeline that can cause receive-buffer data to be observed across independent channels? 2. Was a related fix made in a later RocketMQ or Netty version? 3. Are disabling the pooled server allocator, keeping decoding on the channel EventLoop, and enabling Netty leak detection appropriate diagnostic mitigations? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
