leizhiyuan opened a new issue, #1428: URL: https://github.com/apache/rocketmq-clients/issues/1428
### Before Creating the Bug Report - [x] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq-clients/discussions). - [x] I have searched the [GitHub Issues](https://github.com/apache/rocketmq-clients/issues) and [GitHub Discussions](https://github.com/apache/rocketmq-clients/discussions) of this repository and believe that this is not a duplicate. - [x] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ. ### Programming Language of the Client Rust ### Runtime Platform Environment macOS 27 (arm64) ### RocketMQ Version of the Client/Server Client: Rust `rocketmq` 5.0.0, master `a3d1de8b` Server: a managed RocketMQ 5.x service (Tencent Cloud TDMQ for RocketMQ 5), gRPC proxy endpoint ### Run or Compiler Version rustc 1.98.0 ### Describe the Bug The Rust client ignores `SystemProperties.body_encoding` and `SystemProperties.body_digest` of received messages. `MessageView::body()` returns the bytes exactly as received. A compressed body reaches the application still compressed, and a body that fails its digest check is delivered as if it were valid. This affects every consumer (`SimpleConsumer`, `PushConsumer`, `LitePushConsumer`) and the `MessageView` passed to the transaction checker, since they all go through `MessageView::from_pb_message`. **Scenario.** Mixed-client deployments are common during a migration to 5.x: producers still use the Remoting SDK (`rocketmq-client` 4.x / 5.x), and new consumers use the gRPC clients. The Remoting producer compresses bodies larger than `compressMsgBodyOverHowmuch` (4 KiB by default) with ZLIB. The proxy then marks the message as `Encoding::Gzip` (`GrpcConverter` sets `GZIP` whenever the stored message has `COMPRESSED_FLAG`, whatever the algorithm). A Rust consumer receives the compressed bytes, while a Java consumer of the same group receives the original text. Observed on a real server with a normal topic and group: | message | Rust today | Java 5.2.2 | |---|---|---| | 5090-byte text sent by the Remoting producer (`rocketmq-client` 5.3.1, default settings, compressed because > 4 KiB) | 472 bytes starting with `78 5e` (a ZLIB stream) | the original 5090-byte text | | 15-byte text sent by a gRPC producer with `body_encoding = GZIP` | the 35-byte gzip stream | the original 15-byte text | **Java behavior.** `MessageViewImpl#fromProtobuf`: 1. Verifies `body_digest` (CRC32 / MD5 / SHA1) on the body as received. The proxy computes the digest on the stored, i.e. still compressed, body. 2. Decompresses when `body_encoding == GZIP`. `Utilities#decompressBytes` picks the algorithm by its magic number (GZIP `1f 8b`, ZLIB `78`, LZ4, ZSTD), because the `GZIP` label is also used for ZLIB bodies. 3. Sets `corrupted = true` when the digest does not match or the body can not be decompressed (exposed as `MessageView#isCorrupted`). The push consumer does not pass a corrupted message to the listener: `StandardConsumeService` nacks it (`ProcessQueueImpl#discardMessage`) and `FifoConsumeService#getNextValidMessage` forwards it to the dead letter queue (`ProcessQueueImpl#discardFifoMessage`). A simple consumer returns it, and the application can check `isCorrupted()`. ### Steps to Reproduce 1. Send a message larger than 4 KiB to a normal topic with the Remoting Java producer (`DefaultMQProducer`, default settings). 2. Receive it with the Rust `SimpleConsumer` (or `PushConsumer`). 3. `message.body()` is a ZLIB stream, not the text that was sent. The Java 5.x `SimpleConsumer` receives the text. ### What Did You Expect to See? The same result as the Java client: - the decompressed body for a `GZIP`-encoded message (GZIP or ZLIB stream); - the digest checked, with a way to tell that a message is corrupted; - corrupted messages not passed to the push consumer's listener. ### What Did You See Instead? The compressed bytes, and no digest check. ### Additional Context Found while comparing the Rust and Java SimpleConsumer end to end on the same server. --- Tracked in #1412. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
