geniusjoe opened a new pull request, #1511: URL: https://github.com/apache/pulsar-client-go/pull/1511
Master Issue: https://github.com/apache/pulsar/pull/21048 Related Issue: https://github.com/apache/pulsar-client-go/pull/805 ### Motivation In the Java SDK (PR https://github.com/apache/pulsar/pull/21048), the DLQ/RLQ producers are hardcoded with `enableBatching(false).enableChunking(true)` to ensure large chunked messages can be forwarded to DLQ/RLQ topics successfully. However, in the Go SDK, the DLQ/RLQ producers are created using `DLQPolicy.ProducerOptions`, which defaults to a zero-value `ProducerOptions{}` (i.e., `EnableChunking: false`, `DisableBatching: false`). This means that if a producer sends a chunked message larger than the broker's `maxMessageSize`, and the consumer is configured with DLQ/RLQ, the DLQ/RLQ producer will fail to forward the message because it cannot chunk it. We considered automatically enabling chunking in the DLQ/RLQ producers when the user has not explicitly configured `ProducerOptions`. However, since `EnableChunking` and `DisableBatching` are both `bool` fields with a zero value of `false`, there is no way to distinguish between: - The user intentionally setting them to `false` (meaning "I don't want chunking") - The user not configuring them at all (meaning "I didn't think about it") Automatically overriding these values would break backward compatibility for users who explicitly set `EnableChunking: false`. Therefore, instead of changing the default behavior, we add documentation to guide users to enable chunking in `DLQPolicy.ProducerOptions` when using chunked messages with DLQ/RLQ. ### Modifications - Added a note to the `EnableChunking` field documentation in `ProducerOptions` (`producer.go`) explaining that when chunking is enabled and the consumer uses DLQ/RLQ, it is recommended to also enable chunking in `DLQPolicy.ProducerOptions`. - Added a note to the `DLQPolicy.ProducerOptions` field documentation (`consumer.go`) explaining that if the messages being consumed were produced with chunking enabled, it is recommended to also enable chunking here. This helps in decoupled producer/consumer scenarios where the consumer developer may not be aware of the producer's chunking configuration. - Added `TestChunkReconsumeLater` integration test to verify chunked messages (5MB, exceeding broker maxMessageSize) can be sent to the retry topic via `ReconsumeLater` and routed to DLQ after exceeding max retries, when `DLQPolicy.ProducerOptions` has chunking enabled. - Added `TestChunkDLQWithNack` integration test to verify chunked messages (5MB) are routed to DLQ after exceeding max redelivery count via `Nack`, when `DLQPolicy.ProducerOptions` has chunking enabled. ### Verifying this change This change added tests and can be verified as follows: - Added `TestChunkReconsumeLater`: sends a 5MB chunked message, verifies it flows through retry topic via `ReconsumeLater` and eventually lands in DLQ after exceeding `MaxDeliveries`. - Added `TestChunkDLQWithNack`: sends a 5MB chunked message, verifies it is routed to DLQ after exceeding max redelivery count via `Nack`, with original properties preserved. Both tests use payloads larger than broker `maxMessageSize` (~1MB) to ensure that DLQ/RLQ producers **must** have chunking enabled to successfully forward messages. Without chunking enabled in `DLQPolicy.ProducerOptions`, these tests would timeout. ### Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API: no - The schema: no - The default values of configurations: no - The wire protocol: no ### Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? GoDocs - If a feature is not applicable for documentation, explain why? This PR adds documentation (GoDoc comments) to guide users on how to properly configure DLQ/RLQ when using chunked messages. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
