nodece commented on code in PR #24022:
URL: https://github.com/apache/pulsar/pull/24022#discussion_r1970908673
##########
pip/pip-409.md:
##########
@@ -0,0 +1,73 @@
+
+# PIP-409: support producer configuration for retry/dead letter topic producer
+
+# Background knowledge
+
+Retry topic is a topic that stores messages that have failed to be processed
by the consumer, or the consumer can't process the message at the moment.
+The consumer can retry the message after a certain period of time. After
several retries, the message will be moved to the dead letter topic.
+
+There is a retry topic producer in consumer instance. Each time the consumer
call `consumer.reconsumeLater(msg, 3, TimeUnit.SECONDS);`
+to retry the message after 3 seconds, the hidden producer will send the
corresponding message to retry topic, and ack the message in original topic.
+
+After several retries, the message will be sent to dead letter topic instead
of retry topic.
+
+
+# Motivation
+
+Currently, we don't support configure the producer of retry/dead letter topic.
But enable the chunk message feature
+and disable the batch configuration in hard code, which can't handle many
situations. For example, when the throughput
+of message of retry topic become considerable, the resource consumed by the
un-batched messages is pretty large.
+There is no reason that we disable the batch message feature.
+
+For better control for the retry/dead letter topic feature, we can support
configuration for the producer of
+retry/dead letter topic.
+
+# Goals
+
+## In Scope
+
+- Support configuration for the producer of retry/dead letter topic.
+
+
+# Detailed Design
+
+## Design & Implementation Details
+
+- add a new class `DeadLetterProducerConfig`.
+Instead of reusing `ProducerConfigurationData`, we introduce a new config
class. First of all, this is because `ProducerConfigurationData`
+lies in the `pulsar-client-original` module, and we don't want to introduce it
as a new dependency in the `pulsar-client-api` module.
+Secondly, we actually don't need all the configurations in
`ProducerConfigurationData` for the producer of retry/dead letter topic.
+
+- add two new configurations in `DeadLetterPolicy`, both of them are of type
`DeadLetterProducerConfig`.
+ - `retryTopicProducerConfig`: the configuration for the producer of retry
topic.
+ - `deadLetterProducerConfig`: the configuration for the producer of dead
letter topic.
+
+```java
+public class DeadLetterPolicy implements Serializable {
+ /**
+ * Configuration used to create a producer that will send messages to the
dead letter topic.
+ */
+ private DeadLetterProducerConfig deadLetterProducerConfig;
Review Comment:
Please see implement: https://github.com/apache/pulsar/pull/24020/
The `DeadLetterProducerConfig` does not extend `ProducerConfigurationData`
because they are in different packages (impl/api).
As a result, a separate `DeadLetterProducerConfig` class was introduced.
However, there is concern that over time, this class could become functionally
equivalent to `ProducerConfigurationData`. To address this, I suggest that a
Function is used to apply the configuration and return a new `ProducerBuilder`,
which avoids direct inheritance and maintains flexibility.
```java
public class DeadLetterPolicy {
// ....
private java.util.function.Function<ProducerBuilder<?>, ProducerBuilder<?>>
deadLetterProducerConfig;
private java.util.function.Function<ProducerBuilder<?>, ProducerBuilder<?>>
retryLetterProducerConfig;
}
```
In the `ConsumerImp`:
```java
ProducerBuilder<T> apply = (ProducerBuilder<T>)
this.deadLetterPolicy.getRetryLetterProducerConfig()
.apply(client.newProducer(schema)
.topic(this.deadLetterPolicy.getRetryLetterTopic())
.enableBatching(false)
.blockIfQueueFull(false));
retryLetterProducer = apply.create();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
