Hi all,

Following the DISCUSS thread [1], I'd like to start the vote for PIP-498:
Share retry and dead-letter producers across partitions of the same
subscription.

Pull Request:
  https://github.com/apache/pulsar/pull/27129

Rendered:

https://github.com/zjxxzjwang/pulsar-wzj/blob/master-retry-producer-pip/pip/pip-498.md

Summary
-------
For a subscription on a P-partition topic, each partition sub-consumer today
owns its own retryLetterProducer and deadLetterProducer, even though all of
them write to the same subscription-level *-RETRY / *-DLQ topic. For a
32-partition topic consumed by 10 pods this amounts to 640 broker-side
producers for just 2 destination topics.

This PIP moves retry/DLQ producer ownership from the per-partition
ConsumerImpl up to the parent MultiTopicsConsumerImpl, collapsing
N x P x 2 broker-side producers to N x 2 per subscription. A new abstraction
RetryOrDlqProducerProvider is introduced, and MultiTopicsConsumerImpl holds
one SharedRetryOrDlqProducerProvider per subscription-level topic and
injects it into every per-partition sub-consumer at construction time.

Full backward compatibility is preserved via a new opt-out ConsumerBuilder
knob:

  ConsumerBuilder#shareRetryAndDeadLetterProducers(boolean)   // default:
true

Setting it to false restores the legacy per-partition producer behavior for
users who depend on per-partition retry/DLQ producer stats.

The vote will be open for at least 72 hours. It is adopted by 3 "+1 binding"
PMC votes and no "-1" votes.

Please vote:

  [ ] +1, Approve the proposal
  [ ]  0, No opinion
  [ ] -1, Do not approve the proposal (please provide reasoning)

Thanks,
Zhejiang Wang

[1] https://lists.apache.org/thread.html/011n4lglc0j01vx0jvshn3bhp94yt6rg

Reply via email to