Hi all, Following the DISCUSS thread [1], I'd like to start the vote for PIP-498: Share retry and dead-letter producers across partitions of the same subscription.
Pull Request: https://github.com/apache/pulsar/pull/27129 Rendered: https://github.com/zjxxzjwang/pulsar-wzj/blob/master-retry-producer-pip/pip/pip-498.md Summary ------- For a subscription on a P-partition topic, each partition sub-consumer today owns its own retryLetterProducer and deadLetterProducer, even though all of them write to the same subscription-level *-RETRY / *-DLQ topic. For a 32-partition topic consumed by 10 pods this amounts to 640 broker-side producers for just 2 destination topics. This PIP moves retry/DLQ producer ownership from the per-partition ConsumerImpl up to the parent MultiTopicsConsumerImpl, collapsing N x P x 2 broker-side producers to N x 2 per subscription. A new abstraction RetryOrDlqProducerProvider is introduced, and MultiTopicsConsumerImpl holds one SharedRetryOrDlqProducerProvider per subscription-level topic and injects it into every per-partition sub-consumer at construction time. Full backward compatibility is preserved via a new opt-out ConsumerBuilder knob: ConsumerBuilder#shareRetryAndDeadLetterProducers(boolean) // default: true Setting it to false restores the legacy per-partition producer behavior for users who depend on per-partition retry/DLQ producer stats. The vote will be open for at least 72 hours. It is adopted by 3 "+1 binding" PMC votes and no "-1" votes. Please vote: [ ] +1, Approve the proposal [ ] 0, No opinion [ ] -1, Do not approve the proposal (please provide reasoning) Thanks, Zhejiang Wang [1] https://lists.apache.org/thread.html/011n4lglc0j01vx0jvshn3bhp94yt6rg
