EugenDueck opened a new pull request #6209: [broker] add feature BrokerDeduplicationAcrossProducers URL: https://github.com/apache/pulsar/pull/6209 ### Motivation *Deduplication of messages works per producer. But in case **redundant producers**, i.e. multiple producers that produce the same messages with the same sequence ids in an active/active, what is needed is deduplication across producers.* ### Modifications *Added a configuration `brokerDeduplicationAcrossProducersEnabled` to `pulsar-broker-common/.../broker/ServiceConfiguration`, which, when set to true (default is false), will make deduplication of work across producers, by ignoring which producers sent the messages. More specifically, instead of using the producerName as part of the key to store the last seen sequence id, a constant value `"__all"` gets used for all producers.* ### Verifying this change *This change added unit tests and can be verified as follows:* - `testIsDuplicateAcrossProducers` (adapting the existing `testIsDuplicate`) - `testIsDuplicateAcrossProducersWithFailure` (adapting the existing `testIsDuplicateWithFailure`) ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - The admin cli options: (yes / no) ### Documentation *This is a new feature that needs to be documented. I will do so as part of this PR after getting and if necessary incorporating feedback for this feature.*
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
