lhotari commented on code in PR #23583: URL: https://github.com/apache/pulsar/pull/23583#discussion_r1840633175
########## pip/pip-392.md: ########## @@ -0,0 +1,94 @@ +# PIP-392: Add configuration to enable consistent hashing to select active consumer for partitioned topic + +# Background knowledge + +After [#19502](https://github.com/apache/pulsar/pull/19502) will use consistent hashing to select active consumer for non-partitioned topic + +# Motivation + +Currently, for partitioned topics, the active consumer is selected using the formula [partitionedIndex % consumerSize](https://github.com/apache/pulsar/blob/137df29f85798b00de75460a1acb91c7bc25453f/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractDispatcherSingleActiveConsumer.java#L129-L130). +This method can lead to uneven distribution of active consumers. + +Consider a scenario with 100 topics named `public/default/topic-{0~100}`, each having `one partition`. +If 10 consumers are created using a `regex` subscription with the `Failover type`, all topic will be assigned to the same consumer(the first connected consumer). This results in an imbalanced distribution of consumers. + +# Goals + +## In Scope +- Address the issue of imbalance for `failover` subscription type consumers in single-partition or few-partition topics. Review Comment: @shibd Thanks for clarifying. Makes sense. PIP-392 won't make the problem worse in that case. One of my assumptions was that in the current solution there's "stickyness" in a way that the active consumer doesn't get switched unless it fails. I guess that's a false assumption from my side. Although it's out-of-scope for PIP-392, it would be great to start making progress in addressing the clear problem with Failover subscriptions which is documented as "In some cases, a partition may have an older active consumer processing messages while a newly switched over active consumer starts receiving new messages. This may lead to message duplication or out-of-order.". That goes against a lot of the expected guarantees for Failover subscriptions and causes problems. I found out that @grssam provided some feedback about this in the past in https://github.com/apache/pulsar-site/pull/633#issuecomment-1625416461 . It could be valuable if @grssam could also review PIP-392. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
