shibd commented on code in PR #23583: URL: https://github.com/apache/pulsar/pull/23583#discussion_r1840387783
########## pip/pip-392.md: ########## @@ -0,0 +1,94 @@ +# PIP-392: Add configuration to enable consistent hashing to select active consumer for partitioned topic + +# Background knowledge + +After [#19502](https://github.com/apache/pulsar/pull/19502) will use consistent hashing to select active consumer for non-partitioned topic + +# Motivation + +Currently, for partitioned topics, the active consumer is selected using the formula [partitionedIndex % consumerSize](https://github.com/apache/pulsar/blob/137df29f85798b00de75460a1acb91c7bc25453f/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractDispatcherSingleActiveConsumer.java#L129-L130). +This method can lead to uneven distribution of active consumers. + +Consider a scenario with 100 topics named `public/default/topic-{0~100}`, each having `one partition`. +If 10 consumers are created using a `regex` subscription with the `Failover type`, all topic will be assigned to the same consumer(the first connected consumer). This results in an imbalanced distribution of consumers. + +# Goals + +## In Scope +- Address the issue of imbalance for `failover` subscription type consumers in single-partition or few-partition topics. Review Comment: > The way how this PR introduces more frequent problems is that the consumer will switch frequently when new consumers get added. A rolling restart is a common problematic case. This is why the PIP document should at least document that this is a known problem and consequence of this change. If I understand correctly, the modulo algorithm also has this situation. Assuming there are 5 partitions with indices partitionIndex: [0, 1, 2, 3, 4] #### 1. The first consumer-0 joins, partitions selected: - consumer-0: [0, 1, 2, 3, 4] ``` partitionIndex % consumerSize = consumerIndex 0 % 1 = 0 1 % 1 = 0 2 % 1 = 0 3 % 1 = 0 4 % 1 = 0 ``` #### 2. When the second consumer-1 joins, partitions selected: - consumer-0: [0, 2, 4] - consumer-1: [1, 3] ``` partitionIndex % consumerSize = consumerIndex 0 % 2 = 0 1 % 2 = 1 2 % 2 = 0 3 % 2 = 1 4 % 2 = 0 ``` `!!`**!!P1 and P3** also get transferred #### 3. When the third consumer-2 joins, partitions selected: - consumer-0: [0, 3] - consumer-1: [1, 4] - consumer-2: [2] ``` partitionIndex % consumerSize = consumerIndex 0 % 3 = 0 1 % 3 = 1 2 % 3 = 2 3 % 3 = 0 4 % 3 = 1 ``` `!!`**P2, P3 and P4** also get transferred Altogether transferred 1. P0: c0 1. P1 from c0 -> c1 2. P2 from c0 -> c2 3. P3 from c0 -> c1 -> c0 4. P4 from c0 -> c1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
