lhotari commented on code in PR #23583: URL: https://github.com/apache/pulsar/pull/23583#discussion_r1837014866
########## pip/pip-392.md: ########## @@ -0,0 +1,94 @@ +# PIP-392: Add configuration to enable consistent hashing to select active consumer for partitioned topic + +# Background knowledge + +After [#19502](https://github.com/apache/pulsar/pull/19502) will use consistent hashing to select active consumer for non-partitioned topic + +# Motivation + +Currently, for partitioned topics, the active consumer is selected using the formula [partitionedIndex % consumerSize](https://github.com/apache/pulsar/blob/137df29f85798b00de75460a1acb91c7bc25453f/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractDispatcherSingleActiveConsumer.java#L129-L130). +This method can lead to uneven distribution of active consumers. + +Consider a scenario with 100 topics named `public/default/topic-{0~100}`, each having `one partition`. +If 10 consumers are created using a `regex` subscription with the `Failover type`, all topic will be assigned to the same consumer(the first connected consumer). This results in an imbalanced distribution of consumers. + +# Goals + +## In Scope +- Address the issue of imbalance for `failover` subscription type consumers in single-partition or few-partition topics. Review Comment: This change will introduce a new problem: Let's say that there's a single connected consumer and another consumer connects, it could switch to use the new consumer. When this happens, there's an existing problem with failover and exclusive consumers where the outstanding messages aren't properly drained before switching. There's a stalled PIP, [PIP-260: Client consumer filter received messages](https://github.com/apache/pulsar/issues/19864), which is related to the issue, however it's not exactly describing the consumer switching scenario. For non-partitioned topics there's a reported issue #15189. After PIP-392 changes, a similar problem would also appear for partitioned topics. I guess addressing #15189 could happen in a separate PIP, but I'd suggest documenting this detail also in the PIP-392 document that it's a known consequence of the change. For consumer switching in failover subscriptions, there should be a way to ensure that pending messages are processed before switching the consumer. Otherwise it leads to duplicate message processing in clients which could be considered as out-of-order message processing from the client application perspective. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
