merlimat commented on issue #6791:
URL: https://github.com/apache/pulsar/pull/6791#issuecomment-618531886
> I see there is a problem in the current algorithm when scaling down the
number of consumers. But I don't think its broken.
The Key Shared subscriptions has 2 stated goals:
1. Distribute the load evenly across available consumers
2. Retain ordering
The current implementation cannot guarantee either of the 2:
1. Apart from the scale down I've seen 1 consumer out of 10 being active
after a rolling restart
2. A simple sequence of "add new consumer" and "remove a different
consumer" is breaking the ordering guarantee.
> So the extreme case you described in the description probably wasn't
really noticed.
The first user I saw using this feature for a very simple use case kept
going into this state after each rolling restart.
For sure, bugs are always around the corner, though once identified we
should should strive to fix them.
> If you have a better algorithm to resolve the key distribution and the
screw problem during split-and-merge,
That's what I said: There is no easy way to do it in that mode and that's
the reason why the textbook solution for this kind of problems is to use
consistent hashing. This is not something I came up with.
So, if you're asking "do you have a better algorithm to implement the goals
of KeyShared subscription", the answer is: yes, consistent hashing.
Instead, if the question is "do you have a better algorithm to resolve the
key distribution and the screw problem during split-and-merge?", my answer is
no and: why is that a requirement? We just need one solution that fulfills the
2 goals (traffic distribution and ordering).
Going back to the original question on having a configuration switch. We can
do that if there are concerns on this implementation, though I think that, at
least by default, the behavior should not have "known" blind spots.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
