Hi Mazzen, Have you looked into incremental cooperative rebalancing? It may help with your issues, at least it can avoid stop the world rebalances.
https://www.confluent.io/blog/incremental-cooperative-rebalancing-in-kafka/ Cheers, Liam Clarke On Mon, 29 Mar. 2021, 8:04 pm Mazen Ezzeddine, < [email protected]> wrote: > Hi all, > Given a replicaset/statefulset of kafka consumers that constitute a > consumer group running on kubernetes, if the number of replicas is x than > sometimes x rebalancing might be triggered since not all of the > replicas/consumers send a join group request in a timely and synced manner > to the group coordinator... This also happens when we perform scale up/down > of consumers which might result in multiple rebalancing rounds… > If we manage that each consumer in the consumer group send along with the > join group request the number of replicas of the replicaset in the > subscriptionUserData (using the kubernetes client API) , and then group > cordinator/server wait until the specified number of replicas join (or > timeout) before launching a rebalance? would that work to strict the number > of rebalancing to one, any hint please? > Thank you. > >
