Hi Kafka Community, I would like to start a discussion on a new KIP proposal: KIP-1151: Minimal movement replica balancing algorithm for reassignment
Motivation Kafka clusters require rebalancing of topic replicas after horizontal scaling to evenly distribute the load. However, the current approach does not consider the existing replica distribution, often leading to unnecessary movements. This KIP proposes an optimized strategy that minimizes movements while achieving an even distribution. Proposal The proposal focuses on: 1.Minimal Movement: Minimize the number of replica relocations during rebalancing. 2.Replica Balancing: Ensure that replicas are evenly distributed across brokers. 3.Anti-Affinity Support: Support rack-aware allocation when enabled. 4.Leader Balancing: Distribute leader replicas evenly across brokers. 5.ISR Order Optimization: Optimize adjacency relationships to prevent failover traffic concentration in case of broker failures. For a detailed description, please refer to the KIP document: KIP-1151: https://cwiki.apache.org/confluence/display/KAFKA/KIP- 1151%3A+Minimal+movement+replica+balancing+algorithm+for+reassignment <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Minimal+movement+replica+balancing+algorithm+for+reassignment> I would appreciate any feedback, comments, or suggestions from the community. Best
