Hi Tom, You're right, I had had a complete brain fade on this.
Turns out that the script I wrote had a logical error - while trying to ensure every zone had a replica, it went through zones in this same order, thus unintentionally ensuring that the two brokers in zone A were quite often the first replica in the list, hence preferred leader for 80% of the partitions... ...and I think this incident has very much sold my team on Cruise Control. Thanks for the reply :) Liam Clarke On Tue, 20 Oct. 2020, 11:49 pm Tom Bentley, <tbent...@redhat.com> wrote: > Hi Liam, > > I think you have a misunderstanding of what preferred leader election does. > All it does is ensure that the "preferred leader" (the first in the list of > replicas for a partition) becomes the actual leader if it can (that is, if > it's in the ISR) and if the current leader is already the preferred leader > you get ElectionNotNeededException. It *doesn't* change the brokers which > are replicating a partition, or change which broker is the preferred > leader. So if the broker assignments are unbalanced, or ignore rack > constraints it doesn't help. > > Obviously you can use kafka-reassign-partitions.sh to change how partitions > are assigned to brokers, including which is the preferred leader, in order > to ensure the cluster is better balanced. But it's up to you to figure out > how to assign replicas to the brokers to achieve that (i.e. for each > partition come up with the list of brokers which should have a replica, > with the preferred one being first in the list). As you note with the > racks, in general there are several criteria to consider and finding a good > assignment is an exercise in constrained optimization. So people often use > external tools to find "good" assignments according to their criteria, and > often to manage the reassignment process too. There are several open source > tools available, such as Cruise Control. > > Hope this helps, > > Tom > > On Fri, Oct 16, 2020 at 7:05 AM Liam Clarke-Hutchinson < > liam.cla...@adscale.co.nz> wrote: > > > And to follow up on this, we rolled one of the 41% brokers, partition > > leadership was redistributed to other brokers, but a preferred leader > > election immediately led to the exact same distribution. > > > > What's worse, is both of the 41% leadership brokers are in the same rack. > > > > Very keen for advice, or in the worst case, recommendations on how to > > manually elect leaders for a topic-partition to get us through the > weekend. > > > > Kind regards, > > > > Liam > > > > On Fri, 16 Oct. 2020, 6:22 pm Liam Clarke-Hutchinson, < > > liam.cla...@adscale.co.nz> wrote: > > > > > Kia ora, > > > > > > I am rather bemused by this one, I'll admit. Kafka version 2.4.0. We've > > > been migrating topic-partitions from old broker machines to new > brokers, > > > have redistributed replicas evenly across the new brokers, removed the > > old > > > brokers, and tried to run a preferred leader election. > > > > > > And when I did that, I got a "LeaderElectionNotNeededException", which > I > > > really disagree with, as two brokers out of six have 41% of partition > > > leaders. > > > > > > I wonder if it's rack awareness messing with us or something? We are > > > running two brokers in three AZs/racks with correctly configured > > rack.ids, > > > but what are the origins of a "LeaderElectionNotNeededException"? And > for > > > the love of God, where is the parameter in the admin client for "I > > > significantly disagree". > > > > > > Kind regards, > > > > > > Liam Clarke-Hutchinson > > > > > >
