Hi Lukasz, that sounds like a worthwhile addition to me! Just a few thoughts that occurred to me while reading your mail "Preferred Leader" as a term is somewhat taken in the Kafka namespace. It refers to the concept that the first leader from the list of replicas is the preferred leader and will be chosen as the leader for that partition if possible. Quite similar to what you are proposing - but without the logic behind it, as currently that preferred leader is chosen mostly at random.
If your need for this is urgent, you could fairly easily use the AdminClient to generate a new partition assignment that takes into account the rack ids of the brokers already, and Kafka would try to honor that preferred leader for your partitions. But this would only work retrospectively, not for new topics, for those a random distribution would again be chosen. Regarding your idea, I agree that replica and leader assignment is a topic that is in need of some love. However, by "just" adding the rack id to this logic we run the risk of making potential future work for things like load or size based replica assignment to brokers / disks harder. I'm not saying we need to do it now, but I think we should consider what it might look like to ensure that your solution can lay to groundwork for later work to build on. Best regards, Sönke On Tue, 14 Apr 2020 at 23:32, Michael K. Edwards <m.k.edwa...@gmail.com> wrote: > #nailedit > > On Tue, Apr 14, 2020 at 2:05 PM Jamie <jamied...@aol.co.uk> wrote: > > > Hi All, > > > > There is a KIP which might be of interest to you: > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120736982 > - > > it sounds like you want to blacklist brokers in DC3? > > > > Thanks, > > > > Jamie > > > > -----Original Message----- > > From: Michael K. Edwards <m.k.edwa...@gmail.com> > > To: dev@kafka.apache.org > > Sent: Tue, 14 Apr 2020 20:50 > > Subject: Re: Preferred Partition Leaders > > > > I have clients with a similar need relating to disaster recovery. (Three > > replicas per partition within a data center / AWS AZ/region, fourth > replica > > elsewhere, ineligible to become the partition leader without manual > > intervention.) > > > > On Tue, Apr 14, 2020 at 12:31 PM Łukasz Antoniak < > > lukasz.anton...@gmail.com> > > wrote: > > > > > Hi Everyone, > > > > > > Recently I came across Kafka setup where two data centers are close to > > each > > > other, but the company could not find a suitable place for the third > one. > > > As a result third DC is little further, lower network throughput, but > > still > > > within range of decent network latency, qualifying for stretch cluster. > > Let > > > us assume that client applications are being deployed only on two > > "primary" > > > DCs. My idea was to minimize network traffic between DC3 and other data > > > centers (ideally only to replication). > > > > > > For Kafka consumer, we can configure rack-awareness, so that consumers > > will > > > read data from closest replica (replica.selector.class). > > > Kafka producers have to send data to partition leaders. There is no way > > to > > > tell that we prefer replica leaders to be running in DC1 and DC2. Kafka > > > will also try to evenly balance leaders across brokers > > > (auto.leader.rebalance.enable). > > > > > > Does it sound like a good feature to make the choice of partition > leaders > > > pluggable? Basically, users would be given list of topic-partitions > with > > > ISRs and rack they are running, and could reshuffle them according to > > > custom logic. > > > > > > Comments appreciated. > > > > > > Kind regards, > > > Lukasz > > > > > > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
