Here's the algorithm as described in AdminUtils.scala: /** * There are 2 goals of replica assignment: * 1. Spread the replicas evenly among brokers. * 2. For partitions assigned to a particular broker, their other replicas are spread over the other brokers. * * To achieve this goal, we: * 1. Assign the first replica of each partition by round-robin, starting from a random position in the broker list. * 2. Assign the remaining replicas of each partition with an increasing shift. * * Here is an example of assigning * broker-0 broker-1 broker-2 broker-3 broker-4 * p0 p1 p2 p3 p4 (1st replica) * p5 p6 p7 p8 p9 (1st replica) * p4 p0 p1 p2 p3 (2nd replica) * p8 p9 p5 p6 p7 (2nd replica) * p3 p4 p0 p1 p2 (3nd replica) * p7 p8 p9 p5 p6 (3nd replica) */
So, assignment is partially random (we start at a random position) and then round-robin to spread replicas and partitions out. The information is persisted in ZooKeeper. Hope this helps. Gwen On Mon, Apr 13, 2015 at 12:15 PM, Bill Hastings <bllhasti...@gmail.com> wrote: > Thanks for the reference. But it doesn't seem to cover how a particular > topic is assigned to a particular broker and also how the replicas are > chosen. For eg if I have brokers A B C D and E what algorithm is used to > assign a topic X and partition 1 to brokers B D and E if the chosen > replication factor is 3. Is this information persisted somewhere? Sorry > just getting my feet wet. > > On Mon, Apr 13, 2015 at 11:57 AM, Jiangjie Qin <j...@linkedin.com.invalid> > wrote: > >> A quick reference. >> http://www.slideshare.net/junrao/kafka-replication-apachecon2013 >> >> >> On 4/12/15, 11:36 PM, "Bill Hastings" <bllhasti...@gmail.com> wrote: >> >> >Hi Guys >> > >> >How do topics get assigned to brokers? I mean if I were to create a topic >> >X >> >and publish to it how does Kafka assign the topic and the message to a >> >particular broker? If I have create a topic with say 10 partitions how >> >does >> >kafka assign each partition to a different broker? >> > >> >-- >> >Cheers >> >Bill >> >> > > > -- > Cheers > Bill
