I put an answer to this on Stack Overflow. Basically, that's not how RF works for Kafka. It's not a guarantee, it's just how the partitions are created, and how it is reported when something is down (under replicated partitions). While there is an option to do auto leader rebalancing, there's no equivalent option for auto partition migration. It's a fairly resource intensive task, moving all that data around. If you want to move replicas around when something is down, you have to do it manually.
That said, it may be interesting to consider. -Todd On Tue, Sep 15, 2015 at 7:47 AM, Dhyan <d.muralidha...@yottaa.com> wrote: > Hi All, > > Below is my partition information for the topic **xx_json_topic** .This is > a Kafka cluster with three nodes . > > All nodes up : > > Topic: xx_json_topic PartitionCount:4 > ReplicationFactor:2 Configs: > Topic: xx_json_topic Partition: 0 Leader: 1 Replicas: > 3,1 Isr: 3,1 > Topic: xx_json_topic Partition: 1 Leader: 2 Replicas: > 1,2 Isr: 2,1 > Topic: xx_json_topic Partition: 2 Leader: 2 Replicas: > 2,3 Isr: 2,3 > Topic: xx_json_topic Partition: 3 Leader: 3 Replicas: > 3,2 Isr: 2,3 > > At this point.. if i bring down the node "node-1" ..It looks like below : > > Topic: xx_json_topic PartitionCount:4 > ReplicationFactor:2 Configs: > Topic: xx_json_topic Partition: 0 Leader: 3 Replicas: > 3,1 Isr: 3 > Topic: xx_json_topic Partition: 1 Leader: 2 Replicas: > 1,2 Isr: 2 > Topic: xx_json_topic Partition: 2 Leader: 2 Replicas: > 2,3 Isr: 2,3 > Topic: xx_json_topic Partition: 3 Leader: 3 Replicas: > 3,2 Isr: 2,3 > > My question is ..if kafka knows that the node-1 is down and it needs to > maintain the replication factor-2 ,wouldn't it make node 3 a replica for > partition-1 && node-2 a replica for partition-0 then make node-3 and > node-2 part of their Isr ? > > Or you think Kafka doesn't promise that... > If replication factor is 2 ..It doesn't mean that data will be available > in atleast 2 nodes at all time(---like consistency level in Cassandra) . > > I also have this question posted below : > > http://stackoverflow.com/questions/32588784/would-kafka-create-a-new-follower-if-one-of-its-replica-is-down-to-keep-up-with > < > http://stackoverflow.com/questions/32588784/would-kafka-create-a-new-follower-if-one-of-its-replica-is-down-to-keep-up-with > > > > —Dhyan
