Geoff, First, if you haven't already, take a look at https://cwiki.apache.org/confluence/display/KAFKA/KIP-6+-+New+reassignment+partition+logic+for+rebalancing and the associated JIRA.
My opinion is that we shouldn't do anything to the existing partitions -- it would probably violate the principle of least surprise if you started moving data around when all I asked you to do was add new partitions. I think this aligns well with the basic approach of KIP 6 (which is not approved yet, but I think folks are generally in favor of once it is fully specified). The goal is to move as little data as possible, but achieve the best balance. I think if the user wanted any *other* partitions rebalanced, they should use the reassignment tool, which KIP 6 will hopefully make a lot easier soon. But at a minimum, adding the new partitions should try not to make the balance any worse than it already is. Not sure what this means for fixing the JIRA though -- reusing what KIP 6 is doing but applying it to the new partitions seems like the ideal solution. The current createTopic code doesn't take into account existing assignments, but I don't think it makes sense to duplicate effort between the two patches. I'm not sure what the timeframe of KIP 6, but if its not too far away you might just mark it as blocking KAFKA-1313, prepare a patch against the KAFKA-1792's latest version, and maybe update KAFKA-1313's fix version to 0.8.3 to match KAFKA-1792's. -Ewen On Mon, Mar 9, 2015 at 3:14 PM, Geoffrey Anderson <ge...@confluent.io> wrote: > Hi dev list, I have a few questions regarding KAFKA-1313 ( > https://issues.apache.org/jira/browse/KAFKA-1313) > > It is currently possible albeit in a kindof hacky way to increase > replication factor on a topic by using the partition reassignment tool with > a hand-crafted reassignment plan. > > The preferred approach to making this better is to add support for > replication-factor in the alter-topic command. For example, > > bin/kafka-topics.sh --zookeeper localhost:2181 --topic test --alter > --replication-factor 2 > > > Another approach is use the partition reassignment tool to auto-generate a > reassignment plan which results in increased replication. > > The advantage of the alter topics approach is that it seems logical to > group this functionality with the other alter topic stuff. However, a > couple questions: > - Should partitions automatically be reassigned to brokers and shuffled > when the replication factor is changed? > - If so, it would be best to be able to update the replication factor of > multiple topics at once so that the automatic reassignment would take place > only once for the whole batch of changes. > > One advantage of the partition reassignment approach is that it's easier to > generate a reassignment plan which updates multiple topics at once, and you > have a chance to review the reassignment plan before putting it into effect > with the --execute flag. > > Thanks, > Geoff > -- Thanks, Ewen
