Hey Joe, 1. Could you add details to the Public Interface section of the KIP? This should include the proposed changes to the partition reassignment tool. Also, maybe the new option can be named --rebalance instead of --re-balance? 2. It makes sense to list --decommission-broker as part of this KIP. Similarly, shouldn't we also have an --add-broker option? The way I see this is that there are several events when a partition reassignment is required. Before this functionality is automated on the broker, the tool will generate an ideal replica placement for each such event. The users should merely have to specify the nature of the event e.g. adding a broker or decommissioning an existing broker or merely rebalancing. 3. If I understand the KIP correctly, the upgrade plan for this feature includes removing the existing --generate option on the partition reassignment tool in 0.8.3 while adding all the new options in the same release. Is that correct?
Thanks, Neha On Thu, Jan 22, 2015 at 9:23 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > Ditto on this one. Can you give the algorithm we want to implement? > > Also I think in terms of scope this is just proposing to change the logic > in ReassignPartitionsCommand? I think we've had the discussion various > times on the mailing list that what people really want is just for Kafka to > do it's best to balance data in an online fashion (for some definition of > balance). i.e. if you add a new node partitions would slowly migrate to it, > and if a node dies, partitions slowly migrate off it. This could > potentially be more work, but I'm not sure how much more. Has anyone > thought about how to do it? > > -Jay > > On Wed, Jan 21, 2015 at 10:11 PM, Joe Stein <joe.st...@stealth.ly> wrote: > > > Posted a KIP for --re-balance for partition assignment in reassignment > > tool. > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-6+-+New+reassignment+partition+logic+for+re-balancing > > > > JIRA https://issues.apache.org/jira/browse/KAFKA-1792 > > > > While going through the KIP I thought of one thing from the JIRA that we > > should change. We should preserve --generate to be existing functionality > > for the next release it is in. If folks want to use --re-balance then > > great, it just won't break any upgrade paths, yet. > > > > /******************************************* > > Joe Stein > > Founder, Principal Consultant > > Big Data Open Source Security LLC > > http://www.stealth.ly > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > > ********************************************/ > > > -- Thanks, Neha
