> > 1. Replica Throttling - I agree this is rather important to get done. > However, it may also be argued that this problem is orthogonal. We do not > have these protections currently yet we do run partition reassignment > fairly often. Having said that, I'm perfectly happy to tackle KIP-46 after > this problem is solved. I understand it is actively being discussed in > KAFKA-1464.
I think we are saying the same thing here. Replica throttling is required to be able to pull off any partition reassignment action. It removes the guesswork that comes from picking a batch size that is expressed in terms of partition count, which is an annoying hack. > 2. Pluggable policies - Can you elaborate on the need for pluggable > policies in the partition reassignment tool? Even if we make it pluggable > to begin with, this needs to ship with a default policy that makes sense > for most users. IMO, partition count is the most intuitive default and is > analogous to how we stripe partitions for new topics. Agree about the default. I was arguing for making it pluggable so we make it easy to test multiple policies. For instance, partition count is a decent one but I can imagine how one would want a policy that optimizes for balancing data sizes for instance. > 3. Even if the trigger were fully manual (as it is now), we could still > have the controller generate the assignment as per a configured policy i.e. > effectively the tool is built into Kafka itself. Following this approach to > begin with makes it easier to fully automate in the future since we will > only need to automate the trigger later. I would be much more comfortable adding the capability to move large amounts of data to the controller, after we are very sure that the default policy is well tested and the replica throttling works. If so, then it is just a matter of placing the trigger in the controller vs in the tool. But I'm skeptical of adding more things to the already messy controller, especially without being sure about how well it works. Thanks, Neha On Tue, Feb 9, 2016 at 12:53 PM, Aditya Auradkar < aaurad...@linkedin.com.invalid> wrote: > Hi Neha, > > Thanks for the detailed reply and apologies for my late response. I do have > a few comments. > > 1. Replica Throttling - I agree this is rather important to get done. > However, it may also be argued that this problem is orthogonal. We do not > have these protections currently yet we do run partition reassignment > fairly often. Having said that, I'm perfectly happy to tackle KIP-46 after > this problem is solved. I understand it is actively being discussed in > KAFKA-1464. > > 2. Pluggable policies - Can you elaborate on the need for pluggable > policies in the partition reassignment tool? Even if we make it pluggable > to begin with, this needs to ship with a default policy that makes sense > for most users. IMO, partition count is the most intuitive default and is > analogous to how we stripe partitions for new topics. > > 3. Even if the trigger were fully manual (as it is now), we could still > have the controller generate the assignment as per a configured policy i.e. > effectively the tool is built into Kafka itself. Following this approach to > begin with makes it easier to fully automate in the future since we will > only need to automate the trigger later. > > Aditya > > > > On Wed, Feb 3, 2016 at 1:57 PM, Neha Narkhede <n...@confluent.io> wrote: > > > Adi, > > > > Thanks for the write-up. Here are my thoughts: > > > > I think you are suggesting a way of automating resurrecting a topic’s > > replication factor in the presence of a specific scenario: in the event > of > > permanent broker failures. I agree that the partition reassignment > > mechanism should be used to add replicas when they are lost to permanent > > broker failures. But I think the KIP probably chews off more than we can > > digest. > > > > Before we automate detection of permanent broker failures and have the > > controller mitigate through automatic data balancing, I’d like to point > out > > that our current difficulty is not that but the ability to generate a > > workable partition assignment for rebalancing data in a cluster. > > > > There are 2 problems with partition rebalancing today: > > > > 1. Lack of replica throttling for balancing data: In the absence of > > replica throttling, even if you come up with an assignment that might > be > > workable, it isn’t practical to kick it off without worrying about > > bringing > > the entire cluster down. I don’t think the hack of moving partitions > in > > batches is effective as it at-best a best guess. > > 2. Lack of support for policies in the rebalance tool that > automatically > > generate a workable partition assignment: There is no easy way to > > generate > > a partition reassignment JSON file. An example of a policy is “end up > > with > > an equal number of partitions on every broker while minimizing data > > movement”. There might be other policies that might make sense, we’d > > have > > to experiment. > > > > Broadly speaking, the data balancing problem is comprised of 3 parts: > > > > 1. Trigger: An event that triggers data balancing to take place. > KIP-46 > > suggests a specific trigger and that is permanent broker failure. But > > there > > might be several other events that might make sense — Cluster > expansion, > > decommissioning brokers, data imbalance > > 2. Policy: Given a set of constraints, generate a target partition > > assignment that can be executed when triggered. > > 3. Mechanism: Given a partition assignment, make the state changes and > > actually move the data until the target assignment is achieved. > > > > Currently, the trigger is manual through the rebalance tool, there is no > > support for any viable policy today and we have a built-in mechanism > that, > > given a policy and upon a trigger, moves data in a cluster but does not > > support throttling. > > > > Given that both the policy and the throttling improvement to the > mechanism > > are hard problems and given our past experience of operationalizing > > partition reassignment (required months of testing before we got it > right), > > I strongly recommend attacking this problem in stages. I think a more > > practical approach would be to add the concept of pluggable policies in > the > > rebalance tool, implement a practical policy that generates a workable > > partition assignment upon triggering the tool and improve the mechanism > to > > support throttling so that a given policy can succeed without manual > > intervention. If we solved these problems first, the rebalance tool would > > be much more accessible to Kafka users and operators. > > > > Assuming that we do this, the problem that KIP-46 aims to solve becomes > > much easier. You can separate the detection of permanent broker failures > > (trigger) from the mitigation (above-mentioned improvements to data > > balancing). The latter will be a native capability in Kafka. Detecting > > permanent hardware failures is much easily done via an external script > that > > uses a simple health check. (Part 1 of KIP-46). > > > > I agree that it will be great to *eventually* be able to fully automate > > both the trigger as well as the policies while also improving the > > mechanism. But I’m highly skeptical of big-bang approaches that go from a > > completely manual and cumbersome process to a fully automated one, > > especially when that involves large-scale data movement in a running > > cluster. Once we stabilize these changes and feel confident that they > work, > > we can push the policy into the controller and have it automatically be > > triggered based on different events. > > > > Thanks, > > Neha > > > > On Tue, Feb 2, 2016 at 6:13 PM, Aditya Auradkar < > > aaurad...@linkedin.com.invalid> wrote: > > > > > Hey everyone, > > > > > > I just created a kip to discuss automated replica reassignment when we > > lose > > > a broker in the cluster. > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-46%3A+Self+Healing+Kafka > > > > > > Any feedback is welcome. > > > > > > Thanks, > > > Aditya > > > > > > > > > > > -- > > Thanks, > > Neha > > > -- Thanks, Neha
