Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

On Thu, Jun 27, 2019, at 08:58, Jason Gustafson wrote: > > > > We'd remove nodes from targetReplicas just as soon as they entered the > > ISR. They would become regular replicas at that point. > > > I think we can save a lot of back and forth by working through an example. > Suppose we have the

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

> > We'd remove nodes from targetReplicas just as soon as they entered the > ISR. They would become regular replicas at that point. I think we can save a lot of back and forth by working through an example. Suppose we have the following initial state: replicas: [1, 2, 3] isr: [1, 2, 3] targetRe

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

Hey Colin, So in my understanding this is how the controller handles a reassignment in a simple scenario: 1. When an AlterPartitionReassignmentRequest arrives it updates the partitions' ZK data in /brokers/topics/[topic]/partitions/[partitionId]/state with targetReplicas 2. Sends out LeaderAndIsr

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

On Wed, Jun 26, 2019, at 12:02, Jason Gustafson wrote: > Hi Colin, > > Responses below and another question: > > > I guess the thought process here is that most reassignment tools want to > > know about all the reassignments that are going on. If you don't know all > > the pending reassignments,

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

Hi Colin, Responses below and another question: I guess the thought process here is that most reassignment tools want to > know about all the reassignments that are going on. If you don't know all > the pending reassignments, then it's hard to say whether adding a new one > is a good idea, or ca

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

On Tue, Jun 25, 2019, at 18:37, Jason Gustafson wrote: > Hi Colin, > > Took another pass on the KIP. Looks good overall. A few questions below: > > 1. I wasn't clear why `currentReplicas` is an optional field. Wouldn't we > always have a current set of replicas? Good point. When I wrote that I

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

On Wed, Jun 19, 2019, at 03:36, Stanislav Kozlovski wrote: > Hey there Colin, > > Thanks for the work on this KIP. It is a much-needed improvement and I'm > excited to see it. Sorry for coming in so late to the discussion, I have > one question to better understand the change and a small suggestio

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

Hi Colin, Took another pass on the KIP. Looks good overall. A few questions below: 1. I wasn't clear why `currentReplicas` is an optional field. Wouldn't we always have a current set of replicas? 2. Seems the only option is to list all active partition reassignments? I think we have tended to re

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

Hey there Colin, Thanks for the work on this KIP. It is a much-needed improvement and I'm excited to see it. Sorry for coming in so late to the discussion, I have one question to better understand the change and a small suggestion. I see we allow reassignment cancellation at the partition level -

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

On Mon, May 6, 2019, at 07:39, Ismael Juma wrote: > Hi Colin, > > A quick comment. > > On Sat, May 4, 2019 at 11:18 PM Colin McCabe wrote: > > > The big advantage of doing batching on the controller is that the > > controller has more information about what is going on in the cluster. So > > i

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

Hi Colin, A quick comment. On Sat, May 4, 2019 at 11:18 PM Colin McCabe wrote: > The big advantage of doing batching on the controller is that the > controller has more information about what is going on in the cluster. So > it can schedule reassignments in a more optimal way. For instance, i

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

Hi Colin, Thanks for explaining all this, it makes sense. Viktor On Sun, May 5, 2019 at 8:18 AM Colin McCabe wrote: > On Thu, May 2, 2019, at 09:35, Viktor Somogyi-Vass wrote: > > Hey Colin & George, > > > > Thinking on George's points I was wondering if it's feasible to submit a > > big reass

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

On Thu, May 2, 2019, at 09:35, Viktor Somogyi-Vass wrote: > Hey Colin & George, > > Thinking on George's points I was wondering if it's feasible to submit a > big reassignment to the controller and thus Zookeeper as frequent writes > are slow as the quorum has to synchronize. Perhaps it should be

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

On Tue, Apr 30, 2019, at 23:33, George Li wrote: > Hi Colin, > > Thanks for KIP-455!  yes. KIP-236, etc. will depend on it.  It is the > good direction to go for the RP  > > Regarding storing the new reassignments & original replicas at the > topic/partition level.  I have some concerns when c

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

Hey Colin & George, Thinking on George's points I was wondering if it's feasible to submit a big reassignment to the controller and thus Zookeeper as frequent writes are slow as the quorum has to synchronize. Perhaps it should be the responsibility of KIP-435

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

Hi Colin, Thanks for KIP-455!  yes. KIP-236, etc. will depend on it.  It is the good direction to go for the RPC.  Regarding storing the new reassignments & original replicas at the topic/partition level.  I have some concerns when controller is failing over, and the scalability of scanning t

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

Hi Viktor, Good question. The answer is that the Admin Client knows how to send messages to the controller node. Take a look at electPreferredLeaders, createTopics, deleteTopics, etc. for some examples of how this is done in the code. kafka-reassign-partitions.sh would be able to simply use t

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

Hi Colin, How would the changed kafka-reassign-partitions.sh command work? Would it automatically figure out the controller node if you pass some bootstrap nodes with --bootstrap-server or are we deferring this implementation to the users? Thanks, Viktor On Sat, Apr 20, 2019 at 1:51 AM Colin McC

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

On Wed, Apr 17, 2019, at 17:23, Robert Barrett wrote: > Thanks for the KIP, Colin. I have a couple questions: > > 1. What's the reasoning for requiring cancellation of a reassignment before > submitting a new one? It seems like overriding an existing reassignment > could be done with a single upda

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

Thanks for the KIP, Colin. I have a couple questions: 1. What's the reasoning for requiring cancellation of a reassignment before submitting a new one? It seems like overriding an existing reassignment could be done with a single update to /brokers/topics/[topic]/partitions/[partitionId]/state and

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

link: https://cwiki.apache.org/confluence/display/KAFKA/KIP-455%3A+Create+an+Administrative+API+for+Replica+Reassignment C. On Mon, Apr 15, 2019, at 18:07, Colin McCabe wrote: > Hi all, > > We've been having discussions on a few different KIPs (KIP-236, > KIP-435, etc.) about what the Admin Cl

[DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

Hi all, We've been having discussions on a few different KIPs (KIP-236, KIP-435, etc.) about what the Admin Client replica reassignment API should look like. The current API is really hard to extend and maintain, which is a big source of problems. I think it makes sense to have a KIP that est