Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2020-03-26 Thread Colin McCabe
te: > > > > Thanks Colin, George. Can we restart the voting for this KIP. > > > > > > > > Thanks, > > > > Koushik > > > > > > > > -Original Message- > > > > From: Colin McCabe > > > > Sent: Wednesd

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-10-25 Thread Stanislav Kozlovski
ote: > > > Thanks Colin, George. Can we restart the voting for this KIP. > > > > > > Thanks, > > > Koushik > > > > > > -----Original Message----- > > > From: Colin McCabe > > > Sent: Wednesday, August 7, 2019 5:17 PM > > >

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-08-28 Thread Colin McCabe
is KIP. > > > > Thanks, > > Koushik > > > > -Original Message- > > From: Colin McCabe > > Sent: Wednesday, August 7, 2019 5:17 PM > > To: dev@kafka.apache.org > > Subject: Re: [VOTE] KIP-455: Create an Administrative API for Replica

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-08-08 Thread Colin McCabe
> -Original Message- > From: Colin McCabe > Sent: Wednesday, August 7, 2019 5:17 PM > To: dev@kafka.apache.org > Subject: Re: [VOTE] KIP-455: Create an Administrative API for Replica > Reassignment > > On Wed, Aug 7, 2019, at 15:41, George Li wrote: > > T

RE: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-08-08 Thread Koushik Chitta
Thanks Colin, George. Can we restart the voting for this KIP. Thanks, Koushik -Original Message- From: Colin McCabe Sent: Wednesday, August 7, 2019 5:17 PM To: dev@kafka.apache.org Subject: Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment On Wed, Aug 7

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-08-07 Thread Colin McCabe
t; > > OAR = {1, 2, 3} and RAR = {4,5,6} > > > > > > AR leader/isr > > > {1,2,3,4,5,6} 1/{1,2,3,4,6} => LeaderISRRequest > > > was lost/skipped for 5 and the reassignment operation will be waiting > > > indefinitely

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-08-07 Thread George Li
insync. > > > > > > > > Thanks, > > Koushik > > > > -Original Message- > > From: Jun Rao > > Sent: Friday, August 2, 2019 10:04 AM > > To: dev > > Subject: Re: [VOTE] KIP-455: Create an Administrative API for Replica >

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-08-06 Thread Colin McCabe
der/isr > {1,2,3,4,5,6}1/{1,2,3,4,6} => LeaderISRRequest > was lost/skipped for 5 and the reassignment operation will be waiting > indefinitely for the 5 to be insync. > > > > Thanks, > Koushik > > -Original Message----- > From: Jun Rao &

RE: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-08-06 Thread Koushik Chitta
nks, Koushik -Original Message- From: Jun Rao Sent: Friday, August 2, 2019 10:04 AM To: dev Subject: Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment Hi, Colin, First, since we are changing the format of LeaderAndIsrRequest, which is an inter broker requ

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-08-02 Thread Jun Rao
Hi, Colin, First, since we are changing the format of LeaderAndIsrRequest, which is an inter broker request, it seems that we will need IBP during rolling upgrade. Could we add that to the compatibility section? Regarding UnsupportedVersionException, even without ZK node version bump, we

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-08-01 Thread Colin McCabe
On Thu, Aug 1, 2019, at 12:00, Jun Rao wrote: > Hi, Colin, > > 10. Sounds good. > > 13. Our current convention is to bump up the version of ZK value if there > is any format change. For example, we have bumped up the version of the > value in /brokers/ids/nnn multiple times and all of those

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-08-01 Thread Jun Rao
Hi, Colin, 10. Sounds good. 13. Our current convention is to bump up the version of ZK value if there is any format change. For example, we have bumped up the version of the value in /brokers/ids/nnn multiple times and all of those changes are compatible (just adding new fields). This has the

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-07-31 Thread Colin McCabe
Hi Jun, Thanks for taking another look at this. On Wed, Jul 31, 2019, at 09:22, Jun Rao wrote: > Hi, Stan, > > Thanks for the explanation. > > 10. If those new fields in LeaderAndIsr are only needed for future work, > perhaps they should be added when we do the future work instead of now? I

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-07-31 Thread Jun Rao
Hi, Stan, Thanks for the explanation. 10. If those new fields in LeaderAndIsr are only needed for future work, perhaps they should be added when we do the future work instead of now? Jun On Wed, Jul 31, 2019 at 2:30 AM Stanislav Kozlovski wrote: > Hey Jun, > > I think I can answer some of

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-07-31 Thread Stanislav Kozlovski
Hey Jun, I think I can answer some of your questions on behalf of Colin -- he can confirm if I'm correct > 10. The KIP adds two new fields (AddingReplicas and RemovingReplicas) to LeaderAndIsr request. Could you explain how these 2 fields will be used? Sorry for not explaining this in the KIP -

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-07-30 Thread Jun Rao
Hi, Colin, Thanks for the KIP. Sorry for the late reply. LGTM overall. A few detailed comments below. 10. The KIP adds two new fields (AddingReplicas and RemovingReplicas) to LeaderAndIsr request. Could you explain how these 2 fields will be used? Should we include those two fields in

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-07-22 Thread Colin McCabe
Hi all, With three non-binding +1 votes from Viktor Somogyi-Vass, Robert Barrett, and George Li, and 3 binding +1 votes from Gwen Shapira, Jason Gustafson, and myself, the vote passes. Thanks, everyone! best, Colin On Fri, Jul 19, 2019, at 08:55, Robert Barrett wrote: > +1 (non-binding).

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-07-19 Thread Robert Barrett
+1 (non-binding). Thanks for the KIP! On Thu, Jul 18, 2019 at 5:59 PM George Li wrote: > +1 (non-binding) > > > > Thanks for addressing the comments. > George > > On Thursday, July 18, 2019, 05:03:58 PM PDT, Gwen Shapira < > g...@confluent.io> wrote: > > Renewing my +1, thank you Colin

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-07-18 Thread George Li
+1 (non-binding) Thanks for addressing the comments.  George On Thursday, July 18, 2019, 05:03:58 PM PDT, Gwen Shapira wrote: Renewing my +1, thank you Colin and Stan for working through all the questions, edge cases, requests and alternatives. We ended up with a great protocol.

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-07-18 Thread Gwen Shapira
Renewing my +1, thank you Colin and Stan for working through all the questions, edge cases, requests and alternatives. We ended up with a great protocol. On Thu, Jul 18, 2019 at 4:54 PM Jason Gustafson wrote: > > +1 Thanks for the KIP. Really looking forward to this! > > -Jason > > On Wed, Jul

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-07-18 Thread Jason Gustafson
+1 Thanks for the KIP. Really looking forward to this! -Jason On Wed, Jul 17, 2019 at 1:41 PM Colin McCabe wrote: > Thanks, Stanislav. Let's restart the vote to reflect the fact that we've > made significant changes. The new vote will go for 3 days as usual. > > I'll start with my +1

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-07-17 Thread Colin McCabe
Thanks, Stanislav. Let's restart the vote to reflect the fact that we've made significant changes. The new vote will go for 3 days as usual. I'll start with my +1 (binding). best, Colin On Wed, Jul 17, 2019, at 08:56, Stanislav Kozlovski wrote: > Hey everybody, > > We have further iterated

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-07-17 Thread Stanislav Kozlovski
Hey everybody, We have further iterated on the KIP in the accompanying discussion thread and I'd like to propose we resume the vote. Some notable changes: - we will store reassignment information in the `/brokers/topics/[topic]` - we will internally use two collections to represent a

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-05-22 Thread Colin McCabe
Hi George, Thanks for taking a look. I am working on getting a PR done as a proof-of-concept. I'll post it soon. Then we'll finish up the vote. best, Colin On Tue, May 21, 2019, at 17:33, George Li wrote: > Hi Colin,   > >  Great! Looking forward to these features.    +1 (non-binding) >

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-05-21 Thread George Li
Hi Colin,    Great! Looking forward to these features.    +1 (non-binding) What is the estimated timeline to have this implemented?  If any help is needed in the implementation of cancelling reassignments,  I can help if there is spare cycle.  Thanks, George On Thursday, May 16,

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-05-16 Thread Colin McCabe
Hi George, Yes, KIP-455 allows the reassignment of individual partitions to be cancelled. I think it's very important for these operations to be at the partition level. best, Colin On Tue, May 14, 2019, at 16:34, George Li wrote: > Hi Colin, > > Thanks for the updated KIP.  It has very good

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-05-14 Thread George Li
Hi Colin, Thanks for the updated KIP.  It has very good improvements of Kafka reassignment operations.  One question, looks like the KIP includes the Cancellation of individual pending reassignments as well when the AlterPartitionReasisgnmentRequest has empty replicas for the

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-05-10 Thread Colin McCabe
On Fri, May 10, 2019, at 17:34, Colin McCabe wrote: > On Fri, May 10, 2019, at 16:43, Jason Gustafson wrote: > > Hi Colin, > > > > I think storing reassignment state at the partition level is the right move > > and I also agree that replicas should understand that there is a > > reassignment in

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-05-10 Thread Colin McCabe
On Fri, May 10, 2019, at 16:43, Jason Gustafson wrote: > Hi Colin, > > I think storing reassignment state at the partition level is the right move > and I also agree that replicas should understand that there is a > reassignment in progress. This makes KIP-352 a trivial follow-up for > example.

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-05-10 Thread Jason Gustafson
Hi Colin, I think storing reassignment state at the partition level is the right move and I also agree that replicas should understand that there is a reassignment in progress. This makes KIP-352 a trivial follow-up for example. The only doubt I have is whether the leader and isr znode is the

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-05-10 Thread Gwen Shapira
+1 (binding) Looks great, and will be awesome to have this new capability. On Wed, May 8, 2019 at 10:23 PM Colin McCabe wrote: > Hi all, > > I'd like to start the vote for KIP-455: Create an Administrative API for > Replica Reassignment. I think this KIP is important since it will unlock >

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-05-09 Thread Robert Barrett
+1 (non-binding) Thanks for the KIP, Colin! On Thu, May 9, 2019 at 8:27 AM Colin McCabe wrote: > Hi Viktor, > > There is a jira -- KAFKA-8345. The PR is not quite ready yet, but > hopefully soon :) > > best, > Colin > > On Thu, May 9, 2019, at 01:13, Viktor Somogyi-Vass wrote: > > +1

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-05-09 Thread Colin McCabe
Hi Viktor, There is a jira -- KAFKA-8345. The PR is not quite ready yet, but hopefully soon :) best, Colin On Thu, May 9, 2019, at 01:13, Viktor Somogyi-Vass wrote: > +1 (non-binding) > > Thanks Colin, this is great stuff. Does a jira (or maybe even a PR :) ) for > this exist yet? > >

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-05-09 Thread Viktor Somogyi-Vass
+1 (non-binding) Thanks Colin, this is great stuff. Does a jira (or maybe even a PR :) ) for this exist yet? Viktor On Thu, May 9, 2019 at 7:23 AM Colin McCabe wrote: > Hi all, > > I'd like to start the vote for KIP-455: Create an Administrative API for > Replica Reassignment. I think this