Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

Due to the incorrect initial numbering of this KIP, discussion has moved to this thread: http://mail-archives.apache.org/mod_mbox/kafka-dev/201707.mbox/%3CCAMd5Ysy3bY7Fq2xA3sk6BWW6%3D9TjT4%2Bya7mufRf6Wgre-S-UPg%40mail.gmail.com%3E On 20 July 2017 at 16:47, Ismael Juma wrote:

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

Hi Tom, Yes, a poll-based approach sounds good since reassignment can take a long time. I even think that it should be manual. That is, the user should run a command to ask for the status of the rebalance instead of the tool doing it automatically. Ismael On Thu, Jul 20, 2017 at 4:12 AM, Tom

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

Hi Ismael, I've been working on the progress reporting assuming that it would be acceptable for the ReassignPartitionsCommand to poll the AdminClient API (and in turn the AdminClient API to poll the broker) in order to report progress in an interactive way. The alternative would, of course, be

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

Ah, thank you! I took the number from the "Next KIP Number: 178" on the KIP index and didn't check the tables. So this is now KIP-179. The old link will point you to the right place. On 19 July 2017 at 15:55, Ismael Juma wrote: > One more thing, it looks like there is already

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

One more thing, it looks like there is already a KIP-178: KIP-178: Size-based log directory selection strategy Ismael On Wed, Jul 19, 2017 at 7:05 AM, Tom Bentley wrote: > OK, I will work on adding support for this to the KIP, with the intention > of a two part

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

OK, I will work on adding support for this to the KIP, with the intention of a two part implementation. On 19 July 2017 at 14:59, Ismael Juma wrote: > Hi Tom, > > It's fine for the tool not to have this functionality from the start. > However, since we're adding new Kafka

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

Hi Tom, It's fine for the tool not to have this functionality from the start. However, since we're adding new Kafka protocol APIs, we need to consider some of these details to ensure we're building towards the end state, if that makes sense. Protocol APIs are used by multiple clients, so there is

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

Hi Ismael, Answers in-line: 1. Have you considered how progress would be reported? Partition > reassignment can take a long time and it would be good to have a mechanism > for progress reporting. > The ReassignPartitionsCommand doesn't currently have a mechanism to track progress. All you can

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

Hi Tom, Thanks for the KIP. Two quick comments: 1. Have you considered how progress would be reported? Partition reassignment can take a long time and it would be good to have a mechanism for progress reporting. 2. Removals can only happen in major releases. In your example, the removal could

[DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

Hi, I've written KIP-178 ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-178+-+Change+ReassignPartitionsCommand+to+use+AdminClient) for changing the ReassignPartitionsCommand (a.k.a. kafka-reassign-partitions.sh)to remove the dependency on ZooKeeper and instead provide the same