[
https://issues.apache.org/jira/browse/KAFKA-7854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma resolved KAFKA-7854.
--------------------------------
Resolution: Won't Fix
I'm marking this as "Won't fix" since KIP-455 introduced a Kafka protocol API
that provides the desired functionality. Setting reassignment via the znode is
deprecated and will be removed in a future release.
> Behavior change in controller picking up partition reassignment tasks since
> 1.1.0
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-7854
> URL: https://issues.apache.org/jira/browse/KAFKA-7854
> Project: Kafka
> Issue Type: Improvement
> Components: controller
> Reporter: Zhanxiang (Patrick) Huang
> Priority: Major
>
> After [https://github.com/apache/kafka/pull/4143,] the controller does not
> subscribe to data change on /admin/reassign_partitions any more (in order to
> avoid unnecessarily loading the reassignment data again after controller
> updating the znode) as opposed to the previous kafka versions. However, there
> are systems built around kafka relying on the previous behavior to
> incrementally update the list of partition reassignment since kafka does not
> natively support that.
>
> For example, [cruise control|https://github.com/linkedin/cruise-control] can
> rely on the previous behavior (controller listening to data changes) to
> maintain the reassignment concurrency by dynamically updating the data in the
> reassignment znode instead of waiting for the current batch to finish and
> doing reassignment batch by batch, which can significantly reduce the
> rebalance time in production clusters. Although directly updating the znode
> can somehow be viewed as an anti-pattern in the long term, this is necessary
> since kafka does not natively support incrementally submit more reassignment
> tasks. However, after our kafka clusters migrate from 0.11 to 2.0, cruise
> control no longer works because the controller behavior has changed. This
> reveals the following problems:
> * These behavior changes may be viewed as internal changes so compatibility
> is not guaranteed but I think by convention people do view this as public
> interfaces and rely on the compatibility. In this case, I think we should
> clearly document the data contract for the partition reassignment task to
> avoid misusage and making controller changes that break the defined data
> contract. There may be other cases (e.g. topic deletion) whose data contracts
> need to be clearly defined and we should keep it in mind when making
> controller changes.
> * Kafka does not natively support incrementally submit more reassignment
> tasks. If we do want to support that nicely, we should consider change how we
> store the reassignment data to store the data in child nodes and let the
> controller listen on child node changes, similar to what we do for
> /admin/delete_topics.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
