[ https://issues.apache.org/jira/browse/KAFKA-7854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ismael Juma resolved KAFKA-7854. -------------------------------- Resolution: Won't Fix I'm marking this as "Won't fix" since KIP-455 introduced a Kafka protocol API that provides the desired functionality. Setting reassignment via the znode is deprecated and will be removed in a future release. > Behavior change in controller picking up partition reassignment tasks since > 1.1.0 > --------------------------------------------------------------------------------- > > Key: KAFKA-7854 > URL: https://issues.apache.org/jira/browse/KAFKA-7854 > Project: Kafka > Issue Type: Improvement > Components: controller > Reporter: Zhanxiang (Patrick) Huang > Priority: Major > > After [https://github.com/apache/kafka/pull/4143,] the controller does not > subscribe to data change on /admin/reassign_partitions any more (in order to > avoid unnecessarily loading the reassignment data again after controller > updating the znode) as opposed to the previous kafka versions. However, there > are systems built around kafka relying on the previous behavior to > incrementally update the list of partition reassignment since kafka does not > natively support that. > > For example, [cruise control|https://github.com/linkedin/cruise-control] can > rely on the previous behavior (controller listening to data changes) to > maintain the reassignment concurrency by dynamically updating the data in the > reassignment znode instead of waiting for the current batch to finish and > doing reassignment batch by batch, which can significantly reduce the > rebalance time in production clusters. Although directly updating the znode > can somehow be viewed as an anti-pattern in the long term, this is necessary > since kafka does not natively support incrementally submit more reassignment > tasks. However, after our kafka clusters migrate from 0.11 to 2.0, cruise > control no longer works because the controller behavior has changed. This > reveals the following problems: > * These behavior changes may be viewed as internal changes so compatibility > is not guaranteed but I think by convention people do view this as public > interfaces and rely on the compatibility. In this case, I think we should > clearly document the data contract for the partition reassignment task to > avoid misusage and making controller changes that break the defined data > contract. There may be other cases (e.g. topic deletion) whose data contracts > need to be clearly defined and we should keep it in mind when making > controller changes. > * Kafka does not natively support incrementally submit more reassignment > tasks. If we do want to support that nicely, we should consider change how we > store the reassignment data to store the data in child nodes and let the > controller listen on child node changes, similar to what we do for > /admin/delete_topics. -- This message was sent by Atlassian Jira (v8.3.4#803005)