C0urante opened a new pull request, #16001: URL: https://github.com/apache/kafka/pull/16001
[Jira](https://issues.apache.org/jira/browse/KAFKA-9228) Currently, if a connector generates the same set of task configs as the set that already exists in the config topic, then this new set of task configs is not written to the config topic, and the connector's tasks are not (immediately*) restarted. This is intentional behavior to prevent infinite rebalance loops when using eager rebalancing, and most of the time it comes with no drawbacks. However, if the runtime-controlled properties for a connector (such as its key/value/header converters, or Kafka client overrides) are modified, then this behavior can cause the updates to not be applied (immediately*) to the connector's tasks. In order to address this, we forcibly rewrite task configs to the config topic when we detect a change in the connector config, even if they are identical to the existing task configs. Changes to the connector config are tracked by taking a hash of the connector config and including it in the config topic when writing task configs. If no hash has been written to the config topic yet, then we do not compare hashes. This is done in order to prevent upgrades to newer workers from causing all tasks on the cluster to be immediately restarted. As a final note, this bug should not be very prevalent in the Kafka Connect ecosystem, since most connectors will unintentionally propagate changes in runtime-controlled properties to their tasks. This is because the classic idiom for connectors is to track the properties provided in [Connector::start](https://kafka.apache.org/37/javadoc/org/apache/kafka/connect/connector/Connector.html#start(java.util.Map)) and use either an identical clone or a slightly-modified copy of those properties in the return value of [Connector::taskConfigs](https://kafka.apache.org/37/javadoc/org/apache/kafka/connect/connector/Connector.html#taskConfigs(int)). \* - Tasks can be restarted later on as a result of workers joining/leaving the cluster, users manually triggering restarts via the REST API, or other causes. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
