[
https://issues.apache.org/jira/browse/KAFKA-9228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993990#comment-16993990
]
Chris Egerton commented on KAFKA-9228:
--------------------------------------
This issue is much harder to reproduce than initially suspected.
When the {{Connector::start}} method is invoked, the framework passes in the
entire user-supplied connector configuration, including converter and client
configs. Most connectors will retain that configuration and then, from their
{{Connector::taskConfigs}} method, return either verbatim or slightly-modified
copies of it. This has the consequence that, through quirk of connector
development style, it's likely that changing the converter and/or client
configs for a connector will result in a change for its task configs as well
(even if those converter and/or configs are completely ignored by both the
connector and its tasks). Any change in the task configs for a connector will
result in its tasks being restarted, which then causes any changes to its
converters and/or clients to be picked up.
However, there are two connectors that do _not_ follow this pattern: the
[FileStreamSourceConnector|https://github.com/apache/kafka/blob/717ce42a6d68d3ac8d9478451a31423b5d43234e/connect/file/src/main/java/org/apache/kafka/connect/file/FileStreamSourceConnector.java#L78-L86]
and the
[FileStreamSinkConnector|https://github.com/apache/kafka/blob/717ce42a6d68d3ac8d9478451a31423b5d43234e/connect/file/src/main/java/org/apache/kafka/connect/file/FileStreamSinkConnector.java#L62-L69].
Any changes to the configs of either of these connectors that only affect
their converters and/or Kafka clients will not propagate to their tasks until
their tasks are restarted (either manually, due to rebalance, or via some other
cause).
> Reconfigured converters and clients may not be propagated to connector tasks
> ----------------------------------------------------------------------------
>
> Key: KAFKA-9228
> URL: https://issues.apache.org/jira/browse/KAFKA-9228
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Affects Versions: 2.3.0, 2.4.0, 2.3.1, 2.3.2
> Reporter: Chris Egerton
> Assignee: Chris Egerton
> Priority: Major
>
> If an existing connector is reconfigured but the only changes are to its
> converters and/or Kafka clients (enabled as of
> [KIP-458|https://cwiki.apache.org/confluence/display/KAFKA/KIP-458%3A+Connector+Client+Config+Override+Policy]),
> the changes will not propagate to its tasks unless the connector also
> generates task configs that differ from the existing task configs. Even after
> this point, if the connector tasks are reconfigured, they will still not pick
> up on the new converter and/or Kafka client configs.
> This is because the {{DistributedHerder}} only writes new task configurations
> to the connect config topic [if the connector-provided task configs differ
> from the task configs already in the config
> topic|https://github.com/apache/kafka/blob/e499c960e4f9cfc462f1a05a110d79ffa1c5b322/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1285-L1332],
> and neither of those contain converter or Kafka client configs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
