[
https://issues.apache.org/jira/browse/KAFKA-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16160845#comment-16160845
]
Per Steffensen commented on KAFKA-5505:
---------------------------------------
Just realized that SourceConnector has a initialize method that is given the
"existing tasks configuration". That will help me a little.
Besides that I was thinking that it may be easier to add more light-weight
callbacks to the system via ConnectorContext. Today you only have
requestTaskReconfiguration, which I guess tells the system that something
changed - anything could have changed: several tasks added, several tasks
removed, config change etc. What if we added two additional callbacks on
ConnectorContext - addTask and removeTask - used to add and remove tasks
respectively. That way the system does not have to do a complicated check of
what changed and act appropriately (do the minimum required actions) as it
probably has to with requestTaskReconfiguration today. We are leaving more of
the detailed management of tasks to the concrete connector itself (if it wants
to).
Will that be easier?
Guess, since this is new callbacks, that it will be easier to guarantee
backwards compatibility on requestTaskReconfiguration?
> Connect: Do not restart connector and existing tasks on task-set change
> -----------------------------------------------------------------------
>
> Key: KAFKA-5505
> URL: https://issues.apache.org/jira/browse/KAFKA-5505
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Affects Versions: 0.10.2.1
> Reporter: Per Steffensen
>
> I am writing a connector with a frequently changing task-set. It is really
> not working very well, because the connector and all existing tasks are
> restarted when the set of tasks changes. E.g. if the connector is running
> with 10 tasks, and an additional task is needed, the connector itself and all
> 10 existing tasks are restarted, just to make the 11th task run also. My
> tasks have a fairly heavy initialization, making it extra annoying. I would
> like to see a change, introducing a "mode", where only new/deleted tasks are
> started/stopped when notifying the system that the set of tasks changed
> (calling context.requestTaskReconfiguration() - or something similar).
> Discussed this issue a little on d...@kafka.apache.org in the thread "Kafka
> Connect: To much restarting with a SourceConnector with dynamic set of tasks"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
