[
https://issues.apache.org/jira/browse/KAFKA-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107727#comment-16107727
]
Dan Collins commented on KAFKA-5505:
------------------------------------
Hi, I wanted to add on to this given we are working around the same behavior,
but for a different use case. We have an orchestration service that takes care
of wiring sinks out of Kafka using Connect and some of the orchestration
scenarios we have require us to setup or modify tens of configurations in
Connect at a time. Our current approach to work around this is to retry with
backoff and pre-package connector configurations, but it seems like there
should be a better way. It's also a bit painful to get a 409 Conflict on GET
request for current configurations when tasks are rebalancing especially given
we may have multiple orchestration events running simultaneously.
The requested change here to only rebalance for new/deleted tasks would be a
nice improvement as it'd decrease the initialization time/improve availability.
Other thoughts we've had on this would be to expose a new endpoint with bulk
api support or a flag on configurations to indicate that new tasks should not
immediately trigger a rebalance or may be scheduled in the future.
Thanks! -Dan
> Connect: Do not restart connector and existing tasks on task-set change
> -----------------------------------------------------------------------
>
> Key: KAFKA-5505
> URL: https://issues.apache.org/jira/browse/KAFKA-5505
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Affects Versions: 0.10.2.1
> Reporter: Per Steffensen
>
> I am writing a connector with a frequently changing task-set. It is really
> not working very well, because the connector and all existing tasks are
> restarted when the set of tasks changes. E.g. if the connector is running
> with 10 tasks, and an additional task is needed, the connector itself and all
> 10 existing tasks are restarted, just to make the 11th task run also. My
> tasks have a fairly heavy initialization, making it extra annoying. I would
> like to see a change, introducing a "mode", where only new/deleted tasks are
> started/stopped when notifying the system that the set of tasks changed
> (calling context.requestTaskReconfiguration() - or something similar).
> Discussed this issue a little on d...@kafka.apache.org in the thread "Kafka
> Connect: To much restarting with a SourceConnector with dynamic set of tasks"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
