This is currently expected. Internally the Connect cluster uses the same rebalancing process as consumer groups which means it has similar limitations -- all tasks must stop just as you would need to stop consuming from all partitions and commit offsets during a consumer group rebalance.
There's an issue filed about this: https://issues.apache.org/ jira/browse/KAFKA-3351 and it's something we're aware eventually becomes a scalability limitations. There are some ideas about how to avoid this, but nothing concrete on the roadmap yet. -Ewen On Fri, Jan 13, 2017 at 10:32 AM, Mathieu Fenniak < mathieu.fenn...@replicon.com> wrote: > Hey kafka-users, > > Is it normal for a Kafka Connect source connector that > calls requestTaskReconfiguration to cause all the connectors on the > kafka-connect distributed system to be stopped and started? > > One of my three connectors (2x source, 1x sink) runs a background thread > that will occasionally invoke `context.requestTaskReconfiguration()` when > it detects new work that it wants to distribute to its tasks. When this > occurs, I observe all three of the connectors stop and start. One of the > other connectors doesn't have the smoothest stop/start cycle, so I'm hoping > that this might be avoidable? > > I'm running on Kafka Connect 0.10.1.0. > > Mathieu >
