Re: Kafka Connect distributed mode rebalance

Ewen, thanks for the reply. I will admit that in the current state my connector use is not optimal. Today it is a combination of JDBC source connectors and a custom sink connector. The original rollout was before single message transforms (SMT) weren't available (2.0.1 was our first deployment,

Re: Kafka Connect distributed mode rebalance

Btw, if you can share, I would be curious what connectors you're using and why you need so many. I'd be interested if a modification to the connector could also simplify things for you. -Ewen On Wed, Jul 26, 2017 at 12:33 AM, Ewen Cheslack-Postava wrote: > Stephen, > > Cool,

Re: Kafka Connect distributed mode rebalance

Stephen, Cool, that is a *lot* of connectors! Regarding rebalances, the reason this happens is that Kafka Connect is trying to keep the total work of the cluster balanced across the workers. If you add/remove connectors or the # of workers change, then we need to go through another round

Kafka Connect distributed mode rebalance

I'm seeing some behavior with the DistributedHerder that I am trying to understand. I'm working on setting up a cluster of kafka connect nodes and have a relatively large number of connectors to submit to it (392 connectors right now that will soon become over 1100). As for the deployment of it I