Hi Yeikel, Sharing my two cents. Would let others chime in to add to this.
Based on my understanding, if connect workers (which are all part of the same cluster) can communicate with the kafka brokers (which happens to be the Group Coordinator and facilitates Connect Leader Election via Group Membership Protocol), then only 1 connect worker will be elected as leader amongst all others in the cluster. Outside of that, I believe a bunch of REST calls to connect workers are forwarded to the connect leader (if the REST request lands on a connect worker which isn't a leader). In case of a non-retriable network partition between the non-leader worker and leader worker, those REST requests will fail. I'm referring to REST requests like CREATE / UPDATE / DELETE. Hope this helps a little. Thanks, -Nikhil On Sun, 24 Sept 2023 at 06:36, Yeikel Santana <em...@yeikel.com> wrote: > Hello everyone,I'm currently designing a new Kafka Connect cluster, and > I'm trying to understand how connectivity functions among workers.In my > setup, I have a single Kafka Connect cluster connected to the same Kafka > topics and Kafka cluster. However, the workers are deployed in > geographically separated data centers, each of which is fully isolated at > the networkI suspect that this setup might not work with Kafka Connect > because my current understanding is that ALL workers need to communicate > with the leader for task coordination and heartbeats.In terms of leader > election, can this result in multiple leaders and other potential > issues?Any input and suggestions would be appreciated
