Thank you, Nikhil. I did notice that challenge you're describing with the REST updates when I had more than one worker within the same datacenter.
Luckily, solving that was relatively simple as all my workers can communicate within the same data center, and all I need to do is to ensure that the update is initiated from the same datacenter as the leader. From what I tested so far, this seems to work fine. My biggest concern was regarding other operations such as heartbeats or general coordination. If that happens through Kafka, then I should be fine.Thank you for taking the time ---- On Mon, 25 Sep 2023 09:45:43 -0400 nikhilsrivastava4...@gmail.com wrote ----Hi Yeikel, Sharing my two cents. Would let others chime in to add to this. Based on my understanding, if connect workers (which are all part of the same cluster) can communicate with the kafka brokers (which happens to be the Group Coordinator and facilitates Connect Leader Election via Group Membership Protocol), then only 1 connect worker will be elected as leader amongst all others in the cluster. Outside of that, I believe a bunch of REST calls to connect workers are forwarded to the connect leader (if the REST request lands on a connect worker which isn't a leader). In case of a non-retriable network partition between the non-leader worker and leader worker, those REST requests will fail. I'm referring to REST requests like CREATE / UPDATE / DELETE. Hope this helps a little. Thanks, -Nikhil On Sun, 24 Sept 2023 at 06:36, Yeikel Santana <em...@yeikel.com> wrote: > Hello everyone,I'm currently designing a new Kafka Connect cluster, and > I'm trying to understand how connectivity functions among workers.In my > setup, I have a single Kafka Connect cluster connected to the same Kafka > topics and Kafka cluster. However, the workers are deployed in > geographically separated data centers, each of which is fully isolated at > the networkI suspect that this setup might not work with Kafka Connect > because my current understanding is that ALL workers need to communicate > with the leader for task coordination and heartbeats.In terms of leader > election, can this result in multiple leaders and other potential > issues?Any input and suggestions would be appreciated
