Hi,
(PS - I did not use users list as this concerns some internals)
I was trying to deploy the following config on a K8s cluster -
Namespace - X1 Connect group id - G1 Bootstrap servers - B1, B2
Namespace - X2 Connect group id - G1 Bootstrap servers - B3, B4
With this configuration, I am seeing multiple rebalances (logs below). So
the question is how is the Membership protocol working?
I have tried to follow some KIPs and they mostly say that the Bootstrap
server is the coordinator of the group. If yes, then logically this
configuration should work just fine as both configs have different
bootstrap servers. But as far as I have tried to understand the code (and
tried to run Kafka Connect), it seems like the members get added in the
group one by one and the head of the list becomes the group co-ordinator. (
if I change Connect Group id in 2nd config, things work)
Also, I wanted some pointers on what is happening at the server-side vs
client-side during the protocol.
Kindly help.
Logs -
[2020-03-16 10:27:13,386] INFO [Worker clientId=connect-1,
groupId=streaming-connect] Current config state offset 12 does not
match group assignment 9164. Forcing rebalance.
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:942)
[2020-03-16 10:27:13,386] INFO [Worker clientId=connect-1,
groupId=streaming-connect] Rebalance started
(org.apache.kafka.connect.runtime.distributed.WorkerCoordinator:213)
[2020-03-16 10:27:13,386] INFO [Worker clientId=connect-1,
groupId=streaming-connect] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:505)
[2020-03-16 10:27:13,388] INFO [Worker clientId=connect-1,
groupId=streaming-connect] Successfully joined group with generation
18535 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:469)
[2020-03-16 10:27:13,389] INFO [Worker clientId=connect-1,
groupId=streaming-connect] Joined group at generation 18535 and got
assignment: Assignment{error=0,
leader='connect-1-3580fdcc-257e-40e7-8243-f7839021599f',
leaderUrl='http://10.13.191.32:8083/', offset=9164, connectorIds=[],
taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0}
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:1397)
[2020-03-16 10:27:13,389] WARN [Worker clientId=connect-1,
groupId=streaming-connect] Catching up to assignment's config offset.
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:909)
[2020-03-16 10:27:13,389] INFO [Worker clientId=connect-1,
groupId=streaming-connect] Current config state offset 12 is behind
group assignment 9164, reading to end of config log
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:970)
Thanks,
Mandeep Gandhi
