[
https://issues.apache.org/jira/browse/KAFKA-9468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17022251#comment-17022251
]
Randall Hauch commented on KAFKA-9468:
--------------------------------------
Thanks, [~EeveeB]. I think it makes sense that the Connect worker should fail
upon startup if the # of partitions on the config topic is not 1, as this can
lead to serious problems.
[~ewencp], do you think this requires a KIP? Technically it is changing
behavior, but distributed Connect is not really functional if the # of config
topic partitions is not 1 and the fact we're not already checking this is
probably a bug that can be fixed without a KIP. WDYT?
> config.storage.topic partition count issue is hard to debug
> -----------------------------------------------------------
>
> Key: KAFKA-9468
> URL: https://issues.apache.org/jira/browse/KAFKA-9468
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Affects Versions: 1.0.2, 1.1.1, 2.0.1, 2.1.1, 2.2.2, 2.4.0, 2.3.1
> Reporter: Evelyn Bayes
> Priority: Minor
>
> When you run connect distributed with 2 or more workers and
> config.storage.topic has more then 1 partition, you can end up with one of
> the workers rebalancing endlessly:
> [2020-01-13 12:53:23,535] INFO [Worker clientId=connect-1,
> groupId=connect-cluster] Current config state offset 37 is behind group
> assignment 63, reading to end of config log
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder)
> [2020-01-13 12:53:23,584] INFO [Worker clientId=connect-1,
> groupId=connect-cluster] Finished reading to end of log and updated config
> snapshot, new config log offset: 37
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder)
> [2020-01-13 12:53:23,584] INFO [Worker clientId=connect-1,
> groupId=connect-cluster] Current config state offset 37 does not match group
> assignment 63. Forcing rebalance.
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder)
>
> In case any person viewing this doesn't know you are only ever meant to
> create this topic with one partition.
>
> *Suggested Solution*
> Make the connect worker check the partition count when it starts and if
> partition count is > 1 Kafka Connect stops and logs the reason why.
> I think this is reasonable as it would stop users just starting out from
> building it incorrectly and would be easy to fix early. For those upgrading
> this would easily be caught in a PRE-PROD environment. And even if they
> upgraded directly in PROD you would only be impacted if upgraded all connect
> workers at the same time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
