[
https://issues.apache.org/jira/browse/KAFKA-9216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17056572#comment-17056572
]
ASF GitHub Bot commented on KAFKA-9216:
---------------------------------------
Evelyn-Bayes commented on pull request #8270: KAFKA-9216: Enforce connect
internal topic configuration at startup
URL: https://github.com/apache/kafka/pull/8270
Currently, if Kafka Connect will create its config backing topic with a fire
and forget approach.
This is fine unless someone has manually created that topic already with the
wrong partition count.
In such a case Kafka Connect "may" run for some time.
Especially if it's in standalone mode and once switched to distributed mode
it will almost certainly fail.
To counter this I've added a check when the KafkaConfigBackingStore is
starting.
This check will throw a ConfigException if there is more than one partition
in the backing store.
This exception is then caught upstream and logged by either:
- class: DistributedHerder, method: run
- class: ConnectStandalone, method: main
After a review I don't believe it impacts any other upstream code.
Finally, to supper this new functionality I've added a public method to
KafkaBasedLog which returns the partition count and a variable to store this.
And, I've created a unit test in KafkaConfigBackingStoreTest to verify the
behaviour.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Enforce connect internal topic configuration at startup
> -------------------------------------------------------
>
> Key: KAFKA-9216
> URL: https://issues.apache.org/jira/browse/KAFKA-9216
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Affects Versions: 0.11.0.0
> Reporter: Randall Hauch
> Assignee: Evelyn Bayes
> Priority: Major
>
> Users sometimes configure Connect's internal topic for configurations with
> more than one partition. One partition is expected, however, and using more
> than one leads to weird behavior that is sometimes not easy to spot.
> Here's one example of a log message:
> {noformat}
> "textPayload": "[2019-11-20 11:12:14,049] INFO [Worker clientId=connect-1,
> groupId=td-connect-server] Current config state offset 284 does not match
> group assignment 274. Forcing rebalance.
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder:942)\n"
> {noformat}
> Would it be possible to add a check in the KafkaConfigBackingStore and
> prevent the worker from starting if connect config partition count !=1 ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
