[
https://issues.apache.org/jira/browse/KAFKA-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Randall Hauch resolved KAFKA-12270.
-----------------------------------
Fix Version/s: 2.6.2
2.7.1
2.8.0
Reviewer: Konstantine Karantasis
Resolution: Fixed
Merged to `trunk` for the upcoming 2.8.0, and cherrypicked to the 2.7 branch
for the next 2.7.1 and to the 2.6 branch for the next 2.6.2.
> Kafka Connect may fail a task when racing to create topic
> ---------------------------------------------------------
>
> Key: KAFKA-12270
> URL: https://issues.apache.org/jira/browse/KAFKA-12270
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Affects Versions: 2.6.0, 2.7.0, 2.8.0
> Reporter: Randall Hauch
> Assignee: Randall Hauch
> Priority: Critical
> Fix For: 2.8.0, 2.7.1, 2.6.2
>
>
> When a source connector configured with many tasks and to use the new topic
> creation feature is run, it is possible that multiple tasks will attempt to
> write to the same topic, will see that the topic does not exist, and then
> race to create the topic. The topic is only created once, but some tasks
> might fail with:
> {code:java}
> org.apache.kafka.connect.errors.ConnectException: Task failed to create new
> topic (name=TOPICX, numPartitions=8, replicationFactor=3,
> replicasAssignments=null, configs={cleanup.policy=delete}). Ensure that the
> task is authorized to create topics or that the topic exists and restart the
> task
> at
> org.apache.kafka.connect.runtime.WorkerSourceTask.maybeCreateTopic(WorkerSourceTask.java:436)
> at
> org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:364)
> at
> org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:264)
> at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:185)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:235)
> ... {code}
> The reason appears to be that the WorkerSourceTask throws an exception if the
> topic creation failed, and does not account for the fact that the topic may
> have been created between the time the WorkerSourceTask lists existing topics
> and tries to create the topic.
>
> See in particular:
> [https://github.com/apache/kafka/blob/5c562efb2d76407011ea88c1ca1b2355079935bc/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java#L415-L423]
>
> This is only an issue when using topic creation settings in the source
> connector configuration, and when running multiple tasks that write to the
> same topic.
> The workaround is to create the topics manually before starting the
> connector, or to simply restart the failed tasks using the REST API.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
