[
https://issues.apache.org/jira/browse/KAFKA-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17816806#comment-17816806
]
Greg Harris commented on KAFKA-9070:
------------------------------------
Adding partitions to a topic is generally an unsafe activity, so I'm not
surprised that Connect has a problem with it.
That said, I think the best workaround for this problem is to use 3.6.0's
[KIP-875|https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect]
feature to export and re-import the offsets to forcibly re-partition the topic.
1. Stop all of the connectors
2. Export the connector offsets to a file with KIP-875.
3. Stop the Connect cluster
4. Delete the offsets topic, and recreate it with the increased number of
partitions. Or, create the new topic and reconfigure the workers to use it
instead.
5. Start the Connect cluster
6. Write the saved offsets back to the cluster via KIP-875
7. Resume all of the connectors.
> Kafka Connect - Get the wrong offset value comes from Kafka Connect after
> increase the number of offset storage topic partition to 3
> ------------------------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-9070
> URL: https://issues.apache.org/jira/browse/KAFKA-9070
> Project: Kafka
> Issue Type: Bug
> Components: connect
> Affects Versions: 2.3.0
> Environment: debezium/connect: 0.10
> mysql: 5.6
> kafka: 2.3.0
> Reporter: kaikai.hou
> Priority: Major
> Labels: connect
>
> I'm using the *Debezium* project with distributed mode.
> *Problem:*
> I found a problem : Get the wrong offset value comes from Kafka Connect after
> increase the number of offset storage topic partition to 3.
> 1. Cluster mode, tow node (container);
> 2. Offset storage topic only have 1 partition;
> 3. Create 3 mysql connector ;
> 4. Change data, all connector have offset recored in partition 0.
> 4. *Increase offset storage topic partition number to 3*.
> 5. Change data, and some connector store their offset records to partition 1
> or partition 2.
> 6.* Restart all connect service, then all connector will read offset records
> from partition 0*.
> 7. Then, the connector that stored offset records to partition 1 or partition
> 2 will get *repeat data*. (the offset records in partition 0 are* too old*
> for this connectors)
> *Debug*
> The Debezium developer checked their code found that the partition handling
> is delegated to Kafka Connect. [their
> reply|https://issues.jboss.org/browse/DBZ-1551?focusedCommentId=13800286&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13800286]
> Then, after they reproduced the problem, they found that [the problem is that
> the incorrect offset is returned by Kafka
> Connect|https://issues.jboss.org/browse/DBZ-1551?focusedCommentId=13801400&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13801400]
> .
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
