[
https://issues.apache.org/jira/browse/FLINK-41088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-41088:
-----------------------------------
Fix Version/s: kafka-5.1.0
> DynamicKafkaSource loses or rejects splits whose formatted split ids collide
> ----------------------------------------------------------------------------
>
> Key: FLINK-41088
> URL: https://issues.apache.org/jira/browse/FLINK-41088
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
> Affects Versions: kafka-5.1.0
> Reporter: Martijn Visser
> Assignee: Martijn Visser
> Priority: Blocker
> Labels: pull-request-available
> Fix For: kafka-5.1.0
>
>
> DynamicKafkaSourceSplit ids are {{<cluster id>-<topic>-<partition>}}, so
> cluster {{a}} with topic {{b-c}} and cluster {{a-b}} with topic {{c}} both
> give {{a-b-c-0}}. The enumerator's recovery path still keys reports by that
> id: {{ReaderRecoveryGate}} merges a re-registering reader's reports by
> {{splitId()}} and keeps only one of the two, and
> {{DynamicKafkaSourceEnumerator#reassignReportedSplits}} fails the restore
> with "Split a-b-c-0 was reported by both reader 0 and reader 1" when both are
> active. In bounded mode the dropped split is never read and the readers still
> get no-more-splits.
> Both reproduce on main (e47cf425) and with FLINK-40710, which already keys
> ownership and the restarted-reader merge by cluster id plus topic partition.
> The fix is the same key in the gate and the initial reassignment.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
