Martijn Visser created FLINK-41088:
--------------------------------------
Summary: DynamicKafkaSource loses or rejects splits whose
formatted split ids collide
Key: FLINK-41088
URL: https://issues.apache.org/jira/browse/FLINK-41088
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Affects Versions: kafka-5.1.0
Reporter: Martijn Visser
Assignee: Martijn Visser
DynamicKafkaSourceSplit ids are {{<cluster id>-<topic>-<partition>}}, so
cluster {{a}} with topic {{b-c}} and cluster {{a-b}} with topic {{c}} both give
{{a-b-c-0}}. The enumerator's recovery path still keys reports by that id:
{{ReaderRecoveryGate}} merges a re-registering reader's reports by
{{splitId()}} and keeps only one of the two, and
{{DynamicKafkaSourceEnumerator#reassignReportedSplits}} fails the restore with
"Split a-b-c-0 was reported by both reader 0 and reader 1" when both are
active. In bounded mode the dropped split is never read and the readers still
get no-more-splits.
Both reproduce on main (e47cf425) and with FLINK-40710, which already keys
ownership and the restarted-reader merge by cluster id plus topic partition.
The fix is the same key in the gate and the initial reassignment.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
