Aleksandr Savonin created FLINK-40469:
-----------------------------------------
Summary: DynamicKafkaSource can lose splits when checkpointing
during recovery reassignment
Key: FLINK-40469
URL: https://issues.apache.org/jira/browse/FLINK-40469
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Reporter: Aleksandr Savonin
After recovery, readers report their restored splits to
DynamicKafkaSourceEnumerator and temporarily hold no splits while metadata
discovery is pending.
These reports are kept only in memory. If a checkpoint completes during this
window and the job later restores from it, the reader state is empty while the
sub-enumerator still marks the splits as assigned. The splits are therefore
never reassigned, causing silent data loss.
Persist pending reader-reported splits in DynamicKafkaSourceEnumState and
restore them through the existing reassignment flow, preserving their
checkpointed offsets.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
