[
https://issues.apache.org/jira/browse/FLINK-40469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18108034#comment-18108034
]
Aleksandr Savonin commented on FLINK-40469:
-------------------------------------------
PR: https://github.com/apache/flink-connector-kafka/pull/295
> DynamicKafkaSource can lose splits when checkpointing during recovery
> reassignment
> ----------------------------------------------------------------------------------
>
> Key: FLINK-40469
> URL: https://issues.apache.org/jira/browse/FLINK-40469
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
> Reporter: Aleksandr Savonin
> Priority: Major
> Labels: pull-request-available
>
> After recovery, readers report their restored splits to
> DynamicKafkaSourceEnumerator and temporarily hold no splits while metadata
> discovery is pending.
> These reports are kept only in memory. If a checkpoint completes during this
> window and the job later restores from it, the reader state is empty while
> the sub-enumerator still marks the splits as assigned. The splits are
> therefore never reassigned, causing silent data loss.
> Persist pending reader-reported splits in DynamicKafkaSourceEnumState and
> restore them through the existing reassignment flow, preserving their
> checkpointed offsets.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
