[
https://issues.apache.org/jira/browse/FLINK-33466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonas Weile updated FLINK-33466:
--------------------------------
Description:
When setting up a bounded Kafka source, if the job is restored from a savepoint
before the source has finished, then the Kafka source will never transition to
a finished state.
This seems to be because the noMoreSplitsAssignment variable in the
SourceReaderBase class is not snapshotted. Therefore, after restoring from a
checkpoint/savepoint, the noMoreSplitsAssignment variable will default to
false, and the first condition in the private helper method
finishedOrAvailableLater() in the SourceReaderBase class will always evaluate
to true.
Since this originates in the base class, the problem should hold for all source
types, not just kafka.
Would it make sense to snapshot the noMoreSplitsAssigntments variable?
I would love to take this on as a first task if appropriate.
was:
When setting up a bounded Kafka source, if the job is restored from a savepoint
before the source has finished, then the Kafka source will never transition to
a finished state.
This seems to be because the noMoreSplitsAssignment variable in the
SourceReaderBase
> Bounded Kafka source never finishes after restore from savepoint
> ----------------------------------------------------------------
>
> Key: FLINK-33466
> URL: https://issues.apache.org/jira/browse/FLINK-33466
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Runtime / Checkpointing
> Affects Versions: 1.17.1
> Reporter: Jonas Weile
> Priority: Major
>
> When setting up a bounded Kafka source, if the job is restored from a
> savepoint before the source has finished, then the Kafka source will never
> transition to a finished state.
> This seems to be because the noMoreSplitsAssignment variable in the
> SourceReaderBase class is not snapshotted. Therefore, after restoring from a
> checkpoint/savepoint, the noMoreSplitsAssignment variable will default to
> false, and the first condition in the private helper method
> finishedOrAvailableLater() in the SourceReaderBase class will always evaluate
> to true.
> Since this originates in the base class, the problem should hold for all
> source types, not just kafka.
>
> Would it make sense to snapshot the noMoreSplitsAssigntments variable?
> I would love to take this on as a first task if appropriate.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
