[jira] [Updated] (FLINK-33466) Bounded Kafka source never finishes after restore from savepoint
[ https://issues.apache.org/jira/browse/FLINK-33466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33466: --- Labels: pull-request-available (was: ) > Bounded Kafka source never finishes after restore from savepoint > > > Key: FLINK-33466 > URL: https://issues.apache.org/jira/browse/FLINK-33466 > Project: Flink > Issue Type: Bug > Components: Connectors / Common, Connectors / Kafka >Affects Versions: 1.17.1 >Reporter: Jonas Weile >Priority: Major > Labels: pull-request-available > > When setting up a bounded Kafka source, if the job is restored from a > savepoint before the source has finished, then the Kafka source will never > transition to a finished state. > This seems to be because the noMoreSplitsAssignment variable in the > SourceReaderBase class is not snapshotted. Therefore, after restoring from a > checkpoint/savepoint, the noMoreSplitsAssignment variable will default to > false, and the first condition in the private helper method > finishedOrAvailableLater() in the SourceReaderBase class will always evaluate > to true. > Since this originates in the base class, the problem should hold for all > source types, not just kafka. > > Would it make sense to snapshot the noMoreSplitsAssigntments variable? > I would love to take this on as a first task if appropriate. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33466) Bounded Kafka source never finishes after restore from savepoint
[ https://issues.apache.org/jira/browse/FLINK-33466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonas Weile updated FLINK-33466: Component/s: Connectors / Kafka > Bounded Kafka source never finishes after restore from savepoint > > > Key: FLINK-33466 > URL: https://issues.apache.org/jira/browse/FLINK-33466 > Project: Flink > Issue Type: Bug > Components: Connectors / Common, Connectors / Kafka >Affects Versions: 1.17.1 >Reporter: Jonas Weile >Priority: Major > > When setting up a bounded Kafka source, if the job is restored from a > savepoint before the source has finished, then the Kafka source will never > transition to a finished state. > This seems to be because the noMoreSplitsAssignment variable in the > SourceReaderBase class is not snapshotted. Therefore, after restoring from a > checkpoint/savepoint, the noMoreSplitsAssignment variable will default to > false, and the first condition in the private helper method > finishedOrAvailableLater() in the SourceReaderBase class will always evaluate > to true. > Since this originates in the base class, the problem should hold for all > source types, not just kafka. > > Would it make sense to snapshot the noMoreSplitsAssigntments variable? > I would love to take this on as a first task if appropriate. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33466) Bounded Kafka source never finishes after restore from savepoint
[ https://issues.apache.org/jira/browse/FLINK-33466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonas Weile updated FLINK-33466: Component/s: Connectors / Common (was: Connectors / Kafka) (was: Runtime / Checkpointing) > Bounded Kafka source never finishes after restore from savepoint > > > Key: FLINK-33466 > URL: https://issues.apache.org/jira/browse/FLINK-33466 > Project: Flink > Issue Type: Bug > Components: Connectors / Common >Affects Versions: 1.17.1 >Reporter: Jonas Weile >Priority: Major > > When setting up a bounded Kafka source, if the job is restored from a > savepoint before the source has finished, then the Kafka source will never > transition to a finished state. > This seems to be because the noMoreSplitsAssignment variable in the > SourceReaderBase class is not snapshotted. Therefore, after restoring from a > checkpoint/savepoint, the noMoreSplitsAssignment variable will default to > false, and the first condition in the private helper method > finishedOrAvailableLater() in the SourceReaderBase class will always evaluate > to true. > Since this originates in the base class, the problem should hold for all > source types, not just kafka. > > Would it make sense to snapshot the noMoreSplitsAssigntments variable? > I would love to take this on as a first task if appropriate. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33466) Bounded Kafka source never finishes after restore from savepoint
[ https://issues.apache.org/jira/browse/FLINK-33466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonas Weile updated FLINK-33466: Description: When setting up a bounded Kafka source, if the job is restored from a savepoint before the source has finished, then the Kafka source will never transition to a finished state. This seems to be because the noMoreSplitsAssignment variable in the SourceReaderBase class is not snapshotted. Therefore, after restoring from a checkpoint/savepoint, the noMoreSplitsAssignment variable will default to false, and the first condition in the private helper method finishedOrAvailableLater() in the SourceReaderBase class will always evaluate to true. Since this originates in the base class, the problem should hold for all source types, not just kafka. Would it make sense to snapshot the noMoreSplitsAssigntments variable? I would love to take this on as a first task if appropriate. was: When setting up a bounded Kafka source, if the job is restored from a savepoint before the source has finished, then the Kafka source will never transition to a finished state. This seems to be because the noMoreSplitsAssignment variable in the SourceReaderBase > Bounded Kafka source never finishes after restore from savepoint > > > Key: FLINK-33466 > URL: https://issues.apache.org/jira/browse/FLINK-33466 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka, Runtime / Checkpointing >Affects Versions: 1.17.1 >Reporter: Jonas Weile >Priority: Major > > When setting up a bounded Kafka source, if the job is restored from a > savepoint before the source has finished, then the Kafka source will never > transition to a finished state. > This seems to be because the noMoreSplitsAssignment variable in the > SourceReaderBase class is not snapshotted. Therefore, after restoring from a > checkpoint/savepoint, the noMoreSplitsAssignment variable will default to > false, and the first condition in the private helper method > finishedOrAvailableLater() in the SourceReaderBase class will always evaluate > to true. > Since this originates in the base class, the problem should hold for all > source types, not just kafka. > > Would it make sense to snapshot the noMoreSplitsAssigntments variable? > I would love to take this on as a first task if appropriate. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33466) Bounded Kafka source never finishes after restore from savepoint
[ https://issues.apache.org/jira/browse/FLINK-33466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonas Weile updated FLINK-33466: Description: When setting up a bounded Kafka source, if the job is restored from a savepoint before the source has finished, then the Kafka source will never transition to a finished state. This seems to be because the noMoreSplitsAssignment variable in the SourceReaderBase was:When setting up a bounded Kafka source with the OffsetsInitializer.latest(), if the job is restored from a savepoint before the source has finished, then the Kafka source will never transition to a finished state. > Bounded Kafka source never finishes after restore from savepoint > > > Key: FLINK-33466 > URL: https://issues.apache.org/jira/browse/FLINK-33466 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka, Runtime / Checkpointing >Affects Versions: 1.17.1 >Reporter: Jonas Weile >Priority: Major > > When setting up a bounded Kafka source, if the job is restored from a > savepoint before the source has finished, then the Kafka source will never > transition to a finished state. > > This seems to be because the noMoreSplitsAssignment variable in the > SourceReaderBase -- This message was sent by Atlassian Jira (v8.20.10#820010)
