[
https://issues.apache.org/jira/browse/FLINK-39513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-39513:
-----------------------------------
Labels: pull-request-available (was: )
> CheckpointCoordinator.restoreInitialCheckpointIfPresent ignores
> allowNonRestoredState
> -------------------------------------------------------------------------------------
>
> Key: FLINK-39513
> URL: https://issues.apache.org/jira/browse/FLINK-39513
> Project: Flink
> Issue Type: Bug
> Components: API / Core
> Affects Versions: 2.2.0
> Reporter: Ryan van Huuksloot
> Priority: Minor
> Labels: pull-request-available
>
> `CheckpointCoordinator.restoreInitialCheckpointIfPresent` hardcodes
> `allowNonRestoredState=false` when calling
> `restoreLatestCheckpointedStateInternal`, so the HA-checkpoint-restore path
> rejects any checkpoint state that cannot be mapped to an operator in the
> current JobGraph regardless of the user's
> `execution.savepoint.ignore-unclaimed-state` (or
> `execution.state-recovery.ignore-unclaimed-state`) setting.
> The current code on master:
> [https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java#L1733|https://github.com/apache/flink/blob/b12302d87e53df5bd191918faad4dda80fbcba85/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java#L1733]
> The method was introduced in [FLINK-20222] under the documented assumption
> `// JobManager failover means JobGraphs match exactly.` That assumption held
> for pure JM failover, where the JobGraph is guaranteed identical across the
> restart. It no longer holds for the broader set of scenarios that invoke this
> method today — most notably JM startups into a cluster with stale HA
> checkpoint metadata from a modified previous JobGraph (e.g., Flink Kubernetes
> Operator `last-state` upgrades after a non-graceful shutdown, or
> crash-retries mid-upgrade).
> The savepoint-restore path (`CheckpointCoordinator.restoreSavepoint`) in the
> same class correctly reads `allowNonRestoredState` from
> `SavepointRestoreSettings` and honors it. The checkpoint-restore path should
> follow the same pattern.
> Observed failure in the JobManager:
> IllegalStateException: There is no operator for the state
> <operator-id-hash>
> at StateAssignmentOperation.checkStateMappingCompleteness
> at CheckpointCoordinator.restoreLatestCheckpointedStateInternal
> at CheckpointCoordinator.restoreInitialCheckpointIfPresent
> at DefaultExecutionGraphFactory.createAndRestoreExecutionGraph
> followed by a JM crashloop.
> *Proposed fix*: parameterize `allowNonRestoredState` on
> `restoreInitialCheckpointIfPresent` (the same shape as
> `restoreLatestCheckpointedStateToAll`, which already takes the flag as a
> parameter) and have the single production caller,
> `DefaultExecutionGraphFactory.createAndRestoreExecutionGraph`, read
> `SavepointConfigOptions.SAVEPOINT_IGNORE_UNCLAIMED_STATE` from the
> `Configuration` and pass it in.
> *Related Ticket*: [FLINK-30638] — same user-facing symptom filed against the
> Kubernetes Operator component. The runtime root cause was correctly
> identified in the comments on that ticket by but the ticket was left with a
> workaround (switching to savepoint upgrade mode) rather than a fix. This
> ticket tracks the underlying runtime bug.
> Bug report courtesy of: Omid Hemmati
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
