hemmatio commented on PR #27991: URL: https://github.com/apache/flink/pull/27991#issuecomment-4296895788
> @gaborgsomogyi: Can you elaborate please? The most common trigger (from what we have seen internally) is user code changes across deployments. This happens with the Flink Kubernetes Operator's `last-state` upgrade mode as follows: 1. The user modifies their job (adds/removes/renames an operator, which is the standard use case of `allowNonRestoredState`) 2. The operator performs a `last-state` upgrade: Tears down the cluster without taking a fresh savepoint, relying on checkpoint metadata for the restore. 3. If the prior JM's shutdown was not graceful (ex: crash during upgrade, OOM, pod eviction), the per-job HA ConfigMap survives. The new JM rebuilds the graph from the modified user code, finds the old checkpoint, and hits the state that doesn't map to any operator in the new graph. 4. `restoreInitialCheckpointIfPresent` rejects it via the hardcoded `false`, regardless of the value of `execution.state-recovery.ignore-unclaimed-state`. --- The asymmetry is the main issue, as there are two different outcomes depending on upgrade mode for the same job, and the same `allowNonRestoredState: true`: - `upgradeMode: savepoint`: works, orphaned state skipped, job restores - `upgradeMode: last-state`: fails with `IllegalStateException: There is no operator for the state ...` This problem was also reported in [FLINK-30638](https://issues.apache.org/jira/browse/FLINK-30638), where Gyula Fora correctly diagnosed it as a runtime issue in the comments. This PR finally addresses the bug. --- The SQL plan drift case would also hit the same code path. Essentially, anything that produces a mismatch between the checkpoint's operator ID set and the new JobGraph's operator IDs will trigger it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
