Jing Hung created FLINK-41373:
---------------------------------
Summary: Blue/Green: after an aborted transition, the next deploy
restores from a stale savepoint
Key: FLINK-41373
URL: https://issues.apache.org/jira/browse/FLINK-41373
Project: Flink
Issue Type: Bug
Components: Kubernetes Operator
Affects Versions: 1.16.0, 1.14.0, 1.15.0
Reporter: Jing Hung
h3. Summary:
FlinkBlueGreenDeployment restores from a stale savepoint after an aborted
transition because the suspended child keeps
status.jobStatus.upgradeSavepointPath
h3. Problem
When a Blue/Green transition is aborted, {{abortDeployment()}} suspends the
failed child FlinkDeployment. It does not delete it. The suspended child keeps
{{status.jobStatus.upgradeSavepointPath}} from its last deployment.
On the next transition, the B/G controller takes a fresh savepoint and writes
it to the child's {{{}spec.job.initialSavepointPath{}}}. Because the child
already exists, the reconciler handles it as a restore of a suspended job, and
{{AbstractJobReconciler.restoreJob()}} reads
{{{}status.jobStatus.upgradeSavepointPath{}}}. The fresh
{{initialSavepointPath}} is ignored, and the job restores from the old
savepoint.
The transition reports success, so the problem is silent. In our production
incident the job restored from a savepoint 9 days old, and the state between
the two savepoints was lost.
h3. Steps to reproduce
# Create a FlinkBlueGreenDeployment with {{{}upgradeMode: savepoint{}}}. Wait
for ACTIVE_BLUE.
# Change the spec with an image that does not exist. The controller takes
savepoint S1 and creates GREEN from S1.
# Wait for the abort. GREEN is suspended with
{{{}status.jobStatus.upgradeSavepointPath = S1{}}}.
# Change the spec again with a valid image. The controller takes savepoint S2
and sets GREEN {{{}initialSavepointPath = S2{}}}.
*Expected:* GREEN restores from S2.
*Actual:* GREEN restores from S1.
FLINK-39270 and FLINK-40470 do not fix this. The FLINK-39270 gate passes when
HA metadata is empty, and FLINK-40470 applies only when the spec and status
savepoint paths are equal.
h3. Proposed fix
In {{{}abortDeployment(){}}}, delete the failed child instead of suspending it,
and fall back to suspend if the delete fails. The next transition then creates
the child again with no status, so the reconciler uses
{{{}spec.job.initialSavepointPath{}}}. This keeps the fix in the B/G controller
and does not change FlinkDeployment restore semantics.
Behavior change: after an abort, the failed child no longer remains as
SUSPENDED. The abort reason stays in the FlinkBlueGreenDeployment status.error
and events.
Alternatives considered:
* Patch the child's {{status.jobStatus.upgradeSavepointPath}} to match the new
{{initialSavepointPath}} in {{prepareFlinkDeployment()}}. This has a race with
the FlinkDeployment reconciler, which can read the status before the patch
arrives.
* Clear the child status on abort. Our tests showed this causes a stateless
start (state loss).
* Make {{restoreJob()}} prefer {{spec.job.initialSavepointPath}} when it
differs from status. This changes core FlinkDeployment semantics for all users,
so would prefer to fix it in the B/G controller.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
