hyeon kim created FLINK-40470:
---------------------------------
Summary: Savepoint redeploy loops forever when the deployment
attempt fails
Key: FLINK-40470
URL: https://issues.apache.org/jira/browse/FLINK-40470
Project: Flink
Issue Type: Bug
Components: Kubernetes Operator
Affects Versions: kubernetes-operator-1.15.0
Reporter: hyeon kim
When {{job.savepointRedeployNonce}} is changed,
{{AbstractJobReconciler#redeployWithSavepoint}} cancels the running job
(deleting the whole cluster) and redeploys from {{initialSavepointPath}}.
Unlike every other deployment path (first deployment, suspend/resume, session
deploy), the target spec is only recorded into the resource status *after* the
deploy call succeeds:
{code:java}
cancelJob(ctx, SuspendMode.STATELESS);
...
deploy(...); // <- if this throws...
ReconciliationUtils.updateStatusForDeployedSpec(...); // <- ...the new nonce
is never recorded
{code}
If {{deploy}} throws for any reason (transient API server error, or an error
raised after the cluster resources were actually created — e.g.
{{KubernetesClusterDescriptor#deployApplicationCluster}} fails resolving the
rest endpoint of a freshly created LoadBalancer service on RBAC-restricted
clusters where listing nodes is forbidden), the new nonce is never recorded in
{{lastReconciledSpec}}. Every subsequent reconciliation classifies the diff as
{{SAVEPOINT_REDEPLOY}} again and repeats the full delete + redeploy cycle
indefinitely.
Observed in production on a managed Kubernetes platform where {{nodes}} list is
forbidden for namespaced tenants: with {{kubernetes.rest-service.exposed.type:
LoadBalancer}}, a savepoint redeploy looped delete/recreate for 23 minutes
until manual intervention.
All other job deployment paths already call
{{ReconciliationUtils.updateStatusBeforeDeploymentAttempt}} before deploying
(see {{AbstractJobReconciler#reconcileSpecChange}} for the suspended->running
transition and
{{AbstractFlinkResourceReconciler#updateStatusBeforeFirstDeployment}}), which
makes a failed attempt recover through the regular upgrading/observe flow
instead of retriggering the redeploy.
*Proposed fix:* record the target spec with
{{updateStatusBeforeDeploymentAttempt}} +
{{statusRecorder.patchAndCacheStatus}} before the deploy call in
{{redeployWithSavepoint}}, mirroring the suspend/resume path. A failed deploy
then recovers on the next reconciliation by restoring from the recorded
{{upgradeSavepointPath}} (which already points at {{initialSavepointPath}}),
without repeating the delete + redeploy cycle.
*Open question:* with this fix, the retry recovers through the regular
suspended->running restore path. That path only restores from
{{upgradeSavepointPath}} when the effective upgrade mode is not STATELESS, so
for a spec with {{upgradeMode: stateless}} the retry falls back to an
empty-state restore instead of the requested savepoint (still strictly better
than the current infinite delete/redeploy loop). Distinguishing "interrupted
savepoint redeploy" from "user deliberately switched to a stateless resume" in
{{reconcileSpecChange}} needs a semantic decision, so I left that corner as a
follow-up — happy to address it in this ticket if there is a preferred approach.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
