Dennis-Mircea Ciupitu created FLINK-41589:
---------------------------------------------
Summary: Harden lifecycle state computations
Key: FLINK-41589
URL: https://issues.apache.org/jira/browse/FLINK-41589
Project: Flink
Issue Type: Improvement
Components: Kubernetes Operator
Reporter: Dennis-Mircea Ciupitu
h1. Problem
While testing a different feature, I ended up seeing this combination:
{noformat}
NAME AGE JOBMANAGER STATUS JOB STATUS LIFECYCLE STATE
basic-example 8m24s MISSING RECONCILING DEPLOYED
{noformat}
In reality, nothing was deployed, the job had been reset to its initial
{{RECONCILING}} state, and the lifecycle column still said {{{}DEPLOYED{}}}.
Had the job been observed {{RUNNING}} once before the JobManager vanished, the
same rows would have said {{{}STABLE{}}}.
h1. Root cause
{{CommonStatus#getLifecycleState()}} is derived from the reconciliation state,
the two spec copies, {{status.error}} and the Flink job state. The JobManager
deployment status enters only through one override: {{MISSING}} or {{ERROR}} is
promoted to {{FAILED}} when {{status.error}} contains one of three substrings
("Manual restore required", "HA metadata is not available", "configmaps have
been deleted"). None of the errors the operator records for a lost or broken
JobManager match: the observer writes "Missing JobManager deployment",
{{DeploymentFailedException}} writes the Kubernetes reason (crash loop, OOM,
image pull), and the branch that declines recovery for a stateful job without
HA only logs. So these resources fall through to {{{}DEPLOYED{}}}, or
{{STABLE}} when the spec was marked stable before.
Three properties of the computation make the gap permanent:
* Stability is one way. {{markReconciledSpecAsStable()}} is never undone by
runtime trouble, only by a new spec.
* Reconciliation state is checked first. {{UPGRADING}} and {{ROLLING_BACK}}
shadow a failed job and the override, and {{SUSPENDED}} precedes the {{FAILED}}
job check.
* {{FlinkSessionJob}} shares the method and has no cluster input at all.
h1. Affected combinations
||JobManager status||Job status||Reconciliation||Reported||Expected||
|{{{}MISSING{}}}, recovery pending or
declined|{{RECONCILING}}|{{{}DEPLOYED{}}}, spec not stable|{{DEPLOYED}}|not a
running or healthy state|
|{{{}MISSING{}}}, recovery declined (stateful, no
HA)|{{RECONCILING}}|{{{}DEPLOYED{}}}, spec stable|{{STABLE}}|{{{}FAILED{}}},
nothing will fix it|
|{{ERROR}} (crash loop, OOM, image pull)|{{RECONCILING}}|{{{}DEPLOYED{}}}, spec
stable|{{STABLE}}|not a healthy state|
|any|{{FAILED}}|{{UPGRADING}} (blocked upgrade)|{{UPGRADING}}|the failure is
hidden|
|any|{{CANCELED}} or {{FINISHED}}|{{{}DEPLOYED{}}}, spec
stable|{{STABLE}}|nothing is running|
|session cluster gone (session job)|last observed|{{{}DEPLOYED{}}}, spec
stable|{{STABLE}}|no cluster input exists|
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)