Dennis-Mircea Ciupitu created FLINK-41589:
---------------------------------------------

             Summary: Harden lifecycle state computations
                 Key: FLINK-41589
                 URL: https://issues.apache.org/jira/browse/FLINK-41589
             Project: Flink
          Issue Type: Improvement
          Components: Kubernetes Operator
            Reporter: Dennis-Mircea Ciupitu


h1. Problem

While testing a different feature, I ended up seeing this combination:
{noformat}
NAME               AGE     JOBMANAGER STATUS   JOB STATUS    LIFECYCLE STATE
basic-example      8m24s   MISSING             RECONCILING   DEPLOYED
{noformat}
In reality, nothing was deployed, the job had been reset to its initial 
{{RECONCILING}} state, and the lifecycle column still said {{{}DEPLOYED{}}}. 
Had the job been observed {{RUNNING}} once before the JobManager vanished, the 
same rows would have said {{{}STABLE{}}}.
h1. Root cause

{{CommonStatus#getLifecycleState()}} is derived from the reconciliation state, 
the two spec copies, {{status.error}} and the Flink job state. The JobManager 
deployment status enters only through one override: {{MISSING}} or {{ERROR}} is 
promoted to {{FAILED}} when {{status.error}} contains one of three substrings 
("Manual restore required", "HA metadata is not available", "configmaps have 
been deleted"). None of the errors the operator records for a lost or broken 
JobManager match: the observer writes "Missing JobManager deployment", 
{{DeploymentFailedException}} writes the Kubernetes reason (crash loop, OOM, 
image pull), and the branch that declines recovery for a stateful job without 
HA only logs. So these resources fall through to {{{}DEPLOYED{}}}, or 
{{STABLE}} when the spec was marked stable before.

Three properties of the computation make the gap permanent:
 * Stability is one way. {{markReconciledSpecAsStable()}} is never undone by 
runtime trouble, only by a new spec.
 * Reconciliation state is checked first. {{UPGRADING}} and {{ROLLING_BACK}} 
shadow a failed job and the override, and {{SUSPENDED}} precedes the {{FAILED}} 
job check.
 * {{FlinkSessionJob}} shares the method and has no cluster input at all.

h1. Affected combinations
||JobManager status||Job status||Reconciliation||Reported||Expected||
|{{{}MISSING{}}}, recovery pending or 
declined|{{RECONCILING}}|{{{}DEPLOYED{}}}, spec not stable|{{DEPLOYED}}|not a 
running or healthy state|
|{{{}MISSING{}}}, recovery declined (stateful, no 
HA)|{{RECONCILING}}|{{{}DEPLOYED{}}}, spec stable|{{STABLE}}|{{{}FAILED{}}}, 
nothing will fix it|
|{{ERROR}} (crash loop, OOM, image pull)|{{RECONCILING}}|{{{}DEPLOYED{}}}, spec 
stable|{{STABLE}}|not a healthy state|
|any|{{FAILED}}|{{UPGRADING}} (blocked upgrade)|{{UPGRADING}}|the failure is 
hidden|
|any|{{CANCELED}} or {{FINISHED}}|{{{}DEPLOYED{}}}, spec 
stable|{{STABLE}}|nothing is running|
|session cluster gone (session job)|last observed|{{{}DEPLOYED{}}}, spec 
stable|{{STABLE}}|no cluster input exists|



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to