Tomoyuki NAKAMURA created FLINK-40991:
-----------------------------------------
Summary: Health info stored by an operator before 1.12 fails to
deserialize after upgrading, breaking reconciliation
Key: FLINK-40991
URL: https://issues.apache.org/jira/browse/FLINK-40991
Project: Flink
Issue Type: Bug
Components: Kubernetes Operator
Affects Versions: kubernetes-operator-1.16.1, kubernetes-operator-1.16.0,
kubernetes-operator-1.15.0, kubernetes-operator-1.14.0,
kubernetes-operator-1.12.1, kubernetes-operator-1.13.0,
kubernetes-operator-1.12.0
Reporter: Tomoyuki NAKAMURA
FLINK-37769 (1.12.0) replaced the boolean {{ClusterHealthInfo.healthy}} with
{{healthResult}} ({{ClusterHealthResult}}). {{ClusterHealthInfo.deserialize()}}
uses an ObjectMapper that fails on unknown properties, so the health info an
operator before 1.12 stored in {{status.clusterInfo}} can no longer be read
once the operator is upgraded to 1.12 or later:
{noformat}
Exception while observing cluster health: Could not deserialize
ClusterHealthInfo, this indicates a bug...
... UnrecognizedPropertyException ... (through reference chain:
org.apache.flink.kubernetes.operator.health.ClusterHealthInfo["healthy"])
{noformat}
*Conditions:* {{kubernetes.operator.cluster.health-check.enabled=true}} on the
old operator (it writes the health info only then), and an upgrade from <= 1.11
to >= 1.12. We hit it upgrading 1.11.0 -> 1.16.1 with ~200 FlinkDeployments:
every existing FlinkDeployment was affected.
*Impact:*
* {{ClusterHealthObserver}} catches the exception and logs a WARN on every
observation.
* {{ApplicationReconciler.reconcileOtherChanges()}} calls
{{ClusterHealthEvaluator.getLastValidClusterHealthInfo()}} without catching it,
so every reconciliation that gets that far fails and is retried ("Attempt
count: N"). Spec changes, FAILED-job restarts and periodic snapshots still work
(they come first), but missing-deployment recovery, unhealthy-job restarts and
the terminal JobManager cleanup never run.
* It does not heal: {{ClusterHealthEvaluator.evaluate()}} reads the stored info
before writing the new one, and {{status.clusterInfo}} is updated with
{{putAll}}, so neither restarting the operator nor redeploying the job removes
the stale entry. Only recreating the FlinkDeployment does.
The upgrade docs and the 1.12.0 release notes do not mention it.
*Proposed fix:* map the legacy field onto {{healthResult}} when deserializing,
e.g. a write-only {{@JsonProperty("healthy")}} setter in {{ClusterHealthInfo}}
({{true}} -> {{ClusterHealthResult.healthy()}}, {{false}} ->
{{ClusterHealthResult.error(...)}}), so the stored info is read and replaced by
the next observation. Serialization is unchanged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
