[
https://issues.apache.org/jira/browse/FLINK-40991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomoyuki NAKAMURA updated FLINK-40991:
--------------------------------------
Description:
FLINK-37769 (1.12.0) replaced the boolean {{ClusterHealthInfo.healthy}} with
{{healthResult}} ({{ClusterHealthResult}}). {{ClusterHealthInfo.deserialize()}}
uses an ObjectMapper that fails on unknown properties, so the health info an
operator before 1.12 stored in {{status.clusterInfo}} can no longer be read
once the operator is upgraded to 1.12 or later:
{noformat}
Exception while observing cluster health: Could not deserialize
ClusterHealthInfo, this indicates a bug...
... UnrecognizedPropertyException ... (through reference chain:
org.apache.flink.kubernetes.operator.health.ClusterHealthInfo["healthy"])
{noformat}
*Conditions:* {{kubernetes.operator.cluster.health-check.enabled=true}} on the
old operator (it writes the health info only then), and an upgrade from <= 1.11
to >= 1.12. Every existing FlinkDeployment with stored health info is affected.
*Impact:*
* {{ClusterHealthObserver}} catches the exception and logs a WARN on every
observation.
* {{ApplicationReconciler.reconcileOtherChanges()}} calls
{{ClusterHealthEvaluator.getLastValidClusterHealthInfo()}} without catching it,
so every reconciliation that gets that far fails and is retried ("Attempt
count: N"). Spec changes, FAILED-job restarts and periodic snapshots still work
(they come first), but missing-deployment recovery, unhealthy-job restarts and
the terminal JobManager cleanup never run.
* It does not heal: {{ClusterHealthEvaluator.evaluate()}} reads the stored info
before writing the new one, and {{status.clusterInfo}} is updated with
{{putAll}}, so neither restarting the operator nor redeploying the job removes
the stale entry. Only recreating the FlinkDeployment does.
The upgrade docs and the 1.12.0 release notes do not mention it.
*Proposed fix:* map the legacy field onto {{healthResult}} when deserializing,
e.g. a write-only {{@JsonProperty("healthy")}} setter in {{ClusterHealthInfo}}
({{true}} -> {{ClusterHealthResult.healthy()}}, {{false}} ->
{{ClusterHealthResult.error(...)}}), so the stored info is read and replaced by
the next observation. Serialization is unchanged.
was:
FLINK-37769 (1.12.0) replaced the boolean {{ClusterHealthInfo.healthy}} with
{{healthResult}} ({{ClusterHealthResult}}). {{ClusterHealthInfo.deserialize()}}
uses an ObjectMapper that fails on unknown properties, so the health info an
operator before 1.12 stored in {{status.clusterInfo}} can no longer be read
once the operator is upgraded to 1.12 or later:
{noformat}
Exception while observing cluster health: Could not deserialize
ClusterHealthInfo, this indicates a bug...
... UnrecognizedPropertyException ... (through reference chain:
org.apache.flink.kubernetes.operator.health.ClusterHealthInfo["healthy"])
{noformat}
*Conditions:* {{kubernetes.operator.cluster.health-check.enabled=true}} on the
old operator (it writes the health info only then), and an upgrade from <= 1.11
to >= 1.12. We hit it upgrading 1.11.0 -> 1.16.1 with ~200 FlinkDeployments:
every existing FlinkDeployment was affected.
*Impact:*
* {{ClusterHealthObserver}} catches the exception and logs a WARN on every
observation.
* {{ApplicationReconciler.reconcileOtherChanges()}} calls
{{ClusterHealthEvaluator.getLastValidClusterHealthInfo()}} without catching it,
so every reconciliation that gets that far fails and is retried ("Attempt
count: N"). Spec changes, FAILED-job restarts and periodic snapshots still work
(they come first), but missing-deployment recovery, unhealthy-job restarts and
the terminal JobManager cleanup never run.
* It does not heal: {{ClusterHealthEvaluator.evaluate()}} reads the stored info
before writing the new one, and {{status.clusterInfo}} is updated with
{{putAll}}, so neither restarting the operator nor redeploying the job removes
the stale entry. Only recreating the FlinkDeployment does.
The upgrade docs and the 1.12.0 release notes do not mention it.
*Proposed fix:* map the legacy field onto {{healthResult}} when deserializing,
e.g. a write-only {{@JsonProperty("healthy")}} setter in {{ClusterHealthInfo}}
({{true}} -> {{ClusterHealthResult.healthy()}}, {{false}} ->
{{ClusterHealthResult.error(...)}}), so the stored info is read and replaced by
the next observation. Serialization is unchanged.
> Health info stored by an operator before 1.12 fails to deserialize after
> upgrading, breaking reconciliation
> -----------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40991
> URL: https://issues.apache.org/jira/browse/FLINK-40991
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Affects Versions: kubernetes-operator-1.12.0, kubernetes-operator-1.13.0,
> kubernetes-operator-1.12.1, kubernetes-operator-1.14.0,
> kubernetes-operator-1.15.0, kubernetes-operator-1.16.0,
> kubernetes-operator-1.16.1
> Reporter: Tomoyuki NAKAMURA
> Priority: Major
>
> FLINK-37769 (1.12.0) replaced the boolean {{ClusterHealthInfo.healthy}} with
> {{healthResult}} ({{ClusterHealthResult}}).
> {{ClusterHealthInfo.deserialize()}} uses an ObjectMapper that fails on
> unknown properties, so the health info an operator before 1.12 stored in
> {{status.clusterInfo}} can no longer be read once the operator is upgraded to
> 1.12 or later:
> {noformat}
> Exception while observing cluster health: Could not deserialize
> ClusterHealthInfo, this indicates a bug...
> ... UnrecognizedPropertyException ... (through reference chain:
> org.apache.flink.kubernetes.operator.health.ClusterHealthInfo["healthy"])
> {noformat}
> *Conditions:* {{kubernetes.operator.cluster.health-check.enabled=true}} on
> the old operator (it writes the health info only then), and an upgrade from
> <= 1.11 to >= 1.12. Every existing FlinkDeployment with stored health info is
> affected.
> *Impact:*
> * {{ClusterHealthObserver}} catches the exception and logs a WARN on every
> observation.
> * {{ApplicationReconciler.reconcileOtherChanges()}} calls
> {{ClusterHealthEvaluator.getLastValidClusterHealthInfo()}} without catching
> it, so every reconciliation that gets that far fails and is retried ("Attempt
> count: N"). Spec changes, FAILED-job restarts and periodic snapshots still
> work (they come first), but missing-deployment recovery, unhealthy-job
> restarts and the terminal JobManager cleanup never run.
> * It does not heal: {{ClusterHealthEvaluator.evaluate()}} reads the stored
> info before writing the new one, and {{status.clusterInfo}} is updated with
> {{putAll}}, so neither restarting the operator nor redeploying the job
> removes the stale entry. Only recreating the FlinkDeployment does.
> The upgrade docs and the 1.12.0 release notes do not mention it.
> *Proposed fix:* map the legacy field onto {{healthResult}} when
> deserializing, e.g. a write-only {{@JsonProperty("healthy")}} setter in
> {{ClusterHealthInfo}} ({{true}} -> {{ClusterHealthResult.healthy()}},
> {{false}} -> {{ClusterHealthResult.error(...)}}), so the stored info is read
> and replaced by the next observation. Serialization is unchanged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
