[
https://issues.apache.org/jira/browse/FLINK-40991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18126930#comment-18126930
]
Gyula Fora commented on FLINK-40991:
------------------------------------
if the change is small and non intrusive then it makes sense .
> Health info stored by an operator before 1.12 fails to deserialize after
> upgrading, breaking reconciliation
> -----------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40991
> URL: https://issues.apache.org/jira/browse/FLINK-40991
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Affects Versions: kubernetes-operator-1.12.0, kubernetes-operator-1.13.0,
> kubernetes-operator-1.12.1, kubernetes-operator-1.14.0,
> kubernetes-operator-1.15.0, kubernetes-operator-1.16.0,
> kubernetes-operator-1.16.1
> Reporter: Tomoyuki NAKAMURA
> Priority: Major
> Labels: pull-request-available
>
> FLINK-37769 (1.12.0) replaced the boolean {{ClusterHealthInfo.healthy}} with
> {{healthResult}} ({{ClusterHealthResult}}).
> {{ClusterHealthInfo.deserialize()}} uses an ObjectMapper that fails on
> unknown properties, so the health info an operator before 1.12 stored in
> {{status.clusterInfo}} can no longer be read once the operator is upgraded to
> 1.12 or later:
> {noformat}
> Exception while observing cluster health: Could not deserialize
> ClusterHealthInfo, this indicates a bug...
> ... UnrecognizedPropertyException ... (through reference chain:
> org.apache.flink.kubernetes.operator.health.ClusterHealthInfo["healthy"])
> {noformat}
> *Conditions:* {{kubernetes.operator.cluster.health-check.enabled=true}} on
> the old operator (it writes the health info only then), and an upgrade from
> <= 1.11 to >= 1.12. Every existing FlinkDeployment with stored health info is
> affected.
> *Impact:*
> * {{ClusterHealthObserver}} catches the exception and logs a WARN on every
> observation.
> * {{ApplicationReconciler.reconcileOtherChanges()}} calls
> {{ClusterHealthEvaluator.getLastValidClusterHealthInfo()}} without catching
> it, so every reconciliation that gets that far fails and is retried ("Attempt
> count: N"). Spec changes, FAILED-job restarts and periodic snapshots still
> work (they come first), but missing-deployment recovery, unhealthy-job
> restarts and the terminal JobManager cleanup never run.
> * It does not heal: {{ClusterHealthEvaluator.evaluate()}} reads the stored
> info before writing the new one, and {{status.clusterInfo}} is updated with
> {{putAll}}, so neither restarting the operator nor redeploying the job
> removes the stale entry. Only recreating the FlinkDeployment does.
> The upgrade docs and the 1.12.0 release notes do not mention it.
> *Proposed fix:* map the legacy field onto {{healthResult}} when
> deserializing, e.g. a write-only {{@JsonProperty("healthy")}} setter in
> {{ClusterHealthInfo}} ({{true}} -> {{ClusterHealthResult.healthy()}},
> {{false}} -> {{ClusterHealthResult.error(...)}}), so the stored info is read
> and replaced by the next observation. Serialization is unchanged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
