[
https://issues.apache.org/jira/browse/FLINK-40991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18126929#comment-18126929
]
Tomoyuki NAKAMURA commented on FLINK-40991:
-------------------------------------------
Thank you for taking a look, and sorry if I didn't explain it clearly. The
failure actually happens in the currently supported versions (1.12 and later,
including 1.16), when they read the health info left by 1.11 or earlier. So any
user who had the cluster health check enabled before 1.12 runs into it whenever
they upgrade, also via 1.12, and the affected deployments keep failing part of
every reconciliation until they are recreated.
The fix is a small, write-only compatibility setter, so I thought it might save
those users from recreating every FlinkDeployment. I may be missing some
context, though, so if there is a concern with carrying it, I would be happy to
hear it. Alternatively, I could turn the PR into a note in the upgrade docs
that explains how to clear the stale entry.
> Health info stored by an operator before 1.12 fails to deserialize after
> upgrading, breaking reconciliation
> -----------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40991
> URL: https://issues.apache.org/jira/browse/FLINK-40991
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Affects Versions: kubernetes-operator-1.12.0, kubernetes-operator-1.13.0,
> kubernetes-operator-1.12.1, kubernetes-operator-1.14.0,
> kubernetes-operator-1.15.0, kubernetes-operator-1.16.0,
> kubernetes-operator-1.16.1
> Reporter: Tomoyuki NAKAMURA
> Priority: Major
> Labels: pull-request-available
>
> FLINK-37769 (1.12.0) replaced the boolean {{ClusterHealthInfo.healthy}} with
> {{healthResult}} ({{ClusterHealthResult}}).
> {{ClusterHealthInfo.deserialize()}} uses an ObjectMapper that fails on
> unknown properties, so the health info an operator before 1.12 stored in
> {{status.clusterInfo}} can no longer be read once the operator is upgraded to
> 1.12 or later:
> {noformat}
> Exception while observing cluster health: Could not deserialize
> ClusterHealthInfo, this indicates a bug...
> ... UnrecognizedPropertyException ... (through reference chain:
> org.apache.flink.kubernetes.operator.health.ClusterHealthInfo["healthy"])
> {noformat}
> *Conditions:* {{kubernetes.operator.cluster.health-check.enabled=true}} on
> the old operator (it writes the health info only then), and an upgrade from
> <= 1.11 to >= 1.12. Every existing FlinkDeployment with stored health info is
> affected.
> *Impact:*
> * {{ClusterHealthObserver}} catches the exception and logs a WARN on every
> observation.
> * {{ApplicationReconciler.reconcileOtherChanges()}} calls
> {{ClusterHealthEvaluator.getLastValidClusterHealthInfo()}} without catching
> it, so every reconciliation that gets that far fails and is retried ("Attempt
> count: N"). Spec changes, FAILED-job restarts and periodic snapshots still
> work (they come first), but missing-deployment recovery, unhealthy-job
> restarts and the terminal JobManager cleanup never run.
> * It does not heal: {{ClusterHealthEvaluator.evaluate()}} reads the stored
> info before writing the new one, and {{status.clusterInfo}} is updated with
> {{putAll}}, so neither restarting the operator nor redeploying the job
> removes the stale entry. Only recreating the FlinkDeployment does.
> The upgrade docs and the 1.12.0 release notes do not mention it.
> *Proposed fix:* map the legacy field onto {{healthResult}} when
> deserializing, e.g. a write-only {{@JsonProperty("healthy")}} setter in
> {{ClusterHealthInfo}} ({{true}} -> {{ClusterHealthResult.healthy()}},
> {{false}} -> {{ClusterHealthResult.error(...)}}), so the stored info is read
> and replaced by the next observation. Serialization is unchanged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
