[
https://issues.apache.org/jira/browse/FLINK-41050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-41050:
-----------------------------------
Labels: pull-request-available (was: )
> Canary health checks revert canary metadata changes and restart a healthy
> operator
> -----------------------------------------------------------------------------------
>
> Key: FLINK-41050
> URL: https://issues.apache.org/jira/browse/FLINK-41050
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Reporter: Dennis-Mircea Ciupitu
> Priority: Major
> Labels: pull-request-available
>
> h1. Problem
> Canary resources let the operator health probe verify that spec changes are
> still picked up and reconciled. Every
> {{kubernetes.operator.health.canary.resource.timeout}} cycle (60s by
> default), the operator bumps the canary {{spec.restartNonce}} and checks that
> the previous bump was reconciled in time. If not, the probe reports the
> operator unhealthy and Kubernetes restarts it.
> The way the canary spec is updated has several problems. Users can lose their
> changes to the canary, the operator can modify resources that are no longer
> canaries, and the operator can be restarted although it works fine:
> # *Metadata changes on the canary are reverted* - Labels or annotations
> added to a canary (by a user, Argo CD, Kustomize, ...) are removed again on
> the next canary check. GitOps tools then keep fighting with the operator.
> # *Removing the canary label does not stop the canary checks* - The label is
> added back on the next check, so a resource cannot stop being a canary
> without being deleted.
> # *Any failed spec update makes the operator unhealthy* - A single failed
> update, for example a transient API server error (or a not found error = 404)
> or a concurrent write, makes the next check fail. The unhealthy state lasts
> until the check after that, longer than the default liveness probe budget, so
> the operator is restarted although it reconciles correctly.
> # *A canary that is gone or no longer watched restarts the operator* - If
> the canary is deleted without the operator cleanup (e.g. its finalizer was
> removed manually), or its namespace is removed from
> {{kubernetes.operator.watched.namespaces}} at runtime
> ({{{}kubernetes.operator.dynamic.namespaces.enabled: true{}}}), every
> following update fails and the operator is restarted.
> h1. Root cause
> h2. The canary is updated from a stale copy
> The spec update uses {{replace()}} on the canary copy cached at the last
> reconcile. Metadata-only changes do not change the generation, so they are
> not reconciled and the cached copy (labels, annotations, resource version)
> goes stale. The update then gets a {{{}409 Conflict{}}}, and the fabric8
> client retries {{replace()}} by refreshing only the resource version and
> sending the same stale copy again. The update succeeds and overwrites every
> metadata change made since the last reconcile, including the removal of the
> canary label.
> h2. The health check does not depend on the spec update
> A failed update is only logged ("Could not bump canary deployment, it may
> have been deleted"), and the health check is scheduled anyway. The check then
> compares the reconciled generation with a generation that was never written,
> so it always fails, whatever the cause of the failure:
> * a transient error fails one check, which is enough to restart the operator.
> * a permanent error ({{{}404{}}} for a canary deleted without cleanup,
> {{403}} for a namespace that is no longer watched and whose Role was removed)
> fails every check.
> h2. The canary is not dropped when the operator stops owning it
> The tracked canaries are only dropped by the canary deletion cleanup. Nothing
> drops a canary when the operator stops watching its namespace, or when the
> canary label is removed, so the operator keeps updating resources it no
> longer reconciles.
> h1. How to reproduce
> h2. Metadata changes reverted
> # Operator watching the {{default}} namespace, apply the example canary:
> {code:bash}
> kubectl apply -f examples/canary.yaml
> {code}
> # After a canary check (e.g. {{{}restartNonce: 2{}}}), add a label and an
> annotation:
> {code:bash}
> kubectl label flinkdeployment canary repro=409
> kubectl annotate flinkdeployment canary repro/note=keep-me
> {code}
> # After the next canary check ({{{}restartNonce: 3{}}}), the label {{repro}}
> and the annotation {{repro/note}} are gone:
> {code:bash}
> kubectl get flinkdeployment canary -o
> jsonpath='{.metadata.labels}{"\n"}{.metadata.annotations}{"\n"}{.spec}{"\n"}'
> {code}
> Removing the {{flink.apache.org/canary}} label instead of adding one behaves
> the same way, the label comes back on the next check.
> h2. Namespace removed from the watch list
> # Operator installed with Helm, namespace-scoped RBAC, {{watchNamespaces:
> [default, canary]}} and {{{}kubernetes.operator.dynamic.namespaces.enabled:
> true{}}}. Apply the canary in the {{canary}} namespace and wait until it is
> reconciled:
> {code:bash}
> kubectl create namespace canary
> kubectl apply -f examples/canary.yaml -n canary
> {code}
> # Run {{helm upgrade}} with {{{}watchNamespaces: [default]{}}}, without
> deleting the namespace. The operator stops watching {{canary}} and its Role
> there is removed.
> # The next canary update fails, the check is evaluated anyway, and the probe
> reports unhealthy until the operator is restarted:
> {code:java}
> WARN o.a.f.k.o.h.CanaryResourceManager Could not bump canary deployment, it
> may have been deleted
> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
> PUT at: .../namespaces/canary/flinkdeployments/canary. Message:
> flinkdeployments.flink.apache.org "canary" is forbidden: User
> "system:serviceaccount:default:flink-operator" cannot update resource
> "flinkdeployments" in API group "flink.apache.org" in the namespace "canary".
> Received status: Status(apiVersion=v1, code=403, ...)
> ...
> ERROR o.a.f.k.o.h.CanaryResourceManager Canary deployment
> ResourceID{name='canary', namespace='canary'} latest spec not reconciled.
> Expected generation larger than 2, received 2
> ERROR o.a.f.k.operator.health.HealthProbe Unhealthy canary resources
> ERROR o.a.f.k.operator.health.HealthProbe Unhealthy canary resources
> ERROR o.a.f.k.operator.health.HealthProbe Unhealthy canary resources
> INFO i.javaoperatorsdk.operator.Operator Operator SDK 5.5.0 is shutting
> down...
> {code}
> {code:bash}
> kubectl get pods
> NAME READY STATUS RESTARTS
> AGE
> flink-kubernetes-operator-5bdbb9d45f-pvbsl 2/2 Running 1 (43s ago)
> 3m24s
> {code}
> h2. Canary deleted without cleanup
> # Apply the example canary and wait for the first canary check.
> # Remove its finalizer and delete it:
> {code:bash}
> kubectl patch flinkdeployment canary --type=json
> -p='[{"op":"remove","path":"/metadata/finalizers"}]'
> kubectl delete flinkdeployment canary
> {code}
> # Every following check fails and the operator is restarted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
