Dennis-Mircea Ciupitu created FLINK-41050:
---------------------------------------------
Summary: Canary health checks revert canary metadata changes and
restart a healthy operator
Key: FLINK-41050
URL: https://issues.apache.org/jira/browse/FLINK-41050
Project: Flink
Issue Type: Bug
Components: Kubernetes Operator
Reporter: Dennis-Mircea Ciupitu
h1. Problem
Canary resources let the operator health probe verify that spec changes are
still picked up and reconciled. Every
{{kubernetes.operator.health.canary.resource.timeout}} cycle (60s by default),
the operator bumps the canary {{spec.restartNonce}} and checks that the
previous bump was reconciled in time. If not, the probe reports the operator
unhealthy and Kubernetes restarts it.
The way the canary spec is updated has several problems. Users can lose their
changes to the canary, the operator can modify resources that are no longer
canaries, and the operator can be restarted although it works fine:
# *Metadata changes on the canary are reverted* - Labels or annotations added
to a canary (by a user, Argo CD, Kustomize, ...) are removed again on the next
canary check. GitOps tools then keep fighting with the operator.
# *Removing the canary label does not stop the canary checks* - The label is
added back on the next check, so a resource cannot stop being a canary without
being deleted.
# *Any failed spec update makes the operator unhealthy* - A single failed
update, for example a transient API server error (or a not found error = 404)
or a concurrent write, makes the next check fail. The unhealthy state lasts
until the check after that, longer than the default liveness probe budget, so
the operator is restarted although it reconciles correctly.
# *A canary that is gone or no longer watched restarts the operator* - If the
canary is deleted without the operator cleanup (e.g. its finalizer was removed
manually), or its namespace is removed from
{{kubernetes.operator.watched.namespaces}} at runtime
({{{}kubernetes.operator.dynamic.namespaces.enabled: true{}}}), every following
update fails and the operator is restarted.
h1. Root cause
h2. The canary is updated from a stale copy
The spec update uses {{replace()}} on the canary copy cached at the last
reconcile. Metadata-only changes do not change the generation, so they are not
reconciled and the cached copy (labels, annotations, resource version) goes
stale. The update then gets a {{{}409 Conflict{}}}, and the fabric8 client
retries {{replace()}} by refreshing only the resource version and sending the
same stale copy again. The update succeeds and overwrites every metadata change
made since the last reconcile, including the removal of the canary label.
h2. The health check does not depend on the spec update
A failed update is only logged ("Could not bump canary deployment, it may have
been deleted"), and the health check is scheduled anyway. The check then
compares the reconciled generation with a generation that was never written, so
it always fails, whatever the cause of the failure:
* a transient error fails one check, which is enough to restart the operator.
* a permanent error ({{{}404{}}} for a canary deleted without cleanup, {{403}}
for a namespace that is no longer watched and whose Role was removed) fails
every check.
h2. The canary is not dropped when the operator stops owning it
The tracked canaries are only dropped by the canary deletion cleanup. Nothing
drops a canary when the operator stops watching its namespace, or when the
canary label is removed, so the operator keeps updating resources it no longer
reconciles.
h1. How to reproduce
h2. Metadata changes reverted
# Operator watching the {{default}} namespace, apply the example canary:
{code:bash}
kubectl apply -f examples/canary.yaml
{code}
# After a canary check (e.g. {{{}restartNonce: 2{}}}), add a label and an
annotation:
{code:bash}
kubectl label flinkdeployment canary repro=409
kubectl annotate flinkdeployment canary repro/note=keep-me
{code}
# After the next canary check ({{{}restartNonce: 3{}}}), the label {{repro}}
and the annotation {{repro/note}} are gone:
{code:bash}
kubectl get flinkdeployment canary -o
jsonpath='{.metadata.labels}{"\n"}{.metadata.annotations}{"\n"}{.spec}{"\n"}'
{code}
Removing the {{flink.apache.org/canary}} label instead of adding one behaves
the same way, the label comes back on the next check.
h2. Namespace removed from the watch list
# Operator installed with Helm, namespace-scoped RBAC, {{watchNamespaces:
[default, canary]}} and {{{}kubernetes.operator.dynamic.namespaces.enabled:
true{}}}. Apply the canary in the {{canary}} namespace and wait until it is
reconciled:
{code:bash}
kubectl create namespace canary
kubectl apply -f examples/canary.yaml -n canary
{code}
# Run {{helm upgrade}} with {{{}watchNamespaces: [default]{}}}, without
deleting the namespace. The operator stops watching {{canary}} and its Role
there is removed.
# The next canary update fails, the check is evaluated anyway, and the probe
reports unhealthy until the operator is restarted:
{code:java}
WARN o.a.f.k.o.h.CanaryResourceManager Could not bump canary deployment, it
may have been deleted
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PUT
at: .../namespaces/canary/flinkdeployments/canary. Message:
flinkdeployments.flink.apache.org "canary" is forbidden: User
"system:serviceaccount:default:flink-operator" cannot update resource
"flinkdeployments" in API group "flink.apache.org" in the namespace "canary".
Received status: Status(apiVersion=v1, code=403, ...)
...
ERROR o.a.f.k.o.h.CanaryResourceManager Canary deployment
ResourceID{name='canary', namespace='canary'} latest spec not reconciled.
Expected generation larger than 2, received 2
ERROR o.a.f.k.operator.health.HealthProbe Unhealthy canary resources
ERROR o.a.f.k.operator.health.HealthProbe Unhealthy canary resources
ERROR o.a.f.k.operator.health.HealthProbe Unhealthy canary resources
INFO i.javaoperatorsdk.operator.Operator Operator SDK 5.5.0 is shutting down...
{code}
{code:bash}
kubectl get pods
NAME READY STATUS RESTARTS AGE
flink-kubernetes-operator-5bdbb9d45f-pvbsl 2/2 Running 1 (43s ago)
3m24s
{code}
h2. Canary deleted without cleanup
# Apply the example canary and wait for the first canary check.
# Remove its finalizer and delete it:
{code:bash}
kubectl patch flinkdeployment canary --type=json
-p='[{"op":"remove","path":"/metadata/finalizers"}]'
kubectl delete flinkdeployment canary
{code}
# Every following check fails and the operator is restarted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
