Dennis-Mircea Ciupitu created FLINK-41288:
---------------------------------------------
Summary: Recreating a canary resource leaves duplicate health
check loops running
Key: FLINK-41288
URL: https://issues.apache.org/jira/browse/FLINK-41288
Project: Flink
Issue Type: Bug
Components: Kubernetes Operator
Reporter: Dennis-Mircea Ciupitu
h1. Problem
Deleting a canary resource and creating it again with the same name within
{{kubernetes.operator.health.canary.resource.timeout}} (e.g. {{kubectl delete
&& kubectl apply}}, {{kubectl replace --force}}) leaves two health check loops
running for the same canary. Each further recreate adds one more loop until the
operator restarts.
Every loop updates the canary spec each period, so the load grows with each
recreate. A loop can also check right after another loop updated the spec,
before it was reconciled, and report the canary unhealthy, which can restart a
healthy operator.
h1. Root cause
The canary state is keyed only by name and namespace, and the scheduled check
is not cancelled when the state is dropped. When a canary with the same name is
tracked again before that check runs, the old check picks up the new state and
keeps running next to the new loop.
The same happens when a namespace is removed from the watched namespaces and
added back within the timeout.
h1. How to reproduce
{code:bash}
kubectl apply -f examples/canary.yaml
# wait for the first "Scheduling canary check" log
kubectl delete -f examples/canary.yaml && kubectl apply -f examples/canary.yaml
{code}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
