[
https://issues.apache.org/jira/browse/FLINK-41288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-41288:
-----------------------------------
Labels: pull-request-available (was: )
> Recreating a canary resource leaves duplicate health check loops running
> ------------------------------------------------------------------------
>
> Key: FLINK-41288
> URL: https://issues.apache.org/jira/browse/FLINK-41288
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Reporter: Dennis-Mircea Ciupitu
> Priority: Major
> Labels: pull-request-available
>
> h1. Problem
> Deleting a canary resource and creating it again with the same name within
> {{kubernetes.operator.health.canary.resource.timeout}} (e.g. {{kubectl delete
> && kubectl apply}}, {{kubectl replace --force}}) leaves two health check
> loops running for the same canary. Each further recreate adds one more loop
> until the operator restarts.
> Every loop updates the canary spec each period, so the load grows with each
> recreate. A loop can also check right after another loop updated the spec,
> before it was reconciled, and report the canary unhealthy, which can restart
> a healthy operator.
> h1. Root cause
> The canary state is keyed only by name and namespace, and the scheduled check
> is not cancelled when the state is dropped. When a canary with the same name
> is tracked again before that check runs, the old check picks up the new state
> and keeps running next to the new loop.
> The same happens when a namespace is removed from the watched namespaces and
> added back within the timeout.
> h1. How to reproduce
> {code:bash}
> kubectl apply -f examples/canary.yaml
> # wait for the first "Scheduling canary check" log
> kubectl delete -f examples/canary.yaml && kubectl apply -f
> examples/canary.yaml
> {code}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
