Dennis-Mircea Ciupitu created FLINK-41302:
---------------------------------------------
Summary: Expose canary reconcile latency as a measure of
reconciliation pool saturation
Key: FLINK-41302
URL: https://issues.apache.org/jira/browse/FLINK-41302
Project: Flink
Issue Type: Improvement
Components: Kubernetes Operator
Reporter: Dennis-Mircea Ciupitu
h1. Motivation
Currently, a canary spec update is reconciled through the same JOSDK event
queue and reconciliation thread pool as every other resource, while the canary
reconciliation itself does almost no work. The time between a canary spec
update and its reconciliation is therefore a direct measure of how long a spec
change waits for a free reconciliation thread, in other words whether the
configured {{kubernetes.operator.reconcile.parallelism}} keeps up with the
actual load.
Today this signal is only used as a pass/fail health check: it is visible in
the logs, and when the latency exceeds
{{kubernetes.operator.health.canary.resource.timeout}} the operator is
restarted. Nothing records how the latency evolves before that point.
h1. Proposal
The proposal is to expose the canary results as operator metrics, per resource
kind and canary:
* the time between a canary spec update and its reconciliation.
* whether the canary is currently healthy.
This lets users size the reconciliation parallelism and the canary timeout
based on the observed latency, and alert on a saturated reconciliation pool
before the health probe restarts the operator.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
