Dennis-Mircea opened a new pull request, #1257:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1257
## What is the purpose of the change
This PR enhances the current health check functionality by making the
operator health probe watch the one signal that proves an informer's watch is
alive, its resource version. A working watch moves it all the time: every event
carries a new one, and the API server sends a bookmark with the newest one
about once a minute even when nothing changes, so the version of a healthy
informer keeps moving even in a quiet namespace. FLINK-41465 describes the gap
this closes, a probe that answers 200 while a watch is open but silent.
With that signal, the operator health probe now:
- fails when the version of any informer has not moved for longer than a
configurable timeout, 15 minutes by default,
- lets the liveness probe restart the operator, whose fresh list picks up
everything that was missed,
- covers every informer, secondary ones included, without API traffic and
without deploying anything.
IIn order to have this new enhancement landing on a clear and solid
baseline, the old health package structure was reformatted first, in its own
commit. The probe must now remember the last version of each informer and when
it changed, and it must be configured. The `HealthProbe` enum singleton could
do neither cleanly, and the package around it made it worse.
The package now has 3 clear layers behind one entry point, and the staleness
check lands in the last one:
<img width="542" height="211" alt="health-check-layering"
src="https://github.com/user-attachments/assets/c05f7d23-9c03-48d0-b968-4afcb8753fe0"
/>
- `OperatorHealthService`, the lifecycle layer: created from the
configuration, started once the operator runs, closed with it, owner of the
`flink-operator-health-probe` thread.
- `OperatorHealthHandler`, the HTTP adapter layer: turns a request into
200 or 500.
- `OperatorHealthProbe`, the core logic layer: decides whether the
operator is healthy and keeps the state needed for it.
## Brief change log
Restructure of the health package, first commit:
- `OperatorHealthService` becomes the only class the operator talks to. It
is created from the configuration, started once the JOSDK operator runs, closed
with it, and never null.
- The service builds `OperatorHealthProbe` at start with everything it
needs, the runtime info and the canary managers. The probe has only final
fields, no registration calls and no setters. Probe and handler are
package-private and the 2 summary types (`InformerHealthSummary` &
`InformerIdentifier`) are nested in the probe.
- The Netty server moves into the service. One named thread serves the
endpoint, which also means the probe is never called concurrently.
- A failing probe answers `ERROR` and logs the exception. Before, the stack
trace went to whoever called port 8085.
Informer staleness check, second commit:
- New option `kubernetes.operator.health.probe.informer.stale-timeout` is
introduced, set to 15 minutes by default. A value <= ' 0 ' turns the check off.
The default sits above the 10-minute etcd progress interval, so a cluster
without the API server watch cache does not restart-loop. The docs say when it
can be lowered.
- The probe remembers the last resource version of every informer and when
it last changed. Crossing the timeout logs one ERROR log naming the informer as
`<controller-name>/<resource-type>/<namespace>`. An informer resource version
that moves again (aka came back to life) logs one INFO log and that informer is
considered non-stale again. Informers that go away after a namespace change are
forgotten.
- The JOSDK `getLastSyncResourceVersion()` method that is mandatory for this
feature to be possible lives on JOSDK's package-private `InformerWrapper`, so
it is read reflectively behind a guard and a `TODO`. Currently, this represents
the best viable option that this feature can adopt until the JOSDK framework
makes this method public.
## Verifying this change
This change can be verified by the newly added unit tests.
The change was also verified manually on Minikube, with a TLS proxy between
the operator and the API server that drops the server-to-client data frames of
every `watch=true` request and forwards everything else (ping, pong and close
frames included).
With the timeout lowered to 2 minutes, the probe flipped to 500 within 2
minutes after the last bookmark, and the liveness probe restarted the
container. Those are the new logs that demostrates the new functionality:
```log
2026-11-12 16:09:29,086 ERROR [flink-operator-health-probe-thread-1]
o.a.f.k.o.health.OperatorHealthProbe Informer
flinkdeploymentcontroller/Deployment/default received no watch event or
bookmark for 128 s, resourceVersion 2567022 unchanged
2026-11-12 16:09:29,086 ERROR [flink-operator-health-probe-thread-1]
o.a.f.k.o.health.OperatorHealthProbe Informer
flinkstatesnapshotcontroller/FlinkStateSnapshot/default received no watch event
or bookmark for 128 s, resourceVersion 2567032 unchanged
2026-11-12 16:09:29,086 ERROR [flink-operator-health-probe-thread-1]
o.a.f.k.o.health.OperatorHealthProbe Informer
flinksessionjobcontroller/FlinkSessionJob/default received no watch event or
bookmark for 128 s, resourceVersion 2567024 unchanged
2026-11-12 16:09:29,086 ERROR [flink-operator-health-probe-thread-1]
o.a.f.k.o.health.OperatorHealthProbe Informer
flinkdeploymentcontroller/FlinkStateSnapshot/default received no watch event or
bookmark for 128 s, resourceVersion 2567032 unchanged
2026-11-12 16:09:29,086 ERROR [flink-operator-health-probe-thread-1]
o.a.f.k.o.health.OperatorHealthProbe Informer
flinkbluegreendeploymentcontroller/FlinkBlueGreenDeployment/default received no
watch event or bookmark for 128 s, resourceVersion 2567034 unchanged
2026-11-12 16:09:29,086 ERROR [flink-operator-health-probe-thread-1]
o.a.f.k.o.health.OperatorHealthProbe Informer
flinksessionjobcontroller/FlinkStateSnapshot/default received no watch event or
bookmark for 128 s, resourceVersion 2567032 unchanged
2026-11-12 16:09:39,086 ERROR [flink-operator-health-probe-thread-1]
o.a.f.k.o.health.OperatorHealthProbe Informer
flinkbluegreendeploymentcontroller/FlinkDeployment/default received no watch
event or bookmark for 129 s, resourceVersion 2567039 unchanged
2026-11-12 16:09:39,086 ERROR [flink-operator-health-probe-thread-1]
o.a.f.k.o.health.OperatorHealthProbe Informer
flinkdeploymentcontroller/FlinkDeployment/default received no watch event or
bookmark for 129 s, resourceVersion 2567039 unchanged
2026-11-12 16:09:39,086 ERROR [flink-operator-health-probe-thread-1]
o.a.f.k.o.health.OperatorHealthProbe Informer
flinkstatesnapshotcontroller/FlinkDeployment/default received no watch event or
bookmark for 129 s, resourceVersion 2567039 unchanged
2026-11-12 16:09:39,086 ERROR [flink-operator-health-probe-thread-1]
o.a.f.k.o.health.OperatorHealthProbe Informer
flinksessionjobcontroller/FlinkDeployment/default received no watch event or
bookmark for 129 s, resourceVersion 2567039 unchanged
```
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changes to the `CustomResourceDescriptors`:
no
- Core observer or reconciler logic that is regularly executed: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs & JavaDocs
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change the checkbox below to `[X]` followed by the name of the tool, and
uncomment the
"Generated-by" line. See the ASF Generative Tooling Guidance for details:
https://www.apache.org/legal/generative-tooling.html
You are responsible for the quality and correctness of every change in this
PR
regardless of the tooling used. Low-effort AI-generated PRs will be closed.
See
AGENTS.md for the full guidance.
-->
- [ ] Yes (please specify the tool below)
<!--
Generated-by: [Tool Name and Version]
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]