Dennis-Mircea opened a new pull request, #1257:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1257

   ## What is the purpose of the change
   
   This PR enhances the current health check functionality by making the 
operator health probe watch the one signal that proves an informer's watch is 
alive, its resource version. A working watch moves it all the time: every event 
carries a new one, and the API server sends a bookmark with the newest one 
about once a minute even when nothing changes, so the version of a healthy 
informer keeps moving even in a quiet namespace. FLINK-41465 describes the gap 
this closes, a probe that answers 200 while a watch is open but silent.
   
   With that signal, the operator health probe now:
   
     - fails when the version of any informer has not moved for longer than a 
configurable timeout, 15 minutes by default,
     - lets the liveness probe restart the operator, whose fresh list picks up 
everything that was missed,
     - covers every informer, secondary ones included, without API traffic and 
without deploying anything. 
   
   IIn order to have this new enhancement landing on a clear and solid 
baseline, the old health package structure was reformatted first, in its own 
commit. The probe must now remember the last version of each informer and when 
it changed, and it must be configured. The `HealthProbe` enum singleton could 
do neither cleanly, and the package around it made it worse.
   
   The package now has 3 clear layers behind one entry point, and the staleness 
check lands in the last one:
   <img width="542" height="211" alt="health-check-layering" 
src="https://github.com/user-attachments/assets/c05f7d23-9c03-48d0-b968-4afcb8753fe0";
 />
     - `OperatorHealthService`, the lifecycle layer: created from the 
configuration, started once the operator runs, closed with it, owner of the 
`flink-operator-health-probe` thread.
     - `OperatorHealthHandler`, the HTTP adapter layer: turns a request into 
200 or 500.
     - `OperatorHealthProbe`, the core logic layer: decides whether the 
operator is healthy and keeps the state needed for it.
   
   ## Brief change log
   
   Restructure of the health package, first commit:
   
   - `OperatorHealthService` becomes the only class the operator talks to. It 
is created from the configuration, started once the JOSDK operator runs, closed 
with it, and never null.
   - The service builds `OperatorHealthProbe` at start with everything it 
needs, the runtime info and the canary managers. The probe has only final 
fields, no registration calls and no setters. Probe and handler are 
package-private and the 2 summary types (`InformerHealthSummary` & 
`InformerIdentifier`) are nested in the probe.
   - The Netty server moves into the service. One named thread serves the 
endpoint, which also means the probe is never called concurrently.
   - A failing probe answers `ERROR` and logs the exception. Before, the stack 
trace went to whoever called port 8085.
   
   Informer staleness check, second commit:
   
   - New option `kubernetes.operator.health.probe.informer.stale-timeout` is 
introduced, set to 15 minutes by default. A value <= ' 0 ' turns the check off. 
The default sits above the 10-minute etcd progress interval, so a cluster 
without the API server watch cache does not restart-loop. The docs say when it 
can be lowered.
   - The probe remembers the last resource version of every informer and when 
it last changed. Crossing the timeout logs one ERROR log naming the informer as 
`<controller-name>/<resource-type>/<namespace>`. An informer resource version 
that moves again (aka came back to life) logs one INFO log and that informer is 
considered non-stale again. Informers that go away after a namespace change are 
forgotten.
   - The JOSDK `getLastSyncResourceVersion()` method that is mandatory for this 
feature to be possible lives on JOSDK's package-private `InformerWrapper`, so 
it is read reflectively behind a guard and a `TODO`. Currently, this represents 
the best viable option that this feature can adopt until the JOSDK framework 
makes this method public.
   
   ## Verifying this change
   This change can be verified by the newly added unit tests.
   
   The change was also verified manually on Minikube, with a TLS proxy between 
the operator and the API server that drops the server-to-client data frames of 
every `watch=true` request and forwards everything else (ping, pong and close 
frames included).
   
   With the timeout lowered to 2 minutes, the probe flipped to 500 within 2 
minutes after the last bookmark, and the liveness probe restarted the 
container. Those are the new logs that demostrates the new functionality:
   ```log
   2026-11-12 16:09:29,086 ERROR [flink-operator-health-probe-thread-1] 
o.a.f.k.o.health.OperatorHealthProbe Informer 
flinkdeploymentcontroller/Deployment/default received no watch event or 
bookmark for 128 s, resourceVersion 2567022 unchanged
   2026-11-12 16:09:29,086 ERROR [flink-operator-health-probe-thread-1] 
o.a.f.k.o.health.OperatorHealthProbe Informer 
flinkstatesnapshotcontroller/FlinkStateSnapshot/default received no watch event 
or bookmark for 128 s, resourceVersion 2567032 unchanged
   2026-11-12 16:09:29,086 ERROR [flink-operator-health-probe-thread-1] 
o.a.f.k.o.health.OperatorHealthProbe Informer 
flinksessionjobcontroller/FlinkSessionJob/default received no watch event or 
bookmark for 128 s, resourceVersion 2567024 unchanged
   2026-11-12 16:09:29,086 ERROR [flink-operator-health-probe-thread-1] 
o.a.f.k.o.health.OperatorHealthProbe Informer 
flinkdeploymentcontroller/FlinkStateSnapshot/default received no watch event or 
bookmark for 128 s, resourceVersion 2567032 unchanged
   2026-11-12 16:09:29,086 ERROR [flink-operator-health-probe-thread-1] 
o.a.f.k.o.health.OperatorHealthProbe Informer 
flinkbluegreendeploymentcontroller/FlinkBlueGreenDeployment/default received no 
watch event or bookmark for 128 s, resourceVersion 2567034 unchanged
   2026-11-12 16:09:29,086 ERROR [flink-operator-health-probe-thread-1] 
o.a.f.k.o.health.OperatorHealthProbe Informer 
flinksessionjobcontroller/FlinkStateSnapshot/default received no watch event or 
bookmark for 128 s, resourceVersion 2567032 unchanged
   2026-11-12 16:09:39,086 ERROR [flink-operator-health-probe-thread-1] 
o.a.f.k.o.health.OperatorHealthProbe Informer 
flinkbluegreendeploymentcontroller/FlinkDeployment/default received no watch 
event or bookmark for 129 s, resourceVersion 2567039 unchanged
   2026-11-12 16:09:39,086 ERROR [flink-operator-health-probe-thread-1] 
o.a.f.k.o.health.OperatorHealthProbe Informer 
flinkdeploymentcontroller/FlinkDeployment/default received no watch event or 
bookmark for 129 s, resourceVersion 2567039 unchanged
   2026-11-12 16:09:39,086 ERROR [flink-operator-health-probe-thread-1] 
o.a.f.k.o.health.OperatorHealthProbe Informer 
flinkstatesnapshotcontroller/FlinkDeployment/default received no watch event or 
bookmark for 129 s, resourceVersion 2567039 unchanged
   2026-11-12 16:09:39,086 ERROR [flink-operator-health-probe-thread-1] 
o.a.f.k.o.health.OperatorHealthProbe Informer 
flinksessionjobcontroller/FlinkDeployment/default received no watch event or 
bookmark for 129 s, resourceVersion 2567039 unchanged
   ```
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changes to the `CustomResourceDescriptors`: 
no
     - Core observer or reconciler logic that is regularly executed: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? yes
     - If yes, how is the feature documented? docs & JavaDocs
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   <!--
   If generative AI tooling has been used in the process of authoring this PR, 
please
   change the checkbox below to `[X]` followed by the name of the tool, and 
uncomment the
   "Generated-by" line. See the ASF Generative Tooling Guidance for details:
   https://www.apache.org/legal/generative-tooling.html
   
   You are responsible for the quality and correctness of every change in this 
PR
   regardless of the tooling used. Low-effort AI-generated PRs will be closed. 
See
   AGENTS.md for the full guidance.
   -->
   
   - [ ] Yes (please specify the tool below)
   
   <!--
   Generated-by: [Tool Name and Version]
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to