Dennis-Mircea Ciupitu created FLINK-41035:
---------------------------------------------
Summary: Operator health probe reports "All informers are
unhealthy" when a single event source fails with stop-on-informer-error=false
Key: FLINK-41035
URL: https://issues.apache.org/jira/browse/FLINK-41035
Project: Flink
Issue Type: Bug
Components: Kubernetes Operator
Reporter: Dennis-Mircea Ciupitu
h1. Problem
With {{{}kubernetes.operator.startup.stop-on-informer-error=false{}}}, the
operator is supposed to start and keep running even if some informers cannot be
started, for example because of missing RBAC permissions. The operator health
probe tolerates informers that were already unhealthy at startup, and only
reports unhealthy if an informer becomes unhealthy later, or if all informers
are unhealthy.
The "all informers are unhealthy" check does not actually look at all
informers. Whether it triggers depends on how many namespaces an event source
covers, not on the real state of the operator:
* *One watched namespace, {{Ingress}} cannot be watched:* the probe logs {{All
informers are unhealthy}} and returns 500, although every other informer
(FlinkDeployment, Deployment, FlinkSessionJob, FlinkStateSnapshot, ...) is
healthy. The startup probe then kills the operator, and this repeats on every
start.
* *Two watched namespaces, {{Ingress}} cannot be watched in one of them:* the
probe returns 200, and the failing informers are tolerated as expected.
The failure is the same in both cases (one resource type cannot be watched in
one namespace), but the outcome is opposite. In the first case
{{stop-on-informer-error=false}} has no effect, because the operator never
stays up.
h1. Root cause
{{InformerHealthSummary.fromRuntimeInfo}} computes "is any informer healthy"
from {{{}RuntimeInfo.unhealthyInformerWrappingEventSourceHealthIndicator(){}}}.
That JOSDK view only returns the event sources that contain at least one
unhealthy informer. Fully healthy event sources are filtered out.
As a result, a healthy informer is only seen when it shares an event source
with an unhealthy one, which in practice means the same resource type in
another watched namespace. With a single watched namespace (or when watching
all namespaces), every event source has exactly one informer, so the check only
ever sees the failing informers and concludes that all informers are unhealthy.
This check runs before the startup tolerance, so the tolerated informers never
get a chance to be accepted.
h1. How to reproduce
h2. Setup
* Operator installed with the Helm chart, watching only the {{default}}
namespace ({{{}watchNamespaces: [default]{}}}), namespace-scoped RBAC.
* {{kubernetes.operator.startup.stop-on-informer-error: false}}
* {{kubernetes.operator.ingress.manage}} left at its default ({{{}true{}}}),
so the FlinkDeployment and FlinkBlueGreenDeployment controllers register an
{{Ingress}} informer.
h2. Steps
# Remove the ingress permission from the operator Role:
{code:bash}
kubectl -n default get role flink-operator -o json \
| jq 'del(.rules[] | select(.apiGroups == ["networking.k8s.io"]))' \
| kubectl replace -f -
{code}
# Restart the operator:
{code:bash}
kubectl rollout restart deploy/flink-kubernetes-operator
{code}
h2. Observed
After the cache sync timeout, the two {{Ingress}} informers are tolerated as
initially unhealthy, the health endpoint starts, and the first probe already
fails:
{noformat}
2026-10-19 18:23:12,211 WARN [ingresses.networking.k8s.io/v1]
i.j.o.p.e.s.informer.InformerWrapper Informer startup error. Will periodically
retry. Informer: networking.k8s.io/v1/namespaces/default/ingresses
2026-10-19 18:23:12,213 WARN [ingresses.networking.k8s.io/v1]
i.j.o.p.e.s.informer.InformerWrapper Informer startup error. Will periodically
retry. Informer: networking.k8s.io/v1/namespaces/default/ingresses
2026-10-19 18:23:12,217 INFO main o.a.f.k.operator.health.HealthProbe
Initially unhealthy informers:
[InformerIdentifier(controllerName=flinkbluegreendeploymentcontroller,
eventSourceName=...InformerEventSource@4893b344, informerName=default),
InformerIdentifier(controllerName=flinkdeploymentcontroller,
eventSourceName=...InformerEventSource@20411320, informerName=default)]
2026-10-19 18:23:12,270 INFO main o.a.f.k.o.h.OperatorHealthService Health
probe HTTP endpoint started on port 8085
2026-10-19 18:23:20,954 ERROR nioEventLoopGroup-3-1
o.a.f.k.operator.health.HealthProbe All informers are unhealthy
{noformat}
{noformat}
Warning Unhealthy kubelet Startup probe failed: HTTP probe failed with
statuscode: 500
{noformat}
The operator container is restarted once the startup probe budget is exhausted,
and the same sequence repeats on every start.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
