[
https://issues.apache.org/jira/browse/FLINK-41465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-41465:
-----------------------------------
Labels: pull-request-available (was: )
> Health probe reports healthy while the informer receives no events
> ------------------------------------------------------------------
>
> Key: FLINK-41465
> URL: https://issues.apache.org/jira/browse/FLINK-41465
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Reporter: Dennis-Mircea Ciupitu
> Priority: Major
> Labels: pull-request-available
>
> h1. Problem
> The operator learns about resources only through the watch connections of its
> informers. The health probe judges an informer by JOSDK's health indicator,
> which comes down to {{isRunning()}} and {{{}hasSynced(){}}}: the informer was
> started and completed its first list. Neither says anything about whether
> events still arrive.
> Two failure classes leave an informer silent while the probe keeps answering
> 200. The operator then ignores new resources and spec changes until someone
> restarts it. Existing deployments are still reconciled on the timer, so
> nothing else looks wrong.
> h2. 1. The watch is alive and delivers nothing
> Seen in production on operator 1.16.1: a FlinkDeployment created in a watched
> namespace had no status, no event and no log line for 12 hours. The API
> server returned it on every read and {{kubectl get --watch}} received its
> events, the operator log showed no connection error at all, and the
> operator's watch connections stayed open. This matches a stale watch cache on
> the API server replica the operator's connections were pinned to. A restart
> fixed it instantly, because the fresh informers listed everything.
> The same state is reproducible with a TLS proxy between the operator and the
> API server that forwards everything except the server-to-client data frames
> of watch connections (events and bookmarks) while keeping ping, pong and
> close frames. The watches stay connected, the pings are answered, and a
> FlinkDeployment created behind the proxy never gets a status, with the health
> endpoint at 200 on every check. The API server's own watch timeout does not
> help: it closes each watch after ten minutes, fabric8 reopens it and the
> replayed ADD event is lost again. A connection that becomes half-open, when
> the server side disappears without a TCP reset, produces a shorter blind
> window of one to two minutes with the same 200 answers.
> A resource created during such a window also depends on the next watch
> working: after a restart the first reconcile only adds the operator
> finalizer, and the actual deployment is triggered by the watch event of that
> update.
> h2. 2. The watch is down and keeps retrying
> A watch that fails after startup is retried forever by fabric8:
> {{{}InformerHealthIndicator.{}}}{{{}isRunning(){}}} and
> {{{}InformerHealthIndicator.{}}}{{{}hasSynced(){}}} stay true, so JOSDK
> reports the informer healthy.
> This is reproducible by deleting a watched namespace on an operator installed
> with namespace-scoped RBAC: the log shows an endless {{Received 403 on
> websocket}} loop while the probe stays OK. The same holds for an API server
> that becomes unreachable after startup. Only the deprecated JOSDK
> {{InformerHealthIndicator.isWatching()}} would show it, and it is not part of
> JOSDK's health status.
> h1. Root cause
> The probe evaluates the client's belief about its watch and the completion of
> the initial sync. It never looks at the one thing that proves a watch is
> alive: data arriving. The canary resources do look at it end to end, but only
> where a canary is deployed, only for the FlinkDeployment informer of that
> namespace, and at the cost of their own write traffic.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)