Dennis-Mircea Ciupitu created FLINK-41441:
---------------------------------------------
Summary: Operator silently runs on the JDK HTTP client instead of
OkHttp
Key: FLINK-41441
URL: https://issues.apache.org/jira/browse/FLINK-41441
Project: Flink
Issue Type: Bug
Components: Kubernetes Operator
Reporter: Dennis-Mircea Ciupitu
h1. Problem
Every official image since 1.13.0 logs this at startup:
{noformat}
2026-11-10 11:44:05,954 i.f.k.c.u.HttpClientUtils [WARN ] The following
httpclient factories were detected on your classpath:
[io.fabric8.kubernetes.client.jdkhttp.JdkHttpClientFactory,
io.fabric8.kubernetes.client.okhttp.OkHttpClientFactory], multiple of which had
the same priority (0) so one was chosen randomly. You should exclude
dependencies that aren’t needed or use an explicit association of the
HttpClient.Factory.
{noformat}
and then runs on fabric8's JDK HTTP client, although the project selects,
documents and tests OkHttp ({{{}fabric8.httpclient.impl{}}} defaults to
{{{}okhttp{}}}, the {{depend-on-okhttp4}} profile, the [IPv6
Configuration|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/deployment/configuration/#ipv6-configuration]
note in the configuration docs). The webhook builds its client from the same
classpath, so it runs on the JDK client too.
h1. Root cause
* The shade plugin merges the two
{{META-INF/services/io.fabric8.kubernetes.client.http.HttpClient$Factory}}
files, JDK first. Neither factory overrides {{{}priority(){}}}, so fabric8
takes the first one listed. "Randomly" in the warning is deterministic in
practice: always the JDK factory.
* JOSDK's {{operator-framework}} depends on {{kubernetes-httpclient-<impl>}}
with the implementation taken from its parent pom property
{{{}fabric8-httpclient-impl.name{}}}, which is {{jdk}} in every JOSDK 5.x
release.
*
The poms only ever exclude the OkHttp client that fabric8 and JOSDK used to
bring, and add the operator's own {{kubernetes-httpclient-<impl>}} next to it.
Nothing excludes {{{}kubernetes-httpclient-jdk{}}}, so since the JOSDK 5.1
upgrade the exclusions match nothing relevant and the JDK client rides in
alongside the OkHttp one.
h1. Impact
The 2 HTTP client implementations differ in how they keep watches alive:
* OkHttp pings every watch WebSocket every 30 seconds
({{{}kubernetes.websocket.ping.interval{}}}), so a connection that dies without
a FIN or RST signal (API server replica replaced, load balancer state dropped)
is detected within a minute.
* The JDK implementation sends no pings and has no keepalive, so the same
connection is only noticed when the client happens to write on it and the
network answers with an RST.
This behaviour was measured locally on Minikube with a connection made
half-open on purpose:
* operator built with a single JDK client factory: a FlinkDeployment created
right after the failure stays without status, event or log line for about 2
minutes, the health endpoint reports 200 throughout, then the watches are
re-opened one by one over the next minutes;
* operator built with a single OkHttp client factory: all connections dropped
46 seconds after the failure, re-established within 15 seconds of the network
returning, the FlinkDeployment reconciled 30 seconds later, no silent window.
So the bug does not make the operator permanently blind, it removes the fastest
detection path and leaves a silent window of minutes after every connection
loss, during which the health probe reports healthy.
h1. Expected Behaviour
* Exactly one fabric8 HTTP client factory on the classpath, the one selected
by {{fabric8.httpclient.impl.}}
* The implementation in use visible in the startup log;
* A guard so that the next change of JOSDK's transitive client cannot regress
this silently.
h1. Related
The production incident that led here (a FlinkDeployment ignored for 12+ hours
with a green health probe, fixed by an operator restart) had no connection
failure in the operator log at all. Its watch was alive and silent, which
points to a stale watch cache on the API server replica the operator was pinned
to. Currently, no HTTP client can detect this issue, and this gap must be
addressed in a separate ticket that improves health-check functionality.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
