Michal S created NIFI-16120:
-------------------------------
Summary: KubernetesLeaderElectionManager.isLeader() does an
uncached Kubernetes API GET on the scheduling hot path, causing excessive lease
requests
Key: NIFI-16120
URL: https://issues.apache.org/jira/browse/NIFI-16120
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Affects Versions: 2.10.0, 2.5.0
Reporter: Michal S
h2. Summary
With
{{nifi.cluster.leader.election.implementation=KubernetesLeaderElectionManager}},
NiFi issues a live, uncached GET on the leader-election Lease on *every* call
to {{isLeader()}}. Because {{isLeader()}} is on the processor scheduling hot
path, a cluster with many "Primary Node Only" processors generates hundreds of
Lease GET requests per second per node against the Kubernetes API server.
h2. Affects
Confirmed on 2.5.0. The relevant code is unchanged on current {{main}}, so all
2.x releases using Kubernetes leader election are affected. Not present with
ZooKeeper (CuratorLeaderElectionManager), which answers isLeader() from local
state.
h2. Root cause
{{org.apache.nifi.kubernetes.leader.election.KubernetesLeaderElectionManager.isLeader(String)}}
calls {{getLeader(roleName)}} ->
{{StandardLeaderElectionCommandProvider.findLeader()}} ->
{{kubernetesClient.leases()...get()}} (a live API GET) on every invocation. The
manager already keeps a local {{roleLeaders}} map, updated by the fabric8
{{LeaderElector}} {{onNewLeader}} callback ({{setRoleLeader}}), but
{{isLeader()}} ignores it.
Call path on the hot loop:
{{ConnectableTask.isRunOnCluster()}} -> {{FlowController.isPrimary()}} ->
{{KubernetesLeaderElectionManager.isLeader()}} -> getLeader() -> live Lease GET.
This runs for every scheduling invocation of a Primary-Node-Only processor (on
non-primary nodes it is invoked and skipped repeatedly), so lease GET volume
scales with (running Primary-Node-Only processors x scheduling frequency x
concurrent tasks).
h2. Impact / evidence
On a 2-node EKS cluster with a few dozen active Primary-Node-Only processors:
~32,000 Lease GET/min per non-leader node (~640/s), all HTTP 200 -> ~46M
Kubernetes audit events/day. Thread dumps show many "Timer-Driven Process
Thread" stacks in isRunOnCluster -> isPrimary -> getLeader ->
BaseOperation.get. Stopping the flows dropped the rate ~99% instantly; an
identical but idle cluster stayed at baseline. Reproduced independently of
fabric8 client version (7.3.1 / 7.7.0 / 7.8.0) and HTTP client implementation
(JDK HttpClient and OkHttp) - it is not a client issue.
h2. Proposed fix
Make {{isLeader()}} read the locally-tracked leader from {{roleLeaders}}
(already maintained by the onNewLeader callback), with a one-time live fallback
only before the first observation. This mirrors CuratorLeaderElectionManager
and removes the API GET from the hot path.
{code:java}
} else {
String leaderId = roleLeaders.get(roleName);
if (leaderId == null) {
leaderId = getLeader(roleName).orElse(null); // one-time fallback
before first observation
}
leader = participantId.equals(leaderId);
...
}
{code}
Verified: steady-state Lease GET drops from ~32k/min to baseline (a few
hundred/min) even under a synthetic load of 21 Primary-Node-Only processors at
~12k tasks/s. A short-TTL cache on getLeader() is an alternative. I can supply
a pull request.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
