[
https://issues.apache.org/jira/browse/NIFI-16120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann reassigned NIFI-16120:
---------------------------------------
Assignee: David Handermann
> KubernetesLeaderElectionManager.isLeader() does an uncached Kubernetes API
> GET on the scheduling hot path, causing excessive lease requests
> -------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-16120
> URL: https://issues.apache.org/jira/browse/NIFI-16120
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 2.5.0, 2.10.0
> Reporter: Michal S
> Assignee: David Handermann
> Priority: Major
>
> h2. Summary
> With
> {{nifi.cluster.leader.election.implementation=KubernetesLeaderElectionManager}},
> NiFi issues a live, uncached GET on the leader-election Lease on *every*
> call to {{isLeader()}}. Because {{isLeader()}} is on the processor scheduling
> hot path, a cluster with many "Primary Node Only" processors generates
> hundreds of Lease GET requests per second per node against the Kubernetes API
> server.
> h2. Affects
> Confirmed on 2.5.0. The relevant code is unchanged on current {{main}}, so
> all 2.x releases using Kubernetes leader election are affected. Not present
> with ZooKeeper (CuratorLeaderElectionManager), which answers isLeader() from
> local state.
> h2. Root cause
> {{org.apache.nifi.kubernetes.leader.election.KubernetesLeaderElectionManager.isLeader(String)}}
> calls {{getLeader(roleName)}} ->
> {{StandardLeaderElectionCommandProvider.findLeader()}} ->
> {{kubernetesClient.leases()...get()}} (a live API GET) on every invocation.
> The manager already keeps a local {{roleLeaders}} map, updated by the fabric8
> {{LeaderElector}} {{onNewLeader}} callback ({{setRoleLeader}}), but
> {{isLeader()}} ignores it.
> Call path on the hot loop:
> {{ConnectableTask.isRunOnCluster()}} -> {{FlowController.isPrimary()}} ->
> {{KubernetesLeaderElectionManager.isLeader()}} -> getLeader() -> live Lease
> GET.
> This runs for every scheduling invocation of a Primary-Node-Only processor
> (on non-primary nodes it is invoked and skipped repeatedly), so lease GET
> volume scales with (running Primary-Node-Only processors x scheduling
> frequency x concurrent tasks).
> h2. Impact / evidence
> On a 2-node EKS cluster with a few dozen active Primary-Node-Only processors:
> ~32,000 Lease GET/min per non-leader node (~640/s), all HTTP 200 -> ~46M
> Kubernetes audit events/day. Thread dumps show many "Timer-Driven Process
> Thread" stacks in isRunOnCluster -> isPrimary -> getLeader ->
> BaseOperation.get. Stopping the flows dropped the rate ~99% instantly; an
> identical but idle cluster stayed at baseline. Reproduced independently of
> fabric8 client version (7.3.1 / 7.7.0 / 7.8.0) and HTTP client implementation
> (JDK HttpClient and OkHttp) - it is not a client issue.
> h2. Proposed fix
> Make {{isLeader()}} read the locally-tracked leader from {{roleLeaders}}
> (already maintained by the onNewLeader callback), with a one-time live
> fallback only before the first observation. This mirrors
> CuratorLeaderElectionManager and removes the API GET from the hot path.
> {code:java}
> } else {
> String leaderId = roleLeaders.get(roleName);
> if (leaderId == null) {
> leaderId = getLeader(roleName).orElse(null); // one-time fallback
> before first observation
> }
> leader = participantId.equals(leaderId);
> ...
> }
> {code}
> Verified: steady-state Lease GET drops from ~32k/min to baseline (a few
> hundred/min) even under a synthetic load of 21 Primary-Node-Only processors
> at ~12k tasks/s. A short-TTL cache on getLeader() is an alternative. I can
> supply a pull request.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
