[
https://issues.apache.org/jira/browse/NIFI-16120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097086#comment-18097086
]
Michal S commented on NIFI-16120:
---------------------------------
Thanks — details below.
Environment (measured)
- NiFi 2.5.0, 2-node cluster on EKS,
\{{nifi.cluster.leader.election.implementation=KubernetesLeaderElectionManager}},
\{{nifi.state.management.provider.cluster=kubernetes-provider}}.
Flow / processor configuration (measured)
- The flow has 1,648 processors across 241 process groups, of which 103 have
Execution Node = "Primary Node Only" (\{{executionNode=PRIMARY}}).
- Schedules of those Primary-Node-Only processors are ordinary: the majority
are CRON at 15-second and daily intervals (\{{0 0/15 * * * ?}}, \{{0/15 * * * *
?}}, \{{0 0 4 * * ?}}) and TIMER_DRIVEN at 3–5 minutes, with a couple at 1
minute / 1 second; the ones we inspected use a single concurrent task.
- One integration flow alone accounts for 45 of the Primary-Node-Only
processors; while it was enabled the cluster had 300 running processors.
Observed behavior (measured)
- On the non-leader node: ~32,000 Lease GET/min (~640/s) against the
\{{primary-node}} lease, \{{verb=get}}, all HTTP 200 (no failures/retries).
Attributed to the non-leader node by source IP.
- Thread dumps show \{{Timer-Driven Process Thread}} stacks in
\{{ConnectableTask.isRunOnCluster}} → \{{FlowController.isPrimary}} →
\{{KubernetesLeaderElectionManager.isLeader}} → \{{getLeader}} →
\{{StandardLeaderElectionCommandProvider.findLeader}} → \{{BaseOperation.get}}
(a live Lease GET). In one snapshot, 8–9 of 100 timer-driven threads were
simultaneously in that live GET.
- Stopping the flow dropped the rate ~99% (to ~190/min) within about a minute.
A second cluster with the same configuration but an idle flow stayed at
baseline (~500/min).
- Reproduced identically with fabric8 kubernetes-client 7.3.1, 7.7.0 and 7.8.0,
and with both the JDK and OkHttp HTTP client implementations — so it is the
call pattern, not the client.
> KubernetesLeaderElectionManager.isLeader() does an uncached Kubernetes API
> GET on the scheduling hot path, causing excessive lease requests
> -------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-16120
> URL: https://issues.apache.org/jira/browse/NIFI-16120
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 2.5.0, 2.10.0
> Reporter: Michal S
> Priority: Major
>
> h2. Summary
> With
> {{nifi.cluster.leader.election.implementation=KubernetesLeaderElectionManager}},
> NiFi issues a live, uncached GET on the leader-election Lease on *every*
> call to {{isLeader()}}. Because {{isLeader()}} is on the processor scheduling
> hot path, a cluster with many "Primary Node Only" processors generates
> hundreds of Lease GET requests per second per node against the Kubernetes API
> server.
> h2. Affects
> Confirmed on 2.5.0. The relevant code is unchanged on current {{main}}, so
> all 2.x releases using Kubernetes leader election are affected. Not present
> with ZooKeeper (CuratorLeaderElectionManager), which answers isLeader() from
> local state.
> h2. Root cause
> {{org.apache.nifi.kubernetes.leader.election.KubernetesLeaderElectionManager.isLeader(String)}}
> calls {{getLeader(roleName)}} ->
> {{StandardLeaderElectionCommandProvider.findLeader()}} ->
> {{kubernetesClient.leases()...get()}} (a live API GET) on every invocation.
> The manager already keeps a local {{roleLeaders}} map, updated by the fabric8
> {{LeaderElector}} {{onNewLeader}} callback ({{setRoleLeader}}), but
> {{isLeader()}} ignores it.
> Call path on the hot loop:
> {{ConnectableTask.isRunOnCluster()}} -> {{FlowController.isPrimary()}} ->
> {{KubernetesLeaderElectionManager.isLeader()}} -> getLeader() -> live Lease
> GET.
> This runs for every scheduling invocation of a Primary-Node-Only processor
> (on non-primary nodes it is invoked and skipped repeatedly), so lease GET
> volume scales with (running Primary-Node-Only processors x scheduling
> frequency x concurrent tasks).
> h2. Impact / evidence
> On a 2-node EKS cluster with a few dozen active Primary-Node-Only processors:
> ~32,000 Lease GET/min per non-leader node (~640/s), all HTTP 200 -> ~46M
> Kubernetes audit events/day. Thread dumps show many "Timer-Driven Process
> Thread" stacks in isRunOnCluster -> isPrimary -> getLeader ->
> BaseOperation.get. Stopping the flows dropped the rate ~99% instantly; an
> identical but idle cluster stayed at baseline. Reproduced independently of
> fabric8 client version (7.3.1 / 7.7.0 / 7.8.0) and HTTP client implementation
> (JDK HttpClient and OkHttp) - it is not a client issue.
> h2. Proposed fix
> Make {{isLeader()}} read the locally-tracked leader from {{roleLeaders}}
> (already maintained by the onNewLeader callback), with a one-time live
> fallback only before the first observation. This mirrors
> CuratorLeaderElectionManager and removes the API GET from the hot path.
> {code:java}
> } else {
> String leaderId = roleLeaders.get(roleName);
> if (leaderId == null) {
> leaderId = getLeader(roleName).orElse(null); // one-time fallback
> before first observation
> }
> leader = participantId.equals(leaderId);
> ...
> }
> {code}
> Verified: steady-state Lease GET drops from ~32k/min to baseline (a few
> hundred/min) even under a synthetic load of 21 Primary-Node-Only processors
> at ~12k tasks/s. A short-TTL cache on getLeader() is an alternative. I can
> supply a pull request.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
