[
https://issues.apache.org/jira/browse/NIFI-16120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097336#comment-18097336
]
David Handermann commented on NIFI-16120:
-----------------------------------------
Thanks for the additional background [~karmnik], having over 1600 Processors
running on the Primary Node at various intervals would certainly involve many
repeated calls to get Leader status.
The initial proposed fix looks like it would result in some issues related to
accuracy of status, but I'm evaluating an approach to implementing caching of
the leadership status for a short period of time, which should provide a
significant reduction in the number of calls to the Kubernetes API. I will
follow up with more details soon.
> KubernetesLeaderElectionManager.isLeader() does an uncached Kubernetes API
> GET on the scheduling hot path, causing excessive lease requests
> -------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-16120
> URL: https://issues.apache.org/jira/browse/NIFI-16120
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 2.5.0, 2.10.0
> Reporter: Michal S
> Assignee: David Handermann
> Priority: Major
>
> h2. Summary
> With
> {{nifi.cluster.leader.election.implementation=KubernetesLeaderElectionManager}},
> NiFi issues a live, uncached GET on the leader-election Lease on *every*
> call to {{isLeader()}}. Because {{isLeader()}} is on the processor scheduling
> hot path, a cluster with many "Primary Node Only" processors generates
> hundreds of Lease GET requests per second per node against the Kubernetes API
> server.
> h2. Affects
> Confirmed on 2.5.0. The relevant code is unchanged on current {{main}}, so
> all 2.x releases using Kubernetes leader election are affected. Not present
> with ZooKeeper (CuratorLeaderElectionManager), which answers isLeader() from
> local state.
> h2. Root cause
> {{org.apache.nifi.kubernetes.leader.election.KubernetesLeaderElectionManager.isLeader(String)}}
> calls {{getLeader(roleName)}} ->
> {{StandardLeaderElectionCommandProvider.findLeader()}} ->
> {{kubernetesClient.leases()...get()}} (a live API GET) on every invocation.
> The manager already keeps a local {{roleLeaders}} map, updated by the fabric8
> {{LeaderElector}} {{onNewLeader}} callback ({{setRoleLeader}}), but
> {{isLeader()}} ignores it.
> Call path on the hot loop:
> {{ConnectableTask.isRunOnCluster()}} -> {{FlowController.isPrimary()}} ->
> {{KubernetesLeaderElectionManager.isLeader()}} -> getLeader() -> live Lease
> GET.
> This runs for every scheduling invocation of a Primary-Node-Only processor
> (on non-primary nodes it is invoked and skipped repeatedly), so lease GET
> volume scales with (running Primary-Node-Only processors x scheduling
> frequency x concurrent tasks).
> h2. Impact / evidence
> On a 2-node EKS cluster with a few dozen active Primary-Node-Only processors:
> ~32,000 Lease GET/min per non-leader node (~640/s), all HTTP 200 -> ~46M
> Kubernetes audit events/day. Thread dumps show many "Timer-Driven Process
> Thread" stacks in isRunOnCluster -> isPrimary -> getLeader ->
> BaseOperation.get. Stopping the flows dropped the rate ~99% instantly; an
> identical but idle cluster stayed at baseline. Reproduced independently of
> fabric8 client version (7.3.1 / 7.7.0 / 7.8.0) and HTTP client implementation
> (JDK HttpClient and OkHttp) - it is not a client issue.
> h2. Proposed fix
> Make {{isLeader()}} read the locally-tracked leader from {{roleLeaders}}
> (already maintained by the onNewLeader callback), with a one-time live
> fallback only before the first observation. This mirrors
> CuratorLeaderElectionManager and removes the API GET from the hot path.
> {code:java}
> } else {
> String leaderId = roleLeaders.get(roleName);
> if (leaderId == null) {
> leaderId = getLeader(roleName).orElse(null); // one-time fallback
> before first observation
> }
> leader = participantId.equals(leaderId);
> ...
> }
> {code}
> Verified: steady-state Lease GET drops from ~32k/min to baseline (a few
> hundred/min) even under a synthetic load of 21 Primary-Node-Only processors
> at ~12k tasks/s. A short-TTL cache on getLeader() is an alternative. I can
> supply a pull request.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
