[
https://issues.apache.org/jira/browse/NIFI-16120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097078#comment-18097078
]
David Handermann commented on NIFI-16120:
-----------------------------------------
Thanks for reporting this issue [~karmnik].
As described, the live call to the Kubernetes Lease API can generate many
requests, so implementing a more efficient strategy seems like an important
improvement. Before moving to a proposed solution, however, can you provide
more details on the particular Processor configuration that resulted in
observing this behavior?
The current approach has remained unchanged for some time, so it would be
helpful to understand a bit more about the Processor and flow configuration
that led to the thousands of requests.
As with any kind of caching strategy, it will be important to control how long
to trust the local information before issuing another request to the Kubernetes
REST API. This is some I have evaluated in the past, so after getting more
details, we should be able to move in the direction of an improved solution.
> KubernetesLeaderElectionManager.isLeader() does an uncached Kubernetes API
> GET on the scheduling hot path, causing excessive lease requests
> -------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-16120
> URL: https://issues.apache.org/jira/browse/NIFI-16120
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 2.5.0, 2.10.0
> Reporter: Michal S
> Priority: Major
>
> h2. Summary
> With
> {{nifi.cluster.leader.election.implementation=KubernetesLeaderElectionManager}},
> NiFi issues a live, uncached GET on the leader-election Lease on *every*
> call to {{isLeader()}}. Because {{isLeader()}} is on the processor scheduling
> hot path, a cluster with many "Primary Node Only" processors generates
> hundreds of Lease GET requests per second per node against the Kubernetes API
> server.
> h2. Affects
> Confirmed on 2.5.0. The relevant code is unchanged on current {{main}}, so
> all 2.x releases using Kubernetes leader election are affected. Not present
> with ZooKeeper (CuratorLeaderElectionManager), which answers isLeader() from
> local state.
> h2. Root cause
> {{org.apache.nifi.kubernetes.leader.election.KubernetesLeaderElectionManager.isLeader(String)}}
> calls {{getLeader(roleName)}} ->
> {{StandardLeaderElectionCommandProvider.findLeader()}} ->
> {{kubernetesClient.leases()...get()}} (a live API GET) on every invocation.
> The manager already keeps a local {{roleLeaders}} map, updated by the fabric8
> {{LeaderElector}} {{onNewLeader}} callback ({{setRoleLeader}}), but
> {{isLeader()}} ignores it.
> Call path on the hot loop:
> {{ConnectableTask.isRunOnCluster()}} -> {{FlowController.isPrimary()}} ->
> {{KubernetesLeaderElectionManager.isLeader()}} -> getLeader() -> live Lease
> GET.
> This runs for every scheduling invocation of a Primary-Node-Only processor
> (on non-primary nodes it is invoked and skipped repeatedly), so lease GET
> volume scales with (running Primary-Node-Only processors x scheduling
> frequency x concurrent tasks).
> h2. Impact / evidence
> On a 2-node EKS cluster with a few dozen active Primary-Node-Only processors:
> ~32,000 Lease GET/min per non-leader node (~640/s), all HTTP 200 -> ~46M
> Kubernetes audit events/day. Thread dumps show many "Timer-Driven Process
> Thread" stacks in isRunOnCluster -> isPrimary -> getLeader ->
> BaseOperation.get. Stopping the flows dropped the rate ~99% instantly; an
> identical but idle cluster stayed at baseline. Reproduced independently of
> fabric8 client version (7.3.1 / 7.7.0 / 7.8.0) and HTTP client implementation
> (JDK HttpClient and OkHttp) - it is not a client issue.
> h2. Proposed fix
> Make {{isLeader()}} read the locally-tracked leader from {{roleLeaders}}
> (already maintained by the onNewLeader callback), with a one-time live
> fallback only before the first observation. This mirrors
> CuratorLeaderElectionManager and removes the API GET from the hot path.
> {code:java}
> } else {
> String leaderId = roleLeaders.get(roleName);
> if (leaderId == null) {
> leaderId = getLeader(roleName).orElse(null); // one-time fallback
> before first observation
> }
> leader = participantId.equals(leaderId);
> ...
> }
> {code}
> Verified: steady-state Lease GET drops from ~32k/min to baseline (a few
> hundred/min) even under a synthetic load of 21 Primary-Node-Only processors
> at ~12k tasks/s. A short-TTL cache on getLeader() is an alternative. I can
> supply a pull request.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
