Chenyu Zheng created YARN-11968:
-----------------------------------
Summary: CapacityScheduler FairOrderingPolicy should avoid
repeatedly scanning applications without pending requests
Key: YARN-11968
URL: https://issues.apache.org/jira/browse/YARN-11968
Project: Hadoop YARN
Issue Type: Improvement
Components: capacity scheduler
Reporter: Chenyu Zheng
Assignee: Chenyu Zheng
I found a scenario where CapacityScheduler scheduling performance can degrade.
Assume a leaf queue contains thousands of long-running Flink applications, and
the queue uses {{{}FairOrderingPolicy{}}}. When one of the applications sends a
container {{INCREASE_RESOURCE}} request, YARN creates resource requests for
this container update. This kind of request is an in-place resource increase
for an existing container, so locality cannot be relaxed, i.e.
{{{}relaxLocality=false{}}}.
If the original node of the container does not have enough available resources
for the increase request, the request can remain pending.
In this special case, this increase request may be the only pending request in
the leaf queue, while thousands of other applications have no pending requests.
When the scheduler tries this increase request, it must be satisfied on the
original node of the container. If that node cannot satisfy the resource
condition, this scheduling attempt cannot allocate or reserve the request, and
cannot return a useful assignment result.
Because there is no assignment result to return, LeafQueue cannot stop the
current application iteration early and has to continue checking following
applications. Those thousands of applications have no pending requests, but
they may still be checked one by one. In other words, even though there is only
one real pending request, one scheduling attempt may still scan thousands of
applications without pending requests. This creates many useless scheduling
scans and reduces scheduling performance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
