Tamas Domok created YARN-11022:
----------------------------------
Summary: Fix the documentation for max-parallel-apps in CS
Key: YARN-11022
URL: https://issues.apache.org/jira/browse/YARN-11022
Project: Hadoop YARN
Issue Type: Bug
Components: capacity scheduler
Affects Versions: 3.4.0
Reporter: Tamas Domok
Assignee: Tamas Domok
The documentation does not mention that the max-parallel-apps property is
inherited. The property can be overridden on a per queue basis, but the
parent(s) can also restrict how many parallel apps can be run.
{*}yarn.scheduler.capacity.max-parallel-apps /
yarn.scheduler.capacity.<queue-path>.max-parallel-apps{*}: Maximum number of
applications that can run at the same time. Unlike to maximum-applications,
application submissions are not rejected when this limit is reached. Instead
they stay in ACCEPTED state until they are eligible to run. This can be set for
all queues with yarn.scheduler.capacity.max-parallel-apps and can also be
overridden on a per queue basis by setting
yarn.scheduler.capacity.<queue-path>.max-parallel-apps. Integer value is
expected. By default, there is no limit.
[https://github.com/apache/hadoop/blob/03cfc852791c14fad39db4e5b14104a276c08e59/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSMaxRunningAppsEnforcer.java#L99]
private boolean exceedQueueMaxParallelApps(AbstractCSQueue queue) {
// Check queue and all parent queues while (queue != null) {
if (queue.getNumRunnableApps() >= queue.getMaxParallelApps()) {
LOG.info("Maximum runnable apps exceeded for queue {}",
queue.getQueuePath());
return true;
}
queue = (AbstractCSQueue) queue.getParent();
} return false;
}
Example:
Let's say the user configured the *yarn.scheduler.capacity.max-parallel-apps*
to 250, that will be the default for queues that doesn't override the setting.
([https://github.com/apache/hadoop/blob/32ecaed9c3c06a48ef01d0437e62e8faccd3e9f3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1688])
Given this queue hierarchy:
||root||.a||.a1||.a2||.a3||.a4||
|500|default|50|10|default|15|
||root||.a||.b||
|500|default|50|
- maximum 250 apps can run parallel under the *root.a* queues.
- maximum 50 apps can run parallel under the *root.a.a1* queues.
- maximum 10 apps can run parallel under the *root.a.a1.a2* queues.
- maximum *10* apps can run parallel under the *root.a1.a2.a3* queues. (even
though the max-parallel-apps is not set for .a3 so the default 250 applies for
that queue, but it's parent had a lower value, and children can't exceed that)
- maximum *10* apps can run parallel under the *root.a1.a2.a3.a4* queue. (even
though it's configured for 15, the parents restrict this limit to 10)
- maximum 50 apps can run parallel under the *root.a.b* queue.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
