[
https://issues.apache.org/jira/browse/YARN-11022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-11022:
----------------------------------
Fix Version/s: 3.3.3
> Fix the documentation for max-parallel-apps in CS
> -------------------------------------------------
>
> Key: YARN-11022
> URL: https://issues.apache.org/jira/browse/YARN-11022
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
> Affects Versions: 3.4.0
> Reporter: Tamas Domok
> Assignee: Tamas Domok
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.3
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The documentation does not mention that the max-parallel-apps property is
> inherited. The property can be overridden on a per queue basis, but the
> parent(s) can also restrict how many parallel apps can be run.
>
> {*}yarn.scheduler.capacity.max-parallel-apps /
> yarn.scheduler.capacity.<queue-path>.max-parallel-apps{*}:
> {quote}
> Maximum number of applications that can run at the same time. Unlike to
> maximum-applications, application submissions are not rejected when this
> limit is reached. Instead they stay in ACCEPTED state until they are eligible
> to run. This can be set for all queues with
> yarn.scheduler.capacity.max-parallel-apps and can also be overridden on a per
> queue basis by setting
> yarn.scheduler.capacity.<queue-path>.max-parallel-apps. Integer value is
> expected. By default, there is no limit.
> {quote}
>
> [https://github.com/apache/hadoop/blob/03cfc852791c14fad39db4e5b14104a276c08e59/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSMaxRunningAppsEnforcer.java#L99]
> {code}
> private boolean exceedQueueMaxParallelApps(AbstractCSQueue queue) {
> // Check queue and all parent queues while (queue != null) {
> if (queue.getNumRunnableApps() >= queue.getMaxParallelApps()) {
> LOG.info("Maximum runnable apps exceeded for queue {}",
> queue.getQueuePath());
> return true;
> }
> queue = (AbstractCSQueue) queue.getParent();
> } return false;
> }
> {code}
> Example:
> Let's say the user configured the *yarn.scheduler.capacity.max-parallel-apps*
> to 250, that will be the default for queues that doesn't override the
> setting.
> ([https://github.com/apache/hadoop/blob/32ecaed9c3c06a48ef01d0437e62e8faccd3e9f3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1688])
> Given this queue hierarchy:
> ||root||.a||.a1||.a2||.a3||.a4||
> |500|default|50|10|default|15|
> ||root||.a||.b||
> |500|default|50|
> - maximum 250 apps can run parallel under the *root.a* queues.
> - maximum 50 apps can run parallel under the *root.a.a1* queues.
> - maximum 10 apps can run parallel under the *root.a.a1.a2* queues.
> - maximum *10* apps can run parallel under the *root.a1.a2.a3* queues. (even
> though the max-parallel-apps is not set for .a3 so the default 250 applies
> for that queue, but it's parent had a lower value, and children can't exceed
> that)
> - maximum *10* apps can run parallel under the *root.a1.a2.a3.a4* queue.
> (even though it's configured for 15, the parents restrict this limit to 10)
> - maximum 50 apps can run parallel under the *root.a.b* queue.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
