[
https://issues.apache.org/jira/browse/YARN-11022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-11022:
----------------------------------
Description:
The documentation does not mention that the max-parallel-apps property is
inherited. The property can be overridden on a per queue basis, but the
parent(s) can also restrict how many parallel apps can be run.
{*}yarn.scheduler.capacity.max-parallel-apps /
yarn.scheduler.capacity.<queue-path>.max-parallel-apps{*}:
{quote}
Maximum number of applications that can run at the same time. Unlike to
maximum-applications, application submissions are not rejected when this limit
is reached. Instead they stay in ACCEPTED state until they are eligible to run.
This can be set for all queues with yarn.scheduler.capacity.max-parallel-apps
and can also be overridden on a per queue basis by setting
yarn.scheduler.capacity.<queue-path>.max-parallel-apps. Integer value is
expected. By default, there is no limit.
{quote}
[https://github.com/apache/hadoop/blob/03cfc852791c14fad39db4e5b14104a276c08e59/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSMaxRunningAppsEnforcer.java#L99]
{code}
private boolean exceedQueueMaxParallelApps(AbstractCSQueue queue) {
// Check queue and all parent queues while (queue != null) {
if (queue.getNumRunnableApps() >= queue.getMaxParallelApps()) {
LOG.info("Maximum runnable apps exceeded for queue {}",
queue.getQueuePath());
return true;
}
queue = (AbstractCSQueue) queue.getParent();
} return false;
}
{code}
Example:
Let's say the user configured the *yarn.scheduler.capacity.max-parallel-apps*
to 250, that will be the default for queues that doesn't override the setting.
([https://github.com/apache/hadoop/blob/32ecaed9c3c06a48ef01d0437e62e8faccd3e9f3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1688])
Given this queue hierarchy:
||root||.a||.a1||.a2||.a3||.a4||
|500|default|50|10|default|15|
||root||.a||.b||
|500|default|50|
- maximum 250 apps can run parallel under the *root.a* queues.
- maximum 50 apps can run parallel under the *root.a.a1* queues.
- maximum 10 apps can run parallel under the *root.a.a1.a2* queues.
- maximum *10* apps can run parallel under the *root.a1.a2.a3* queues. (even
though the max-parallel-apps is not set for .a3 so the default 250 applies for
that queue, but it's parent had a lower value, and children can't exceed that)
- maximum *10* apps can run parallel under the *root.a1.a2.a3.a4* queue. (even
though it's configured for 15, the parents restrict this limit to 10)
- maximum 50 apps can run parallel under the *root.a.b* queue.
was:
The documentation does not mention that the max-parallel-apps property is
inherited. The property can be overridden on a per queue basis, but the
parent(s) can also restrict how many parallel apps can be run.
{*}yarn.scheduler.capacity.max-parallel-apps /
yarn.scheduler.capacity.<queue-path>.max-parallel-apps{*}:
{quote}
Maximum number of applications that can run at the same time. Unlike to
maximum-applications, application submissions are not rejected when this limit
is reached. Instead they stay in ACCEPTED state until they are eligible to run.
This can be set for all queues with yarn.scheduler.capacity.max-parallel-apps
and can also be overridden on a per queue basis by setting
yarn.scheduler.capacity.<queue-path>.max-parallel-apps. Integer value is
expected. By default, there is no limit.
{quote}
[https://github.com/apache/hadoop/blob/03cfc852791c14fad39db4e5b14104a276c08e59/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSMaxRunningAppsEnforcer.java#L99]
{code}
private boolean exceedQueueMaxParallelApps(AbstractCSQueue queue) {
// Check queue and all parent queues while (queue != null) {
if (queue.getNumRunnableApps() >= queue.getMaxParallelApps()) {
LOG.info("Maximum runnable apps exceeded for queue {}",
queue.getQueuePath());
return true;
}
queue = (AbstractCSQueue) queue.getParent();
} return false;
}
{code}
Example:
Let's say the user configured the *yarn.scheduler.capacity.max-parallel-apps*
to 250, that will be the default for queues that doesn't override the setting.
([https://github.com/apache/hadoop/blob/32ecaed9c3c06a48ef01d0437e62e8faccd3e9f3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1688])
Given this queue hierarchy:
||root||.a||.a1||.a2||.a3||.a4||
|500|default|50|10|default|15|
||root||.a||.b||
|500|default|50|
- maximum 250 apps can run parallel under the *root.a* queues.
- maximum 50 apps can run parallel under the *root.a.a1* queues.
- maximum 10 apps can run parallel under the *root.a.a1.a2* queues.
- maximum *10* apps can run parallel under the *root.a1.a2.a3* queues. (even
though the max-parallel-apps is not set for .a3 so the default 250 applies for
that queue, but it's parent had a lower value, and children can't exceed that)
- maximum *10* apps can run parallel under the *root.a1.a2.a3.a4* queue. (even
though it's configured for 15, the parents restrict this limit to 10)
- maximum 50 apps can run parallel under the *root.a.b* queue.
> Fix the documentation for max-parallel-apps in CS
> -------------------------------------------------
>
> Key: YARN-11022
> URL: https://issues.apache.org/jira/browse/YARN-11022
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
> Affects Versions: 3.4.0
> Reporter: Tamas Domok
> Assignee: Tamas Domok
> Priority: Major
>
> The documentation does not mention that the max-parallel-apps property is
> inherited. The property can be overridden on a per queue basis, but the
> parent(s) can also restrict how many parallel apps can be run.
>
> {*}yarn.scheduler.capacity.max-parallel-apps /
> yarn.scheduler.capacity.<queue-path>.max-parallel-apps{*}:
> {quote}
> Maximum number of applications that can run at the same time. Unlike to
> maximum-applications, application submissions are not rejected when this
> limit is reached. Instead they stay in ACCEPTED state until they are eligible
> to run. This can be set for all queues with
> yarn.scheduler.capacity.max-parallel-apps and can also be overridden on a per
> queue basis by setting
> yarn.scheduler.capacity.<queue-path>.max-parallel-apps. Integer value is
> expected. By default, there is no limit.
> {quote}
>
> [https://github.com/apache/hadoop/blob/03cfc852791c14fad39db4e5b14104a276c08e59/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSMaxRunningAppsEnforcer.java#L99]
> {code}
> private boolean exceedQueueMaxParallelApps(AbstractCSQueue queue) {
> // Check queue and all parent queues while (queue != null) {
> if (queue.getNumRunnableApps() >= queue.getMaxParallelApps()) {
> LOG.info("Maximum runnable apps exceeded for queue {}",
> queue.getQueuePath());
> return true;
> }
> queue = (AbstractCSQueue) queue.getParent();
> } return false;
> }
> {code}
> Example:
> Let's say the user configured the *yarn.scheduler.capacity.max-parallel-apps*
> to 250, that will be the default for queues that doesn't override the
> setting.
> ([https://github.com/apache/hadoop/blob/32ecaed9c3c06a48ef01d0437e62e8faccd3e9f3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1688])
> Given this queue hierarchy:
> ||root||.a||.a1||.a2||.a3||.a4||
> |500|default|50|10|default|15|
> ||root||.a||.b||
> |500|default|50|
> - maximum 250 apps can run parallel under the *root.a* queues.
> - maximum 50 apps can run parallel under the *root.a.a1* queues.
> - maximum 10 apps can run parallel under the *root.a.a1.a2* queues.
> - maximum *10* apps can run parallel under the *root.a1.a2.a3* queues. (even
> though the max-parallel-apps is not set for .a3 so the default 250 applies
> for that queue, but it's parent had a lower value, and children can't exceed
> that)
> - maximum *10* apps can run parallel under the *root.a1.a2.a3.a4* queue.
> (even though it's configured for 15, the parents restrict this limit to 10)
> - maximum 50 apps can run parallel under the *root.a.b* queue.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
