[
https://issues.apache.org/jira/browse/YARN-5545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naganarasimha G R updated YARN-5545:
------------------------------------
Description:
Issues as part of Max apps in Capacity scheduler:
1. Cap total applications across the queue hierarchy based on existing max app
calculation
2. Introduce a new configuration to take default max apps per queue
irrespective of the queue capacity configuration
3. When the capacity configuration of the default partition is ZERO but queue
has capacity for other partition then app is not getting submitted, though app
is submitted in other partition
Steps to reproduce Issue 3 :
Configure capacity scheduler
yarn.scheduler.capacity.root.default.capacity=0
yarn.scheduler.capacity.root.queue1.accessible-node-labels.labelx.capacity=50
yarn.scheduler.capacity.root.default.accessible-node-labels.labelx.capacity=50
Submit application as below
./yarn jar
../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha2-SNAPSHOT-tests.jar
sleep -Dmapreduce.job.node-label-expression=labelx
-Dmapreduce.job.queuename=default -m 1 -r 1 -mt 10000000 -rt 1
{noformat}
2016-08-21 18:21:31,375 INFO mapreduce.JobSubmitter: Cleaning up the staging
area /tmp/hadoop-yarn/staging/root/.staging/job_1471670113386_0001
java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to
submit application_1471670113386_0001 to YARN :
org.apache.hadoop.security.AccessControlException: Queue root.default already
has 0 applications, cannot accept submission of application:
application_1471670113386_0001
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:316)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:255)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
...
Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit
application_1471670113386_0001 to YARN :
org.apache.hadoop.security.AccessControlException: Queue root.default already
has 0 applications, cannot accept submission of application:
application_1471670113386_0001
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:286)
at
org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:296)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
... 25 more
{noformat}
was:
Issues as part of Max apps in Capacity scheduler:
1. Cap total applications across the queue hierarchy based on existing max app
calculation
2. Introduce a new configuration to take default max apps per queue
irrespective of the queue capacity configuration
3. When the capacity configuration of the default partition is ZERO but queue
has capacity for other partition then app is not getting submitted though
submitted
Configure capacity scheduler
yarn.scheduler.capacity.root.default.capacity=0
yarn.scheduler.capacity.root.queue1.accessible-node-labels.labelx.capacity=50
yarn.scheduler.capacity.root.default.accessible-node-labels.labelx.capacity=50
Submit application as below
./yarn jar
../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha2-SNAPSHOT-tests.jar
sleep -Dmapreduce.job.node-label-expression=labelx
-Dmapreduce.job.queuename=default -m 1 -r 1 -mt 10000000 -rt 1
{noformat}
2016-08-21 18:21:31,375 INFO mapreduce.JobSubmitter: Cleaning up the staging
area /tmp/hadoop-yarn/staging/root/.staging/job_1471670113386_0001
java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to
submit application_1471670113386_0001 to YARN :
org.apache.hadoop.security.AccessControlException: Queue root.default already
has 0 applications, cannot accept submission of application:
application_1471670113386_0001
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:316)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:255)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
...
Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit
application_1471670113386_0001 to YARN :
org.apache.hadoop.security.AccessControlException: Queue root.default already
has 0 applications, cannot accept submission of application:
application_1471670113386_0001
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:286)
at
org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:296)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
... 25 more
{noformat}
> Fix issues related to Max App in capacity scheduler
> ---------------------------------------------------
>
> Key: YARN-5545
> URL: https://issues.apache.org/jira/browse/YARN-5545
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
> Reporter: Bibin A Chundatt
> Assignee: Bibin A Chundatt
> Labels: oct16-medium
> Attachments: YARN-5545.0001.patch, YARN-5545.0002.patch,
> YARN-5545.0003.patch, YARN-5545.0005.patch, YARN-5545.0006.patch,
> YARN-5545.0007.patch, YARN-5545.0008.patch, YARN-5545.004.patch,
> capacity-scheduler.xml
>
>
> Issues as part of Max apps in Capacity scheduler:
> 1. Cap total applications across the queue hierarchy based on existing max
> app calculation
> 2. Introduce a new configuration to take default max apps per queue
> irrespective of the queue capacity configuration
> 3. When the capacity configuration of the default partition is ZERO but queue
> has capacity for other partition then app is not getting submitted, though
> app is submitted in other partition
> Steps to reproduce Issue 3 :
> Configure capacity scheduler
> yarn.scheduler.capacity.root.default.capacity=0
> yarn.scheduler.capacity.root.queue1.accessible-node-labels.labelx.capacity=50
> yarn.scheduler.capacity.root.default.accessible-node-labels.labelx.capacity=50
> Submit application as below
> ./yarn jar
> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha2-SNAPSHOT-tests.jar
> sleep -Dmapreduce.job.node-label-expression=labelx
> -Dmapreduce.job.queuename=default -m 1 -r 1 -mt 10000000 -rt 1
> {noformat}
> 2016-08-21 18:21:31,375 INFO mapreduce.JobSubmitter: Cleaning up the staging
> area /tmp/hadoop-yarn/staging/root/.staging/job_1471670113386_0001
> java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed
> to submit application_1471670113386_0001 to YARN :
> org.apache.hadoop.security.AccessControlException: Queue root.default already
> has 0 applications, cannot accept submission of application:
> application_1471670113386_0001
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:316)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:255)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> ...
> Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit
> application_1471670113386_0001 to YARN :
> org.apache.hadoop.security.AccessControlException: Queue root.default already
> has 0 applications, cannot accept submission of application:
> application_1471670113386_0001
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:286)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:296)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
> ... 25 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
