[jira] [Commented] (YARN-276) Capacity Scheduler can hang when submit many jobs concurrently
[ https://issues.apache.org/jira/browse/YARN-276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539260#comment-13539260 ] nemon lou commented on YARN-276: updating the patch. Four properties have been added to CS web page: Max AM Used Per Queue Percent Actual AM Used Per Queue Percent Max AM Used Percent For Cluster Actual AM Used Percent For Cluster This patch keeps track of AM used resources and checks for it both at cluster level and leaf Queue level. Capacity Scheduler can hang when submit many jobs concurrently -- Key: YARN-276 URL: https://issues.apache.org/jira/browse/YARN-276 Project: Hadoop YARN Issue Type: Bug Components: capacityscheduler Affects Versions: 3.0.0, 2.0.1-alpha Reporter: nemon lou Attachments: YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch Original Estimate: 24h Remaining Estimate: 24h In hadoop2.0.1,When i submit many jobs concurrently at the same time,Capacity scheduler can hang with most resources taken up by AM and don't have enough resources for tasks.And then all applications hang there. The cause is that yarn.scheduler.capacity.maximum-am-resource-percent not check directly.Instead ,this property only used for maxActiveApplications. And maxActiveApplications is computed by minimumAllocation (not by Am actually used). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-276) Capacity Scheduler can hang when submit many jobs concurrently
[
https://issues.apache.org/jira/browse/YARN-276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539263#comment-13539263
]
Hadoop QA commented on YARN-276:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562321/YARN-276.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/258//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/258//console
This message is automatically generated.
Capacity Scheduler can hang when submit many jobs concurrently
--
Key: YARN-276
URL: https://issues.apache.org/jira/browse/YARN-276
Project: Hadoop YARN
Issue Type: Bug
Components: capacityscheduler
Affects Versions: 3.0.0, 2.0.1-alpha
Reporter: nemon lou
Attachments: YARN-276.patch, YARN-276.patch, YARN-276.patch,
YARN-276.patch, YARN-276.patch, YARN-276.patch
Original Estimate: 24h
Remaining Estimate: 24h
In hadoop2.0.1,When i submit many jobs concurrently at the same time,Capacity
scheduler can hang with most resources taken up by AM and don't have enough
resources for tasks.And then all applications hang there.
The cause is that yarn.scheduler.capacity.maximum-am-resource-percent not
check directly.Instead ,this property only used for maxActiveApplications.
And maxActiveApplications is computed by minimumAllocation (not by Am
actually used).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-289) Fair scheduler doesn't satisfy smaller request when bigger one is outstanding
[ https://issues.apache.org/jira/browse/YARN-289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539339#comment-13539339 ] Bikas Saha commented on YARN-289: - Could it be that 2048 request got reserved onto that node and that prevented the 1024 request from being assigned to that node? Fair scheduler doesn't satisfy smaller request when bigger one is outstanding -- Key: YARN-289 URL: https://issues.apache.org/jira/browse/YARN-289 Project: Hadoop YARN Issue Type: Bug Reporter: Sandy Ryza Assignee: Sandy Ryza An application requests a container with 1024 MB. It then requests a container with 2048 MB. A node shows up with 1024 MB available. Even if the application is the only one running, neither request will be scheduled on it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
