[jira] [Updated] (YARN-140) Add capacity-scheduler-default.xml to provide a default set of configurations for the capacity scheduler.
[
https://issues.apache.org/jira/browse/YARN-140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Radwan updated YARN-140:
--
Attachment: YARN-140_rev4.patch
The cause of these failures seems to be a couple of missing default
configurations that were previously loaded from capacity-scheduler.xml. I have
added such defaults to CapacitySchedulerConfiguration.java. I have ran all
these failing tests locally, and all of them succeed except two
(TestApplicationCleanup TestGetGroups) which fail with and without the patch,
so they seem to be unrelated to this patch.
The two additional defaults that I needed to add to fix these tests are:
1- To return the queue root.default as a child for root if no children are
defined for root.
2- To return a capacity of 100.0f for the queue root.default if it exists but
no capacity is specified for it. (I was unsure about this default considering
the case when root has other children specified, but I think it should be fine
since both undefined capacity (i.e., -1) or maximum capacity (i.e., 100) will
represent an invalid configuration.
I am attaching the updated patch.
Add capacity-scheduler-default.xml to provide a default set of configurations
for the capacity scheduler.
-
Key: YARN-140
URL: https://issues.apache.org/jira/browse/YARN-140
Project: Hadoop YARN
Issue Type: Bug
Components: capacityscheduler
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
Attachments: YARN-140.patch, YARN-140_rev2.patch,
YARN-140_rev3.patch, YARN-140_rev4.patch
When setting up the capacity scheduler users are faced with problems like:
{code}
FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error
starting ResourceManager
java.lang.IllegalArgumentException: Illegal capacity of -1 for queue root
{code}
Which basically arises from missing basic configurations, which in many
cases, there is no need to explicitly provide, and a default configuration
will be sufficient. For example, to address the error above, the user need to
add a capacity of 100 to the root queue.
So, we need to add a capacity-scheduler-default.xml, this will be helpful to
provide the basic set of default configurations required to run the capacity
scheduler. The user can still override existing configurations or provide new
ones in capacity-scheduler.xml. This is similar to *-default.xml vs
*-site.xml for yarn, core, mapred, hdfs, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-140) Add capacity-scheduler-default.xml to provide a default set of configurations for the capacity scheduler.
[
https://issues.apache.org/jira/browse/YARN-140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13470281#comment-13470281
]
Arun C Murthy commented on YARN-140:
Ahmed, thanks, this is close - some feedback:
bq. To return the queue root.default as a child for root if no children are
defined for root.
I wouldn't do this. If someone has not defined root.child.queues it is a
serious mis-configuration - let's error out (as it happens today) and not try
to 'guess'.
bq. To return a capacity of 100.0f for the queue root.default if it exists
but no capacity is specified for it.
Again, for same reason as above, this represents a serious mis-configuration
and I wouldn't try to 'guess' capacities - error out is the right answer.
One more nit: Rather than delete/add capacity-scheduler.xml let's just do an
'svn mv' to preserve history.
Furthermore, please add a test-case to TestCapacitySchedulerConf to test for
defaults.
It seems to me that the additional defaults are necessary only for unit tests -
in which case the correct fix is to make a copy of capacity-scheduler.xml into
a test/resources folder and get it on the classpath for our tests.
Add capacity-scheduler-default.xml to provide a default set of configurations
for the capacity scheduler.
-
Key: YARN-140
URL: https://issues.apache.org/jira/browse/YARN-140
Project: Hadoop YARN
Issue Type: Bug
Components: capacityscheduler
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
Attachments: YARN-140.patch, YARN-140_rev2.patch,
YARN-140_rev3.patch, YARN-140_rev4.patch
When setting up the capacity scheduler users are faced with problems like:
{code}
FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error
starting ResourceManager
java.lang.IllegalArgumentException: Illegal capacity of -1 for queue root
{code}
Which basically arises from missing basic configurations, which in many
cases, there is no need to explicitly provide, and a default configuration
will be sufficient. For example, to address the error above, the user need to
add a capacity of 100 to the root queue.
So, we need to add a capacity-scheduler-default.xml, this will be helpful to
provide the basic set of default configurations required to run the capacity
scheduler. The user can still override existing configurations or provide new
ones in capacity-scheduler.xml. This is similar to *-default.xml vs
*-site.xml for yarn, core, mapred, hdfs, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-23) FairScheduler: FSQueueSchedulable#updateDemand() - potential redundant aggregation
[ https://issues.apache.org/jira/browse/YARN-23?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13470284#comment-13470284 ] Hudson commented on YARN-23: Integrated in Hadoop-Hdfs-trunk #1186 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1186/]) YARN-23. FairScheduler: FSQueueSchedulable#updateDemand() - potential redundant aggregation. (kkambatl via tucu) (Revision 1394321) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1394321 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueSchedulable.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSQueueSchedulable.java FairScheduler: FSQueueSchedulable#updateDemand() - potential redundant aggregation -- Key: YARN-23 URL: https://issues.apache.org/jira/browse/YARN-23 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.0.0-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Fix For: 2.0.3-alpha Attachments: trunk-MR-4518.patch, trunk-MR-4518.patch, YARN-23.patch, YARN-23.patch In FS, FSQueueSchedulable#updateDemand() limits the demand to maxTasks only after iterating though all the pools and computing the final demand. By checking if the demand has reached maxTasks in every iteration, we can avoid redundant work, at the expense of one condition check every iteration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-23) FairScheduler: FSQueueSchedulable#updateDemand() - potential redundant aggregation
[ https://issues.apache.org/jira/browse/YARN-23?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13470339#comment-13470339 ] Hudson commented on YARN-23: Integrated in Hadoop-Mapreduce-trunk #1217 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1217/]) YARN-23. FairScheduler: FSQueueSchedulable#updateDemand() - potential redundant aggregation. (kkambatl via tucu) (Revision 1394321) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1394321 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueSchedulable.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSQueueSchedulable.java FairScheduler: FSQueueSchedulable#updateDemand() - potential redundant aggregation -- Key: YARN-23 URL: https://issues.apache.org/jira/browse/YARN-23 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.0.0-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Fix For: 2.0.3-alpha Attachments: trunk-MR-4518.patch, trunk-MR-4518.patch, YARN-23.patch, YARN-23.patch In FS, FSQueueSchedulable#updateDemand() limits the demand to maxTasks only after iterating though all the pools and computing the final demand. By checking if the demand has reached maxTasks in every iteration, we can avoid redundant work, at the expense of one condition check every iteration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-144) MiniMRYarnCluster launches RM and JHS on default ports
Robert Parker created YARN-144: -- Summary: MiniMRYarnCluster launches RM and JHS on default ports Key: YARN-144 URL: https://issues.apache.org/jira/browse/YARN-144 Project: Hadoop YARN Issue Type: Bug Affects Versions: 0.23.4 Reporter: Robert Parker Assignee: Robert Parker Fix For: 0.23.5 MAPREDUCE-3867, MAPREDUCE-3869, and MAPREDUCE-4406 need to be combined and applied to branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-144) MiniMRYarnCluster launches RM and JHS on default ports
[ https://issues.apache.org/jira/browse/YARN-144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Parker updated YARN-144: --- Attachment: YARN-144-branch_0.23.patch MiniMRYarnCluster launches RM and JHS on default ports -- Key: YARN-144 URL: https://issues.apache.org/jira/browse/YARN-144 Project: Hadoop YARN Issue Type: Bug Affects Versions: 0.23.4 Reporter: Robert Parker Assignee: Robert Parker Fix For: 0.23.5 Attachments: YARN-144-branch_0.23.patch MAPREDUCE-3867, MAPREDUCE-3869, and MAPREDUCE-4406 need to be combined and applied to branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-140) Add capacity-scheduler-default.xml to provide a default set of configurations for the capacity scheduler.
[
https://issues.apache.org/jira/browse/YARN-140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13470601#comment-13470601
]
Ahmed Radwan commented on YARN-140:
---
Thanks Arun!
bq. It seems to me that the additional defaults are necessary only for unit
tests - in which case the correct fix is to make a copy of
capacity-scheduler.xml into a test/resources folder and get it on the classpath
for our tests.
Yes, those additional ones are only for tests. I'll move the xml to a
test/resources and have it in the classpath for tests per you recommendation.
bq. Furthermore, please add a test-case to TestCapacitySchedulerConf to test
for defaults.
The only remaining newly added default is the 100% capacity for root queue. I
already modified the TestCapapcityScheduler to verify that, and I'll check if
any other tests are needed.
I'll upload an updated version of the patch.
Add capacity-scheduler-default.xml to provide a default set of configurations
for the capacity scheduler.
-
Key: YARN-140
URL: https://issues.apache.org/jira/browse/YARN-140
Project: Hadoop YARN
Issue Type: Bug
Components: capacityscheduler
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
Attachments: YARN-140.patch, YARN-140_rev2.patch,
YARN-140_rev3.patch, YARN-140_rev4.patch
When setting up the capacity scheduler users are faced with problems like:
{code}
FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error
starting ResourceManager
java.lang.IllegalArgumentException: Illegal capacity of -1 for queue root
{code}
Which basically arises from missing basic configurations, which in many
cases, there is no need to explicitly provide, and a default configuration
will be sufficient. For example, to address the error above, the user need to
add a capacity of 100 to the root queue.
So, we need to add a capacity-scheduler-default.xml, this will be helpful to
provide the basic set of default configurations required to run the capacity
scheduler. The user can still override existing configurations or provide new
ones in capacity-scheduler.xml. This is similar to *-default.xml vs
*-site.xml for yarn, core, mapred, hdfs, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-33) LocalDirsHandler should validate the configured local and log dirs
[ https://issues.apache.org/jira/browse/YARN-33?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated YARN-33: -- Attachment: YARN-33-trunk-v6.patch Thanks Sid for the review. Updated the patch with your comments and fixed the broken testcase. Thanks, Mayank LocalDirsHandler should validate the configured local and log dirs -- Key: YARN-33 URL: https://issues.apache.org/jira/browse/YARN-33 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.0.2-alpha Reporter: Mayank Bansal Assignee: Mayank Bansal Attachments: YARN-33-trunk-v1.patch, YARN-33-trunk-v2.patch, YARN-33-trunk-v3.patch, YARN-33-trunk-v4.patch, YARN-33-trunk-v5.patch, YARN-33-trunk-v6.patch WHen yarn.nodemanager.log-dirs is with file:// URI then startup of node manager creates the directory like file:// under CWD. WHich should not be there. Thanks, Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-127) Move RMAdmin tool to the client package
[
https://issues.apache.org/jira/browse/YARN-127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13470804#comment-13470804
]
Hadoop QA commented on YARN-127:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12548073/YARN-127-20121005.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.client.TestYarnClient
org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-YARN-Build/71//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/71//console
This message is automatically generated.
Move RMAdmin tool to the client package
---
Key: YARN-127
URL: https://issues.apache.org/jira/browse/YARN-127
Project: Hadoop YARN
Issue Type: Bug
Reporter: Vinod Kumar Vavilapalli
Assignee: Vinod Kumar Vavilapalli
Attachments: YARN-127-20120924.txt, YARN-127-20121005.txt
It belongs to the client package and not the RM clearly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-147) Add support for CPU isolation/monitoring of containers
[
https://issues.apache.org/jira/browse/YARN-147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13470908#comment-13470908
]
Hadoop QA commented on YARN-147:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12548069/YARN-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-YARN-Build/73//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/73//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/73//console
This message is automatically generated.
Add support for CPU isolation/monitoring of containers
--
Key: YARN-147
URL: https://issues.apache.org/jira/browse/YARN-147
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 2.0.3-alpha
Reporter: Alejandro Abdelnur
Assignee: Andrew Ferguson
Fix For: 2.0.3-alpha
Attachments: YARN-3.patch
This is a clone for YARN-3 to be able to submit the patch as YARN-3 does not
show the SUBMIT PATCH button.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-140) Add capacity-scheduler-default.xml to provide a default set of configurations for the capacity scheduler.
[
https://issues.apache.org/jira/browse/YARN-140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13470909#comment-13470909
]
Hadoop QA commented on YARN-140:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12548080/YARN-140_rev5_onlyForJenkins.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-YARN-Build/72//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/72//console
This message is automatically generated.
Add capacity-scheduler-default.xml to provide a default set of configurations
for the capacity scheduler.
-
Key: YARN-140
URL: https://issues.apache.org/jira/browse/YARN-140
Project: Hadoop YARN
Issue Type: Bug
Components: capacityscheduler
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
Attachments: YARN-140.patch, YARN-140_rev2.patch,
YARN-140_rev3.patch, YARN-140_rev4.patch, YARN-140_rev5_onlyForJenkins.patch,
YARN-140_rev5.patch, YARN-140_rev5_svn_mv.patch
When setting up the capacity scheduler users are faced with problems like:
{code}
FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error
starting ResourceManager
java.lang.IllegalArgumentException: Illegal capacity of -1 for queue root
{code}
Which basically arises from missing basic configurations, which in many
cases, there is no need to explicitly provide, and a default configuration
will be sufficient. For example, to address the error above, the user need to
add a capacity of 100 to the root queue.
So, we need to add a capacity-scheduler-default.xml, this will be helpful to
provide the basic set of default configurations required to run the capacity
scheduler. The user can still override existing configurations or provide new
ones in capacity-scheduler.xml. This is similar to *-default.xml vs
*-site.xml for yarn, core, mapred, hdfs, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
