[jira] [Commented] (YARN-307) NodeManager should log container launch command.
[ https://issues.apache.org/jira/browse/YARN-307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13595744#comment-13595744 ] Abhishek Kapoor commented on YARN-307: -- Do we still need this jira to remain open or suggested approach by Tom works for you [~lohit] ? NodeManager should log container launch command. Key: YARN-307 URL: https://issues.apache.org/jira/browse/YARN-307 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.0.3-alpha Reporter: Lohit Vijayarenu NodeManager's DefaultContainerExecutor seems to log only path of default container executor script instead of contents of script. It would be good to log the execution command so that one could see what is being launched. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-429) capacity-scheduler config missing from yarn-test artifact
[ https://issues.apache.org/jira/browse/YARN-429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13595763#comment-13595763 ] Hudson commented on YARN-429: - Integrated in Hadoop-Yarn-trunk #148 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/148/]) YARN-429. capacity-scheduler config missing from yarn-test artifact. Contributed by Siddharth Seth. (Revision 1453501) Result = SUCCESS hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1453501 Files : * /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/capacity-scheduler.xml capacity-scheduler config missing from yarn-test artifact - Key: YARN-429 URL: https://issues.apache.org/jira/browse/YARN-429 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.0.3-alpha Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Blocker Fix For: 2.0.4-beta Attachments: YARN-429.txt MiniYARNCluster and MiniMRCluster are unusable by downstream projects with the 2.0.3-alpha release, since the capacity-scheduler configuration is missing from the test artifact. hadoop-yarn-server-tests-3.0.0-SNAPSHOT-tests.jar should include the default capacity-scheduler configuration. Also, this doesn't need to be part of the default classpath - and should be moved out of the top level directory in the dist package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-410) New lines in diagnostics for a failed app on the per-application page make it hard to read
[ https://issues.apache.org/jira/browse/YARN-410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13595826#comment-13595826 ] Hudson commented on YARN-410: - Integrated in Hadoop-Hdfs-0.23-Build #546 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/546/]) YARN-410. Fixed RM UI so that the new lines diagnostics for a failed app on the per-application page are translated to html line breaks. Contributed by Omkar Vinit Joshi (Revision 1453638) Result = UNSTABLE jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1453638 Files : * /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/view/InfoBlock.java * /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/webapp/view/TestInfoBlock.java New lines in diagnostics for a failed app on the per-application page make it hard to read -- Key: YARN-410 URL: https://issues.apache.org/jira/browse/YARN-410 Project: Hadoop YARN Issue Type: Bug Reporter: Vinod Kumar Vavilapalli Assignee: omkar vinit joshi Labels: usability Fix For: 0.23.7, 2.0.4-beta Attachments: yarn-410.patch We need to fix the following issues on YARN web-UI: - Remove the Note column from the application list. When a failure happens, this Note spoils the table layout. - When the Application is still not running, the Tracking UI should be title UNASSIGNED, for some reason it is titled ApplicationMaster but (correctly) links to #. - The per-application page has all the RM related information like version, start-time etc. Must be some accidental change by one of the patches. - The diagnostics for a failed app on the per-application page don't retain new lines and wrap'em around - looks hard to read. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-429) capacity-scheduler config missing from yarn-test artifact
[ https://issues.apache.org/jira/browse/YARN-429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13595850#comment-13595850 ] Hudson commented on YARN-429: - Integrated in Hadoop-Hdfs-trunk #1337 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1337/]) YARN-429. capacity-scheduler config missing from yarn-test artifact. Contributed by Siddharth Seth. (Revision 1453501) Result = SUCCESS hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1453501 Files : * /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/capacity-scheduler.xml capacity-scheduler config missing from yarn-test artifact - Key: YARN-429 URL: https://issues.apache.org/jira/browse/YARN-429 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.0.3-alpha Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Blocker Fix For: 2.0.4-beta Attachments: YARN-429.txt MiniYARNCluster and MiniMRCluster are unusable by downstream projects with the 2.0.3-alpha release, since the capacity-scheduler configuration is missing from the test artifact. hadoop-yarn-server-tests-3.0.0-SNAPSHOT-tests.jar should include the default capacity-scheduler configuration. Also, this doesn't need to be part of the default classpath - and should be moved out of the top level directory in the dist package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-449) MRAppMaster classpath not set properly for unit tests in downstream projects
[ https://issues.apache.org/jira/browse/YARN-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13595872#comment-13595872 ] Arun C Murthy commented on YARN-449: [~yuzhih...@gmail.com] Thanks a bunch for helping us debug this. Can you please verify that this is just a HBase issue and we can close this out our end? Thanks again! MRAppMaster classpath not set properly for unit tests in downstream projects Key: YARN-449 URL: https://issues.apache.org/jira/browse/YARN-449 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Siddharth Seth Priority: Blocker Attachments: hbase-7904-v3.txt, hbase-TestHFileOutputFormat-wip.txt, hbase-TestingUtility-wip.txt Post YARN-429, unit tests for HBase continue to fail since the classpath for the MRAppMaster is not being set correctly. Reverting YARN-129 may fix this, but I'm not sure that's the correct solution. My guess is, as Alexandro pointed out in YARN-129, maven classloader magic is messing up java.class.path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-429) capacity-scheduler config missing from yarn-test artifact
[ https://issues.apache.org/jira/browse/YARN-429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13595898#comment-13595898 ] Hudson commented on YARN-429: - Integrated in Hadoop-Mapreduce-trunk #1365 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1365/]) YARN-429. capacity-scheduler config missing from yarn-test artifact. Contributed by Siddharth Seth. (Revision 1453501) Result = SUCCESS hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1453501 Files : * /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/capacity-scheduler.xml capacity-scheduler config missing from yarn-test artifact - Key: YARN-429 URL: https://issues.apache.org/jira/browse/YARN-429 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.0.3-alpha Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Blocker Fix For: 2.0.5-beta Attachments: YARN-429.txt MiniYARNCluster and MiniMRCluster are unusable by downstream projects with the 2.0.3-alpha release, since the capacity-scheduler configuration is missing from the test artifact. hadoop-yarn-server-tests-3.0.0-SNAPSHOT-tests.jar should include the default capacity-scheduler configuration. Also, this doesn't need to be part of the default classpath - and should be moved out of the top level directory in the dist package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-449) MRAppMaster classpath not set properly for unit tests in downstream projects
[ https://issues.apache.org/jira/browse/YARN-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13595906#comment-13595906 ] Ted Yu commented on YARN-449: - Hbase can adjust as Sid suggested. What about Hive ? I didn't finish verification for Hive. MRAppMaster classpath not set properly for unit tests in downstream projects Key: YARN-449 URL: https://issues.apache.org/jira/browse/YARN-449 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Siddharth Seth Priority: Blocker Attachments: hbase-7904-v3.txt, hbase-TestHFileOutputFormat-wip.txt, hbase-TestingUtility-wip.txt Post YARN-429, unit tests for HBase continue to fail since the classpath for the MRAppMaster is not being set correctly. Reverting YARN-129 may fix this, but I'm not sure that's the correct solution. My guess is, as Alexandro pointed out in YARN-129, maven classloader magic is messing up java.class.path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-429) capacity-scheduler config missing from yarn-test artifact
[ https://issues.apache.org/jira/browse/YARN-429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated YARN-429: --- Fix Version/s: (was: 2.0.5-beta) 2.0.4-alpha capacity-scheduler config missing from yarn-test artifact - Key: YARN-429 URL: https://issues.apache.org/jira/browse/YARN-429 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.0.3-alpha Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Blocker Fix For: 2.0.4-alpha Attachments: YARN-429.txt MiniYARNCluster and MiniMRCluster are unusable by downstream projects with the 2.0.3-alpha release, since the capacity-scheduler configuration is missing from the test artifact. hadoop-yarn-server-tests-3.0.0-SNAPSHOT-tests.jar should include the default capacity-scheduler configuration. Also, this doesn't need to be part of the default classpath - and should be moved out of the top level directory in the dist package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-449) MRAppMaster classpath not set properly for unit tests in downstream projects
[ https://issues.apache.org/jira/browse/YARN-449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated YARN-449: --- Target Version/s: 2.0.4-alpha MRAppMaster classpath not set properly for unit tests in downstream projects Key: YARN-449 URL: https://issues.apache.org/jira/browse/YARN-449 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Siddharth Seth Priority: Blocker Attachments: hbase-7904-v3.txt, hbase-TestHFileOutputFormat-wip.txt, hbase-TestingUtility-wip.txt Post YARN-429, unit tests for HBase continue to fail since the classpath for the MRAppMaster is not being set correctly. Reverting YARN-129 may fix this, but I'm not sure that's the correct solution. My guess is, as Alexandro pointed out in YARN-129, maven classloader magic is messing up java.class.path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated YARN-443: --- Attachment: YARN-443-branch-2.patch YARN-443-branch-0.23.patch patches for branch-2 and branch-0.23. allow OS scheduling priority of NM to be different than the containers it launches -- Key: YARN-443 URL: https://issues.apache.org/jira/browse/YARN-443 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 2.0.3-alpha, 0.23.6 Reporter: Thomas Graves Assignee: Thomas Graves Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, YARN-443-branch-2.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch It would be nice if we could have the nodemanager run at a different OS scheduling priority than the containers so that you can still communicate with the nodemanager if the containers out of control. On linux we could launch the nodemanager at a higher priority, but then all the containers it launches would also be at that higher priority, so we need a way for the container executor to launch them at a lower priority. I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated YARN-443: --- Attachment: YARN-443.patch trunk patch. Note that I have nice -n applying to the winutils.exe call to, but unfortunately haven't tested it as I don't have a windows box. Bikas if you have cycles to test I would appreciate it, if not I can remove the windows part and file a follow on jira. allow OS scheduling priority of NM to be different than the containers it launches -- Key: YARN-443 URL: https://issues.apache.org/jira/browse/YARN-443 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 2.0.3-alpha, 0.23.6 Reporter: Thomas Graves Assignee: Thomas Graves Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, YARN-443-branch-2.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch It would be nice if we could have the nodemanager run at a different OS scheduling priority than the containers so that you can still communicate with the nodemanager if the containers out of control. On linux we could launch the nodemanager at a higher priority, but then all the containers it launches would also be at that higher priority, so we need a way for the container executor to launch them at a lower priority. I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-450) Define value for * in the scheduling protocol
[ https://issues.apache.org/jira/browse/YARN-450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13595971#comment-13595971 ] Arun C Murthy commented on YARN-450: How about defining this in YarnConfiguration? Define value for * in the scheduling protocol - Key: YARN-450 URL: https://issues.apache.org/jira/browse/YARN-450 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: YARN-450_1.patch, YARN-450_2.patch The ResourceRequest has a string field to specify node/rack locations. For the cross-rack/cluster-wide location (ie when there is no locality constraint) the * string is used everywhere. However, its not defined anywhere and each piece of code either defines a local constant or uses the string literal. Defining * in the protocol and removing other local references from the code base will be good. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[
https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13595980#comment-13595980
]
Hadoop QA commented on YARN-443:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12572548/YARN-443.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/475//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/475//console
This message is automatically generated.
allow OS scheduling priority of NM to be different than the containers it
launches
--
Key: YARN-443
URL: https://issues.apache.org/jira/browse/YARN-443
Project: Hadoop YARN
Issue Type: Improvement
Components: nodemanager
Affects Versions: 2.0.3-alpha, 0.23.6
Reporter: Thomas Graves
Assignee: Thomas Graves
Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch,
YARN-443-branch-2.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch
It would be nice if we could have the nodemanager run at a different OS
scheduling priority than the containers so that you can still communicate
with the nodemanager if the containers out of control.
On linux we could launch the nodemanager at a higher priority, but then all
the containers it launches would also be at that higher priority, so we need
a way for the container executor to launch them at a lower priority.
I'm not sure how this applies to windows if at all.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-455) NM warns about stopping an unknown container under normal circumstances
Jason Lowe created YARN-455:
---
Summary: NM warns about stopping an unknown container under normal
circumstances
Key: YARN-455
URL: https://issues.apache.org/jira/browse/YARN-455
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 0.23.6, 2.0.3-alpha
Reporter: Jason Lowe
During normal operations the NM can log warnings to its audit log about unknown
containers. For example:
{noformat}
2013-03-06 21:04:55,327 WARN nodemanager.NMAuditLogger: USER=UnknownUser
IP=xx OPERATION=Stop Container RequestTARGET=ContainerManagerImpl
RESULT=FAILURE DESCRIPTION=Trying to stop unknown container!
APPID=application_1359150825713_3947178
CONTAINERID=container_1359150825713_3947178_01_001266
{noformat}
Looking closer at the audit log and the NM log shows that the container
completed successfully and was forgotten by the NM before the stop request
arrived. The NM should avoid warning in these situations since this is a
normal race condition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated YARN-443: --- Attachment: YARN-443.patch added a couple of unit tests. Still need to update the branch-2 and branch-0.23 patches to include the tests. allow OS scheduling priority of NM to be different than the containers it launches -- Key: YARN-443 URL: https://issues.apache.org/jira/browse/YARN-443 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 2.0.3-alpha, 0.23.6 Reporter: Thomas Graves Assignee: Thomas Graves Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, YARN-443-branch-2.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch It would be nice if we could have the nodemanager run at a different OS scheduling priority than the containers so that you can still communicate with the nodemanager if the containers out of control. On linux we could launch the nodemanager at a higher priority, but then all the containers it launches would also be at that higher priority, so we need a way for the container executor to launch them at a lower priority. I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-450) Define value for * in the scheduling protocol
[ https://issues.apache.org/jira/browse/YARN-450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13596095#comment-13596095 ] Bikas Saha commented on YARN-450: - There is already way too much in configuration according to many folks. configuration also does not sound like a good place for something that is api. also see YARN-444. Define value for * in the scheduling protocol - Key: YARN-450 URL: https://issues.apache.org/jira/browse/YARN-450 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: YARN-450_1.patch, YARN-450_2.patch The ResourceRequest has a string field to specify node/rack locations. For the cross-rack/cluster-wide location (ie when there is no locality constraint) the * string is used everywhere. However, its not defined anywhere and each piece of code either defines a local constant or uses the string literal. Defining * in the protocol and removing other local references from the code base will be good. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[
https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13596107#comment-13596107
]
Hadoop QA commented on YARN-443:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12572561/YARN-443.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 one of tests included doesn't have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/476//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/476//console
This message is automatically generated.
allow OS scheduling priority of NM to be different than the containers it
launches
--
Key: YARN-443
URL: https://issues.apache.org/jira/browse/YARN-443
Project: Hadoop YARN
Issue Type: Improvement
Components: nodemanager
Affects Versions: 2.0.3-alpha, 0.23.6
Reporter: Thomas Graves
Assignee: Thomas Graves
Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch,
YARN-443-branch-2.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch,
YARN-443.patch
It would be nice if we could have the nodemanager run at a different OS
scheduling priority than the containers so that you can still communicate
with the nodemanager if the containers out of control.
On linux we could launch the nodemanager at a higher priority, but then all
the containers it launches would also be at that higher priority, so we need
a way for the container executor to launch them at a lower priority.
I'm not sure how this applies to windows if at all.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-456) Add similar support for Windows
Bikas Saha created YARN-456: --- Summary: Add similar support for Windows Key: YARN-456 URL: https://issues.apache.org/jira/browse/YARN-456 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Bikas Saha -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13596141#comment-13596141 ] Bikas Saha commented on YARN-443: - winutils will not support nice. and windows support does not use cygwin. infact, it removes cygwin. so the above patch will likely cause all tests to fail on Windows :) Why dont we set a local variable with the result of the scheduling configuration? And use that in the non-Windows part of the command line generation? Later we can use that value for the windows part when it gets supported. YARN-456. Minor nit, how about a default value in config instead of the hard coded 0 for priority? Also if the comments describing the config could be made easier for someone implementing YARN-456 or using the feature. something like. The value must be a +ve integer and higher values mean lower priority? Then the current comments could explain how the value is used in linux. I wish linuxcontainerexecutor.java did not need to replicate almost all the changes made in defaultcontainerexecutor :( allow OS scheduling priority of NM to be different than the containers it launches -- Key: YARN-443 URL: https://issues.apache.org/jira/browse/YARN-443 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 2.0.3-alpha, 0.23.6 Reporter: Thomas Graves Assignee: Thomas Graves Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, YARN-443-branch-2.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch It would be nice if we could have the nodemanager run at a different OS scheduling priority than the containers so that you can still communicate with the nodemanager if the containers out of control. On linux we could launch the nodemanager at a higher priority, but then all the containers it launches would also be at that higher priority, so we need a way for the container executor to launch them at a lower priority. I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13596169#comment-13596169 ] Thomas Graves commented on YARN-443: Thanks for the review Bikas, I'll remove the bits for modifying the windows stuff and update based on your comments. I agree, I wish it was more common but didn't want to take that on here. I can update the comment to say higher values mean less favorable priority on Linux. We could make that the standard if windows wants to handle converting that. I changed the name to be the adjustment, one because it fit as parameter to nice but also in an attempt to make it less platform specific. I was hoping it would allows us to say things like positive numbers in adjustment on any platform make the container run less favorable to NM and then each platform could interpret that. We don't have to state that now though. It doesn't necessarily have to be +ve either. If you nodemanager has root priveleges it can be negative on linux although I don't know of anyone who would do that. I also didn't want to restrict that for other platforms. allow OS scheduling priority of NM to be different than the containers it launches -- Key: YARN-443 URL: https://issues.apache.org/jira/browse/YARN-443 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 2.0.3-alpha, 0.23.6 Reporter: Thomas Graves Assignee: Thomas Graves Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, YARN-443-branch-2.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch It would be nice if we could have the nodemanager run at a different OS scheduling priority than the containers so that you can still communicate with the nodemanager if the containers out of control. On linux we could launch the nodemanager at a higher priority, but then all the containers it launches would also be at that higher priority, so we need a way for the container executor to launch them at a lower priority. I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-456) allow OS scheduling priority of NM to be different than the containers it launches for Windows
[ https://issues.apache.org/jira/browse/YARN-456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated YARN-456: - Summary: allow OS scheduling priority of NM to be different than the containers it launches for Windows (was: Add similar support for Windows) allow OS scheduling priority of NM to be different than the containers it launches for Windows -- Key: YARN-456 URL: https://issues.apache.org/jira/browse/YARN-456 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Bikas Saha Assignee: Bikas Saha -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-450) Define value for * in the scheduling protocol
[ https://issues.apache.org/jira/browse/YARN-450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13596232#comment-13596232 ] Zhijie Shen commented on YARN-450: -- Yes, the string shouldn't be a configurable item, which seems not to be suitable to be placed in YarnConfiguration. On the other side, I'm wondering whether it good to have a project-wide constant aggregation class, i.e., YarnConstants (like the existing HdfsConstants and ApplicationConstants). The constant string mentioned here and the exit codes mentioned in YARN-444 can be placed in it. Define value for * in the scheduling protocol - Key: YARN-450 URL: https://issues.apache.org/jira/browse/YARN-450 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: YARN-450_1.patch, YARN-450_2.patch The ResourceRequest has a string field to specify node/rack locations. For the cross-rack/cluster-wide location (ie when there is no locality constraint) the * string is used everywhere. However, its not defined anywhere and each piece of code either defines a local constant or uses the string literal. Defining * in the protocol and removing other local references from the code base will be good. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated YARN-443: --- Attachment: YARN-443.patch All patches updated with changes from review and all include tests. allow OS scheduling priority of NM to be different than the containers it launches -- Key: YARN-443 URL: https://issues.apache.org/jira/browse/YARN-443 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 2.0.3-alpha, 0.23.6 Reporter: Thomas Graves Assignee: Thomas Graves Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch It would be nice if we could have the nodemanager run at a different OS scheduling priority than the containers so that you can still communicate with the nodemanager if the containers out of control. On linux we could launch the nodemanager at a higher priority, but then all the containers it launches would also be at that higher priority, so we need a way for the container executor to launch them at a lower priority. I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated YARN-443: --- Attachment: YARN-443-branch-2.patch YARN-443-branch-0.23.patch allow OS scheduling priority of NM to be different than the containers it launches -- Key: YARN-443 URL: https://issues.apache.org/jira/browse/YARN-443 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 2.0.3-alpha, 0.23.6 Reporter: Thomas Graves Assignee: Thomas Graves Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch It would be nice if we could have the nodemanager run at a different OS scheduling priority than the containers so that you can still communicate with the nodemanager if the containers out of control. On linux we could launch the nodemanager at a higher priority, but then all the containers it launches would also be at that higher priority, so we need a way for the container executor to launch them at a lower priority. I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated YARN-443: --- Attachment: YARN-443.patch added timeouts to tests allow OS scheduling priority of NM to be different than the containers it launches -- Key: YARN-443 URL: https://issues.apache.org/jira/browse/YARN-443 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 2.0.3-alpha, 0.23.6 Reporter: Thomas Graves Assignee: Thomas Graves Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch It would be nice if we could have the nodemanager run at a different OS scheduling priority than the containers so that you can still communicate with the nodemanager if the containers out of control. On linux we could launch the nodemanager at a higher priority, but then all the containers it launches would also be at that higher priority, so we need a way for the container executor to launch them at a lower priority. I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[
https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13596325#comment-13596325
]
Hadoop QA commented on YARN-443:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12572597/YARN-443.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 one of tests included doesn't have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/477//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/477//console
This message is automatically generated.
allow OS scheduling priority of NM to be different than the containers it
launches
--
Key: YARN-443
URL: https://issues.apache.org/jira/browse/YARN-443
Project: Hadoop YARN
Issue Type: Improvement
Components: nodemanager
Affects Versions: 2.0.3-alpha, 0.23.6
Reporter: Thomas Graves
Assignee: Thomas Graves
Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch,
YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch,
YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch,
YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch,
YARN-443.patch, YARN-443.patch
It would be nice if we could have the nodemanager run at a different OS
scheduling priority than the containers so that you can still communicate
with the nodemanager if the containers out of control.
On linux we could launch the nodemanager at a higher priority, but then all
the containers it launches would also be at that higher priority, so we need
a way for the container executor to launch them at a lower priority.
I'm not sure how this applies to windows if at all.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13596399#comment-13596399 ] Bikas Saha commented on YARN-443: - I am not that familiar with the LCE code but it looks good overall. There seems to be some command line generation code duplication in the LCE code but I dont think its for this jira. I dont think we need an if(Shell.Windows) in the test for LCE.java, right? allow OS scheduling priority of NM to be different than the containers it launches -- Key: YARN-443 URL: https://issues.apache.org/jira/browse/YARN-443 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 2.0.3-alpha, 0.23.6 Reporter: Thomas Graves Assignee: Thomas Graves Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch It would be nice if we could have the nodemanager run at a different OS scheduling priority than the containers so that you can still communicate with the nodemanager if the containers out of control. On linux we could launch the nodemanager at a higher priority, but then all the containers it launches would also be at that higher priority, so we need a way for the container executor to launch them at a lower priority. I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-196) Nodemanager if started before starting Resource manager is getting shutdown.But if both RM and NM are started and then after if RM is going down,NM is retrying for the RM
[
https://issues.apache.org/jira/browse/YARN-196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13596421#comment-13596421
]
Hitesh Shah commented on YARN-196:
--
@Xuan, the previous comment still does not seem to have been addressed. Latest
patch has:
+ throw new YarnException(Invalid Configuration. +
+ YarnConfiguration.RESOURCEMANAGER_CONNECT_RETRY_INTERVAL_SECS +
+ should not be negative.);
Nodemanager if started before starting Resource manager is getting
shutdown.But if both RM and NM are started and then after if RM is going
down,NM is retrying for the RM.
---
Key: YARN-196
URL: https://issues.apache.org/jira/browse/YARN-196
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 3.0.0, 2.0.0-alpha
Reporter: Ramgopal N
Assignee: Xuan Gong
Attachments: MAPREDUCE-3676.patch, YARN-196.1.patch,
YARN-196.2.patch, YARN-196.3.patch, YARN-196.4.patch, YARN-196.5.patch,
YARN-196.6.patch, YARN-196.7.patch, YARN-196.8.patch
If NM is started before starting the RM ,NM is shutting down with the
following error
{code}
ERROR org.apache.hadoop.yarn.service.CompositeService: Error starting
services org.apache.hadoop.yarn.server.nodemanager.NodeManager
org.apache.avro.AvroRuntimeException:
java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:149)
at
org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.start(NodeManager.java:167)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:242)
Caused by: java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:66)
at
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:182)
at
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:145)
... 3 more
Caused by: com.google.protobuf.ServiceException: java.net.ConnectException:
Call From HOST-10-18-52-230/10.18.52.230 to HOST-10-18-52-250:8025 failed on
connection exception: java.net.ConnectException: Connection refused; For more
details see: http://wiki.apache.org/hadoop/ConnectionRefused
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:131)
at $Proxy23.registerNodeManager(Unknown Source)
at
org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:59)
... 5 more
Caused by: java.net.ConnectException: Call From
HOST-10-18-52-230/10.18.52.230 to HOST-10-18-52-250:8025 failed on connection
exception: java.net.ConnectException: Connection refused; For more details
see: http://wiki.apache.org/hadoop/ConnectionRefused
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:857)
at org.apache.hadoop.ipc.Client.call(Client.java:1141)
at org.apache.hadoop.ipc.Client.call(Client.java:1100)
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:128)
... 7 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:659)
at
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:469)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:563)
at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:211)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1247)
at org.apache.hadoop.ipc.Client.call(Client.java:1117)
... 9 more
2012-01-16 15:04:13,336 WARN org.apache.hadoop.yarn.event.AsyncDispatcher:
AsyncDispatcher thread interrupted
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1934)
at
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13596430#comment-13596430 ] Thomas Graves commented on YARN-443: thanks Bikas, I'll remove that if and upload a new trunk patch shortly. allow OS scheduling priority of NM to be different than the containers it launches -- Key: YARN-443 URL: https://issues.apache.org/jira/browse/YARN-443 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 2.0.3-alpha, 0.23.6 Reporter: Thomas Graves Assignee: Thomas Graves Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch It would be nice if we could have the nodemanager run at a different OS scheduling priority than the containers so that you can still communicate with the nodemanager if the containers out of control. On linux we could launch the nodemanager at a higher priority, but then all the containers it launches would also be at that higher priority, so we need a way for the container executor to launch them at a lower priority. I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[ https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated YARN-443: --- Attachment: YARN-443.patch allow OS scheduling priority of NM to be different than the containers it launches -- Key: YARN-443 URL: https://issues.apache.org/jira/browse/YARN-443 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 2.0.3-alpha, 0.23.6 Reporter: Thomas Graves Assignee: Thomas Graves Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch It would be nice if we could have the nodemanager run at a different OS scheduling priority than the containers so that you can still communicate with the nodemanager if the containers out of control. On linux we could launch the nodemanager at a higher priority, but then all the containers it launches would also be at that higher priority, so we need a way for the container executor to launch them at a lower priority. I'm not sure how this applies to windows if at all. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-449) MRAppMaster classpath not set properly for unit tests in downstream projects
[
https://issues.apache.org/jira/browse/YARN-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13596441#comment-13596441
]
Ted Yu commented on YARN-449:
-
Assuming the JobConf returned by MiniMRCluster contains the conf parameters we
passed to MiniMRCluster at time of start, I don't know if
getMinimalConfiguration() is needed.
In HBase we use shim to hide the difference between hadoop 1.0 and 2.0:
{code}
JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
if (jobConf == null) {
jobConf = mrCluster.createJobConf();
}
{code}
MRAppMaster classpath not set properly for unit tests in downstream projects
Key: YARN-449
URL: https://issues.apache.org/jira/browse/YARN-449
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Siddharth Seth
Priority: Blocker
Attachments: hbase-7904-v3.txt, hbase-TestHFileOutputFormat-wip.txt,
hbase-TestingUtility-wip.txt
Post YARN-429, unit tests for HBase continue to fail since the classpath for
the MRAppMaster is not being set correctly.
Reverting YARN-129 may fix this, but I'm not sure that's the correct
solution. My guess is, as Alexandro pointed out in YARN-129, maven
classloader magic is messing up java.class.path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[
https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13596471#comment-13596471
]
Hadoop QA commented on YARN-443:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12572619/YARN-443.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/479//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/479//console
This message is automatically generated.
allow OS scheduling priority of NM to be different than the containers it
launches
--
Key: YARN-443
URL: https://issues.apache.org/jira/browse/YARN-443
Project: Hadoop YARN
Issue Type: Improvement
Components: nodemanager
Affects Versions: 2.0.3-alpha, 0.23.6
Reporter: Thomas Graves
Assignee: Thomas Graves
Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch,
YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch,
YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch,
YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch,
YARN-443.patch, YARN-443.patch, YARN-443.patch
It would be nice if we could have the nodemanager run at a different OS
scheduling priority than the containers so that you can still communicate
with the nodemanager if the containers out of control.
On linux we could launch the nodemanager at a higher priority, but then all
the containers it launches would also be at that higher priority, so we need
a way for the container executor to launch them at a lower priority.
I'm not sure how this applies to windows if at all.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-443) allow OS scheduling priority of NM to be different than the containers it launches
[
https://issues.apache.org/jira/browse/YARN-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13596514#comment-13596514
]
Bikas Saha commented on YARN-443:
-
+1 looks good to me subject to above LCE caveat :P
Minor nits
Did you intend to keep this log message?
{code}
+ logOutput(e.getMessage());
{code}
You mean than
{code}
+ * On Linux, higher values mean run the containers at a less
+ * favorable priority then the NM.
{code}
allow OS scheduling priority of NM to be different than the containers it
launches
--
Key: YARN-443
URL: https://issues.apache.org/jira/browse/YARN-443
Project: Hadoop YARN
Issue Type: Improvement
Components: nodemanager
Affects Versions: 2.0.3-alpha, 0.23.6
Reporter: Thomas Graves
Assignee: Thomas Graves
Attachments: YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch,
YARN-443-branch-0.23.patch, YARN-443-branch-0.23.patch,
YARN-443-branch-2.patch, YARN-443-branch-2.patch, YARN-443-branch-2.patch,
YARN-443.patch, YARN-443.patch, YARN-443.patch, YARN-443.patch,
YARN-443.patch, YARN-443.patch, YARN-443.patch
It would be nice if we could have the nodemanager run at a different OS
scheduling priority than the containers so that you can still communicate
with the nodemanager if the containers out of control.
On linux we could launch the nodemanager at a higher priority, but then all
the containers it launches would also be at that higher priority, so we need
a way for the container executor to launch them at a lower priority.
I'm not sure how this applies to windows if at all.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-449) MRAppMaster classpath not set properly for unit tests in downstream projects
[ https://issues.apache.org/jira/browse/YARN-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13596658#comment-13596658 ] Ted Yu commented on YARN-449: - There was test failure using patch v3 against hadoop 1.0 See: https://builds.apache.org/job/PreCommit-HBASE-Build/4719/artifact/trunk/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.txt MRAppMaster classpath not set properly for unit tests in downstream projects Key: YARN-449 URL: https://issues.apache.org/jira/browse/YARN-449 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Siddharth Seth Priority: Blocker Attachments: hbase-7904-v3.txt, hbase-TestHFileOutputFormat-wip.txt, hbase-TestingUtility-wip.txt Post YARN-429, unit tests for HBase continue to fail since the classpath for the MRAppMaster is not being set correctly. Reverting YARN-129 may fix this, but I'm not sure that's the correct solution. My guess is, as Alexandro pointed out in YARN-129, maven classloader magic is messing up java.class.path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-457) Setting updated nodes in AMResponsePBImpl
Sandy Ryza created YARN-457:
---
Summary: Setting updated nodes in AMResponsePBImpl
Key: YARN-457
URL: https://issues.apache.org/jira/browse/YARN-457
Project: Hadoop YARN
Issue Type: Bug
Components: api
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
It looks like this is supposed to be a !=:
{code}
if (updatedNodes == null) {
this.updatedNodes.clear();
return;
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-457) Setting updated nodes from null to null causes NPE in AMResponsePBImpl
[
https://issues.apache.org/jira/browse/YARN-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sandy Ryza updated YARN-457:
Description:
{code}
if (updatedNodes == null) {
this.updatedNodes.clear();
return;
}
{code}
If updatedNodes is already null, a NullPointerException is thrown.
was:
It looks like this is supposed to be a !=:
{code}
if (updatedNodes == null) {
this.updatedNodes.clear();
return;
}
{code}
Setting updated nodes from null to null causes NPE in AMResponsePBImpl
--
Key: YARN-457
URL: https://issues.apache.org/jira/browse/YARN-457
Project: Hadoop YARN
Issue Type: Bug
Components: api
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
{code}
if (updatedNodes == null) {
this.updatedNodes.clear();
return;
}
{code}
If updatedNodes is already null, a NullPointerException is thrown.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-457) Setting updated nodes from null to null causes NPE in AMResponsePBImpl
[
https://issues.apache.org/jira/browse/YARN-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sandy Ryza updated YARN-457:
Summary: Setting updated nodes from null to null causes NPE in
AMResponsePBImpl (was: Setting updated nodes in AMResponsePBImpl)
Setting updated nodes from null to null causes NPE in AMResponsePBImpl
--
Key: YARN-457
URL: https://issues.apache.org/jira/browse/YARN-457
Project: Hadoop YARN
Issue Type: Bug
Components: api
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
It looks like this is supposed to be a !=:
{code}
if (updatedNodes == null) {
this.updatedNodes.clear();
return;
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-454) FS could wait until next NODE_UPDATE event to schedule a reserved container
[ https://issues.apache.org/jira/browse/YARN-454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13596686#comment-13596686 ] Sandy Ryza commented on YARN-454: - This is the correct behavior. A reservation in the node.getReservedContainer() sense means that the the node is full, and the reserved container is the next one that gets to run on it when it frees up. FS could wait until next NODE_UPDATE event to schedule a reserved container --- Key: YARN-454 URL: https://issues.apache.org/jira/browse/YARN-454 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.0.3-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla FS#nodeUpdate() allocates reserved containers first. However, it seems (from code observation): if an app reserves a container on a node while FS is scheduling a task on that node from the non-reserved pool, the request is skipped in that NODE_UPDATE event. It is addressed on the next event. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-453) Optimize FS internal data structures
[ https://issues.apache.org/jira/browse/YARN-453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13596688#comment-13596688 ] Sandy Ryza commented on YARN-453: - +1 to the idea Optimize FS internal data structures Key: YARN-453 URL: https://issues.apache.org/jira/browse/YARN-453 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.0.3-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla FS uses lists to store internal queues and sorts them on every heartbeat leading to unnecessary scheduling overhead. Choice of better data structures should reduce the latency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-458) Resource manager address must be placed in four different configs
Sandy Ryza created YARN-458: --- Summary: Resource manager address must be placed in four different configs Key: YARN-458 URL: https://issues.apache.org/jira/browse/YARN-458 Project: Hadoop YARN Issue Type: Bug Reporter: Sandy Ryza Assignee: Sandy Ryza The YARN resourcemanager's address is included in four different configs: yarn.resourcemanager.scheduler.address, yarn.resourcemanager.resource-tracker.address, yarn.resourcemanager.address, and yarn.resourcemanager.admin.address A new user trying to configure a cluster needs to know the names of all these four configs. It would be much easier if they could simply specify yarn.resourcemanager.address and default ports for the other ones would kick in. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-71) Ensure/confirm that the NodeManager cleans up local-dirs on restart
[
https://issues.apache.org/jira/browse/YARN-71?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13596875#comment-13596875
]
Hadoop QA commented on YARN-71:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12572708/YARN.71.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/480//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/480//console
This message is automatically generated.
Ensure/confirm that the NodeManager cleans up local-dirs on restart
---
Key: YARN-71
URL: https://issues.apache.org/jira/browse/YARN-71
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Reporter: Vinod Kumar Vavilapalli
Assignee: Xuan Gong
Priority: Critical
Attachments: YARN-71.1.patch, YARN-71.2.patch, YARN-71.3.patch,
YARN.71.4.patch
We have to make sure that NodeManagers cleanup their local files on restart.
It may already be working like that in which case we should have tests
validating this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
ContainerManager.StopContainer() does not work properly on distributed hadoop cluster environment
Hi sir, I am writing an AppMaster application which is capable of adding/removing container in runtime in Hadoop 2.0.3-alpha cluster. In single node mode Hadoop environment, containers can be started or stopped properly. However, when I tried to a few stop containers which are launched on different machines in distributed mode setup, I got the following problem. Initial setup: Machine1) Container 0: AppMaster Container 1: Application Container Container 2: Application Container Container 3: Application Container Machine2) Container 4: Application Container Container 5: Application Container Container 6: Application Container Machine3) Container 7: Application Container Container 8: Application Container Container 9: Application Container Stop container sequence: 1) Stop Container 4 on machine 2. -- It's OK 2) Stop Container 5 on machine 2. -- It's OK 3) Stop Container 7 on machine 3. -- It does not work and cannot see any message regarding the Container 7 in resource manager log. Afterwards, I cannot stop any other containers at all. Regards, Benson ~ This message (including any attachments) is for the named addressee(s)'s use only. It may contain sensitive, confidential, private proprietary or legally privileged information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, please immediately delete it and all copies of it from your system, destroy any hard copies of it and notify the sender. Any use, disclosure, copying, or distribution of this message and/or any attachments is strictly prohibited.
