[jira] [Commented] (YARN-109) .tmp file is not deleted for localized archives
[ https://issues.apache.org/jira/browse/YARN-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13608705#comment-13608705 ] Mayank Bansal commented on YARN-109: [~ojoshi] There are no changes in call its just I formatted that with hadoop formatter. I modified the unpack for all the cases. I added couple of more test methods to test some more cases , Looks to me enough for this change. Not necessary to test all combinations. I add the timeouts. Thanks, Mayank .tmp file is not deleted for localized archives --- Key: YARN-109 URL: https://issues.apache.org/jira/browse/YARN-109 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 0.23.3, 2.0.0-alpha Reporter: Jason Lowe Assignee: Mayank Bansal Attachments: YARN-109-trunk-1.patch, YARN-109-trunk-2.patch, YARN-109-trunk.patch When archives are localized they are initially created as a .tmp file and unpacked from that file. However the .tmp file is not deleted afterwards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-109) .tmp file is not deleted for localized archives
[ https://issues.apache.org/jira/browse/YARN-109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated YARN-109: --- Attachment: YARN-109-trunk-3.patch .tmp file is not deleted for localized archives --- Key: YARN-109 URL: https://issues.apache.org/jira/browse/YARN-109 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 0.23.3, 2.0.0-alpha Reporter: Jason Lowe Assignee: Mayank Bansal Attachments: YARN-109-trunk-1.patch, YARN-109-trunk-2.patch, YARN-109-trunk-3.patch, YARN-109-trunk.patch When archives are localized they are initially created as a .tmp file and unpacked from that file. However the .tmp file is not deleted afterwards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-109) .tmp file is not deleted for localized archives
[
https://issues.apache.org/jira/browse/YARN-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13608713#comment-13608713
]
Hadoop QA commented on YARN-109:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574743/YARN-109-trunk-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 one of tests included doesn't have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/555//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/555//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/555//console
This message is automatically generated.
.tmp file is not deleted for localized archives
---
Key: YARN-109
URL: https://issues.apache.org/jira/browse/YARN-109
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 0.23.3, 2.0.0-alpha
Reporter: Jason Lowe
Assignee: Mayank Bansal
Attachments: YARN-109-trunk-1.patch, YARN-109-trunk-2.patch,
YARN-109-trunk-3.patch, YARN-109-trunk.patch
When archives are localized they are initially created as a .tmp file and
unpacked from that file. However the .tmp file is not deleted afterwards.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-297) Improve hashCode implementations for PB records
[ https://issues.apache.org/jira/browse/YARN-297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13608829#comment-13608829 ] Hudson commented on YARN-297: - Integrated in Hadoop-Yarn-trunk #162 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/162/]) YARN-297. Improve hashCode implementations for PB records. Contributed by Xuan Gong. (Revision 1459054) Result = SUCCESS hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1459054 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationAttemptId.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationId.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerId.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeId.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Priority.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java Improve hashCode implementations for PB records --- Key: YARN-297 URL: https://issues.apache.org/jira/browse/YARN-297 Project: Hadoop YARN Issue Type: Improvement Reporter: Arun C Murthy Assignee: Xuan Gong Fix For: 2.0.5-beta Attachments: YARN.297.1.patch, YARN-297.2.patch As [~hsn] pointed out in YARN-2, we use very small primes in all our hashCode implementations. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-396) Rationalize AllocateResponse in RM scheduler API
[ https://issues.apache.org/jira/browse/YARN-396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13608891#comment-13608891 ] Hudson commented on YARN-396: - Integrated in Hadoop-Hdfs-trunk #1351 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1351/]) YARN-396. Rationalize AllocateResponse in RM Scheduler API. Contributed by Zhijie Shen. (Revision 1459040) Result = FAILURE hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1459040 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetAllApplicationsResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetClusterNodesResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetQueueUserAclsInfoResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/AMResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/AMResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRecordFactory.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockAM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRMRPCNodeUpdates.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRMRPCResponseId.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestApplicationTokens.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestContainerManagerSecurity.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm
[jira] [Commented] (YARN-297) Improve hashCode implementations for PB records
[ https://issues.apache.org/jira/browse/YARN-297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13608892#comment-13608892 ] Hudson commented on YARN-297: - Integrated in Hadoop-Hdfs-trunk #1351 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1351/]) YARN-297. Improve hashCode implementations for PB records. Contributed by Xuan Gong. (Revision 1459054) Result = FAILURE hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1459054 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationAttemptId.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationId.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerId.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeId.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Priority.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java Improve hashCode implementations for PB records --- Key: YARN-297 URL: https://issues.apache.org/jira/browse/YARN-297 Project: Hadoop YARN Issue Type: Improvement Reporter: Arun C Murthy Assignee: Xuan Gong Fix For: 2.0.5-beta Attachments: YARN.297.1.patch, YARN-297.2.patch As [~hsn] pointed out in YARN-2, we use very small primes in all our hashCode implementations. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-109) .tmp file is not deleted for localized archives
[ https://issues.apache.org/jira/browse/YARN-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13608919#comment-13608919 ] Robert Joseph Evans commented on YARN-109: -- The findbugs is complaining that you are ignoring the return value of the delete call. It should not be a problem so either use the return value to log a warning when it fails or update the findbugs filter to filter out the error. The -1 for the test timeouts is caused by a bug in the script used to detect these, so you can either ignore it, or add a timeout to any @Test that appears in the patch file, including the ones you didn't add :(. In the test, please uncomment the lines to cleanup after the test. Are they causing a problem for the test to pass? or was it just for debugging? Also I personally would prefer to have a few small jar/tar/zip files checked into the repository instead of generating them on they fly for the test. It will speed up the test and have less dependencies on the system being set up with the exact commands, i.e. bash for windows support. Although if you don't feel like changing it I am fine with that too, most of those commands are used by FSDownload already so it is not that critical. .tmp file is not deleted for localized archives --- Key: YARN-109 URL: https://issues.apache.org/jira/browse/YARN-109 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 0.23.3, 2.0.0-alpha Reporter: Jason Lowe Assignee: Mayank Bansal Attachments: YARN-109-trunk-1.patch, YARN-109-trunk-2.patch, YARN-109-trunk-3.patch, YARN-109-trunk.patch When archives are localized they are initially created as a .tmp file and unpacked from that file. However the .tmp file is not deleted afterwards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-297) Improve hashCode implementations for PB records
[ https://issues.apache.org/jira/browse/YARN-297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13608951#comment-13608951 ] Hudson commented on YARN-297: - Integrated in Hadoop-Mapreduce-trunk #1379 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1379/]) YARN-297. Improve hashCode implementations for PB records. Contributed by Xuan Gong. (Revision 1459054) Result = SUCCESS hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1459054 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationAttemptId.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationId.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerId.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeId.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Priority.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java Improve hashCode implementations for PB records --- Key: YARN-297 URL: https://issues.apache.org/jira/browse/YARN-297 Project: Hadoop YARN Issue Type: Improvement Reporter: Arun C Murthy Assignee: Xuan Gong Fix For: 2.0.5-beta Attachments: YARN.297.1.patch, YARN-297.2.patch As [~hsn] pointed out in YARN-2, we use very small primes in all our hashCode implementations. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-479) NM retry behavior for connection to RM should be similar for lost heartbeats
[ https://issues.apache.org/jira/browse/YARN-479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609158#comment-13609158 ] jian he commented on YARN-479: -- 1. while its looping ,nodeStatus may,though unlikely,change, so I put it in the loop. 2. I'm not testing with testNMRegistration. I modified testNMRegistration because otherwise it fail, since by default RESOURCEMANAGER_CONNECT_WAIT_SECS is too long for this test NM retry behavior for connection to RM should be similar for lost heartbeats Key: YARN-479 URL: https://issues.apache.org/jira/browse/YARN-479 Project: Hadoop YARN Issue Type: Bug Reporter: Hitesh Shah Assignee: jian he Attachments: YARN-479.1.patch, YARN-479.2.patch Regardless of connection loss at the start or at an intermediate point, NM's retry behavior to the RM should follow the same flow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-474) CapacityScheduler does not activate applications when configuration is refreshed
[
https://issues.apache.org/jira/browse/YARN-474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609184#comment-13609184
]
Hadoop QA commented on YARN-474:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574695/YARN-474.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/556//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/556//console
This message is automatically generated.
CapacityScheduler does not activate applications when configuration is
refreshed
Key: YARN-474
URL: https://issues.apache.org/jira/browse/YARN-474
Project: Hadoop YARN
Issue Type: Bug
Components: capacityscheduler
Affects Versions: 2.0.3-alpha, 0.23.6
Reporter: Hitesh Shah
Assignee: Zhijie Shen
Attachments: YARN-474.1.patch
Submit 3 applications to a cluster where capacity scheduler limits allow only
1 running application. Modify capacity scheduler config to increase value of
yarn.scheduler.capacity.maximum-am-resource-percent and invoke refresh
queues.
The 2 applications not yet in running state do not get launched even though
limits are increased.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-494) RM should be able to hard stop a lingering app on a NM
Daryn Sharp created YARN-494: Summary: RM should be able to hard stop a lingering app on a NM Key: YARN-494 URL: https://issues.apache.org/jira/browse/YARN-494 Project: Hadoop YARN Issue Type: Bug Components: nodemanager, resourcemanager Affects Versions: 2.0.0-alpha, 3.0.0, 0.23.3 Reporter: Daryn Sharp It's possible for a NM to leak applications that the RM believes have finished. This currently tends to happen when a lingering app jams in log aggregation or misses the notification to begin aggregation. Until aggregation completes, the NMs send app keepalive requests to the RM so it continues renewing the app's tokens. This could be extend to allow the RM to send a hard stop to a NM for an app that has been running for a configurable interval of time after the app has finished. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-495) Containers are not cleaned up when a reboot is sent from RM
jian he created YARN-495: Summary: Containers are not cleaned up when a reboot is sent from RM Key: YARN-495 URL: https://issues.apache.org/jira/browse/YARN-495 Project: Hadoop YARN Issue Type: Bug Reporter: jian he Assignee: jian he When a reboot command is sent from RM, the node manager doesn't clean up the containers while its stopping. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-495) Containers are not cleaned up when a reboot is sent from RM
[ https://issues.apache.org/jira/browse/YARN-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609221#comment-13609221 ] jian he commented on YARN-495: -- When NodeStatusUpdaterImpl gets a REBOOT ,it enqueue a REBOOT event to the NM dispatcher's queue. While this event is getting processed ,it calls stop() and then cleanupContainers(), within cleanupContainers(), it puts the CMgrCompletedContainersEvent onto the queue. This event will not get processed until the stop() finishes, because they are processed by the same dispatcher's thread Containers are not cleaned up when a reboot is sent from RM --- Key: YARN-495 URL: https://issues.apache.org/jira/browse/YARN-495 Project: Hadoop YARN Issue Type: Bug Reporter: jian he Assignee: jian he When a reboot command is sent from RM, the node manager doesn't clean up the containers while its stopping. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-378) ApplicationMaster retry times should be set by Client
[
https://issues.apache.org/jira/browse/YARN-378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609232#comment-13609232
]
Bikas Saha commented on YARN-378:
-
Where are we making sure that the global value in the conf is a sane value (ie
admin has not mistakenly set a bad value)?
Can we write the if condition as if(foo 0 || foo MAX) then foo = MAX). The
nested loop will make the reader of the code think more than needed IMO.
{code}
-this.maxRetries = conf.getInt(YarnConfiguration.RM_AM_MAX_RETRIES,
-YarnConfiguration.DEFAULT_RM_AM_MAX_RETRIES);
+int globalMaxAppAttempts =
conf.getInt(YarnConfiguration.RM_AM_MAX_ATTEMPTS,
+YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS);
+int individualMaxAppAttempts = submissionContext.getMaxAppAttempts();
+if (individualMaxAppAttempts = 0) {
+this.maxAppAttempts = globalMaxAppAttempts;
+} else {
+ if (individualMaxAppAttempts = globalMaxAppAttempts) {
+this.maxAppAttempts = individualMaxAppAttempts;
+ } else {
+this.maxAppAttempts = globalMaxAppAttempts;
{code}
ApplicationMaster retry times should be set by Client
-
Key: YARN-378
URL: https://issues.apache.org/jira/browse/YARN-378
Project: Hadoop YARN
Issue Type: Sub-task
Components: client, resourcemanager
Environment: suse
Reporter: xieguiming
Assignee: Zhijie Shen
Labels: usability
Attachments: YARN-378_10.patch, YARN-378_1.patch, YARN-378_2.patch,
YARN-378_3.patch, YARN-378_4.patch, YARN-378_5.patch, YARN-378_6.patch,
YARN-378_6.patch, YARN-378_7.patch, YARN-378_8.patch, YARN-378_9.patch,
YARN-378_MAPREDUCE-5062.patch
We should support that different client or user have different
ApplicationMaster retry times. It also say that
yarn.resourcemanager.am.max-retries should be set by client.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-109) .tmp file is not deleted for localized archives
[ https://issues.apache.org/jira/browse/YARN-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609233#comment-13609233 ] omkar vinit joshi commented on YARN-109: [~mayank_bansal] Thanks for updating it. :) Now it looks good. .tmp file is not deleted for localized archives --- Key: YARN-109 URL: https://issues.apache.org/jira/browse/YARN-109 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 0.23.3, 2.0.0-alpha Reporter: Jason Lowe Assignee: Mayank Bansal Attachments: YARN-109-trunk-1.patch, YARN-109-trunk-2.patch, YARN-109-trunk-3.patch, YARN-109-trunk.patch When archives are localized they are initially created as a .tmp file and unpacked from that file. However the .tmp file is not deleted afterwards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-378) ApplicationMaster retry times should be set by Client
[
https://issues.apache.org/jira/browse/YARN-378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609267#comment-13609267
]
Zhijie Shen commented on YARN-378:
--
{quote}
Where are we making sure that the global value in the conf is a sane value (ie
admin has not mistakenly set a bad value)?
{quote}
In ResourceManager#validateConfigs
{quote}
Can we write the if condition as if(foo 0 || foo MAX) then foo = MAX). The
nested loop will make the reader of the code think more than needed IMO.
{quote}
Make sense, but I'd like to differentiate the two cases, and log warning
messages. How about the following logic?
{code}
if (individual = 0) {
max = global;
LOG.warn(invalid);
} else if (individual = global) {
max = global;
LOG.warn(larger than global);
} else {
max = individual;
}
{code}
ApplicationMaster retry times should be set by Client
-
Key: YARN-378
URL: https://issues.apache.org/jira/browse/YARN-378
Project: Hadoop YARN
Issue Type: Sub-task
Components: client, resourcemanager
Environment: suse
Reporter: xieguiming
Assignee: Zhijie Shen
Labels: usability
Attachments: YARN-378_10.patch, YARN-378_1.patch, YARN-378_2.patch,
YARN-378_3.patch, YARN-378_4.patch, YARN-378_5.patch, YARN-378_6.patch,
YARN-378_6.patch, YARN-378_7.patch, YARN-378_8.patch, YARN-378_9.patch,
YARN-378_MAPREDUCE-5062.patch
We should support that different client or user have different
ApplicationMaster retry times. It also say that
yarn.resourcemanager.am.max-retries should be set by client.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-209) Capacity scheduler can leave application in pending state
[ https://issues.apache.org/jira/browse/YARN-209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen reassigned YARN-209: Assignee: Zhijie Shen (was: Bikas Saha) Capacity scheduler can leave application in pending state - Key: YARN-209 URL: https://issues.apache.org/jira/browse/YARN-209 Project: Hadoop YARN Issue Type: Bug Reporter: Bikas Saha Assignee: Zhijie Shen Fix For: 3.0.0 Attachments: YARN-209.1.patch, YARN-209-test.patch Say application A is submitted but at that time it does not meet the bar for activation because of resource limit settings for applications. After that if more hardware is added to the system and the application becomes valid it still remains in pending state, likely forever. This might be rare to hit in real life because enough NM's heartbeat to the RM before applications can get submitted. But a change in settings or heartbeat interval might make it easier to repro. In RM restart scenarios, this will likely hit more if its implemented by re-playing events and re-submitting applications to the scheduler before the RPC to NM's is activated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-209) Capacity scheduler can leave application in pending state
[ https://issues.apache.org/jira/browse/YARN-209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609303#comment-13609303 ] Zhijie Shen commented on YARN-209: -- Whenever more hardware is added to the cluster, LeafQueue#updateClusterResource will be triggered. The problem can be fixed by the patch for YARN-474 as well. Capacity scheduler can leave application in pending state - Key: YARN-209 URL: https://issues.apache.org/jira/browse/YARN-209 Project: Hadoop YARN Issue Type: Bug Reporter: Bikas Saha Assignee: Zhijie Shen Fix For: 3.0.0 Attachments: YARN-209.1.patch, YARN-209-test.patch Say application A is submitted but at that time it does not meet the bar for activation because of resource limit settings for applications. After that if more hardware is added to the system and the application becomes valid it still remains in pending state, likely forever. This might be rare to hit in real life because enough NM's heartbeat to the RM before applications can get submitted. But a change in settings or heartbeat interval might make it easier to repro. In RM restart scenarios, this will likely hit more if its implemented by re-playing events and re-submitting applications to the scheduler before the RPC to NM's is activated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-493) NodeManager job control logic flaws on Windows
[
https://issues.apache.org/jira/browse/YARN-493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated YARN-493:
---
Description: Both product and test code contain some platform-specific
assumptions, such as availability of bash for executing a command in a
container and signals to check existence of a process and terminate it. (was:
The tests contain some platform-specific assumptions, such as availability of
bash for executing a command in a container and signals to check existence of a
process and terminate it.)
Summary: NodeManager job control logic flaws on Windows (was:
TestContainerManager fails on Windows)
I'm expanding the scope of this jira to cover some flaws I've discovered in
NodeManager's job control logic on Windows:
# Windows was erroneously flagged as supporting setsid, which caused prepending
of a '-' character to the job ID passed to winutils.
# Exit code from job terminated by winutils task kill differed from
expectations in YARN Java code, so that it couldn't tell the difference between
a killed container vs. a container that had exited with failure.
# Multiple tests were relying on bash scripts and signals for launching and
controlling containers.
I have a patch in progress. With the expanded scope, the patch will fix the
following tests on Windows: {{TestContainerLaunch}}, {{TestContainerManager}},
and {{TestNodeManagerShutdown}}.
NodeManager job control logic flaws on Windows
--
Key: YARN-493
URL: https://issues.apache.org/jira/browse/YARN-493
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Fix For: 3.0.0
Both product and test code contain some platform-specific assumptions, such
as availability of bash for executing a command in a container and signals to
check existence of a process and terminate it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-378) ApplicationMaster retry times should be set by Client
[ https://issues.apache.org/jira/browse/YARN-378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-378: - Attachment: YARN-378_11.patch Improve the if condition in RMAppImpl. ApplicationMaster retry times should be set by Client - Key: YARN-378 URL: https://issues.apache.org/jira/browse/YARN-378 Project: Hadoop YARN Issue Type: Sub-task Components: client, resourcemanager Environment: suse Reporter: xieguiming Assignee: Zhijie Shen Labels: usability Attachments: YARN-378_10.patch, YARN-378_11.patch, YARN-378_1.patch, YARN-378_2.patch, YARN-378_3.patch, YARN-378_4.patch, YARN-378_5.patch, YARN-378_6.patch, YARN-378_6.patch, YARN-378_7.patch, YARN-378_8.patch, YARN-378_9.patch, YARN-378_MAPREDUCE-5062.patch We should support that different client or user have different ApplicationMaster retry times. It also say that yarn.resourcemanager.am.max-retries should be set by client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-378) ApplicationMaster retry times should be set by Client
[ https://issues.apache.org/jira/browse/YARN-378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-378: - Attachment: YARN-378_MAPREDUCE-5062.2.patch Combine the newest patches of YARN-378 and MAPREDUCE-5062 again for Jenkins to verify them. ApplicationMaster retry times should be set by Client - Key: YARN-378 URL: https://issues.apache.org/jira/browse/YARN-378 Project: Hadoop YARN Issue Type: Sub-task Components: client, resourcemanager Environment: suse Reporter: xieguiming Assignee: Zhijie Shen Labels: usability Attachments: YARN-378_10.patch, YARN-378_11.patch, YARN-378_1.patch, YARN-378_2.patch, YARN-378_3.patch, YARN-378_4.patch, YARN-378_5.patch, YARN-378_6.patch, YARN-378_6.patch, YARN-378_7.patch, YARN-378_8.patch, YARN-378_9.patch, YARN-378_MAPREDUCE-5062.2.patch, YARN-378_MAPREDUCE-5062.patch We should support that different client or user have different ApplicationMaster retry times. It also say that yarn.resourcemanager.am.max-retries should be set by client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-488) TestContainerManagerSecurity fails on Windows
[ https://issues.apache.org/jira/browse/YARN-488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609429#comment-13609429 ] Hitesh Shah commented on YARN-488: -- +1. Will commit shortly. TestContainerManagerSecurity fails on Windows - Key: YARN-488 URL: https://issues.apache.org/jira/browse/YARN-488 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: YARN-488.1.patch, YARN-488.2.patch These tests are failing to launch containers correctly when running on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-495) Containers are not terminated when the NM is stopped
[ https://issues.apache.org/jira/browse/YARN-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jian he updated YARN-495: - Summary: Containers are not terminated when the NM is stopped (was: Containers are not cleaned up when a reboot is sent from RM) Containers are not terminated when the NM is stopped Key: YARN-495 URL: https://issues.apache.org/jira/browse/YARN-495 Project: Hadoop YARN Issue Type: Bug Reporter: jian he Assignee: jian he When a reboot command is sent from RM, the node manager doesn't clean up the containers while its stopping. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-488) TestContainerManagerSecurity fails on Windows
[ https://issues.apache.org/jira/browse/YARN-488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609448#comment-13609448 ] Hudson commented on YARN-488: - Integrated in Hadoop-trunk-Commit #3501 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3501/]) YARN-488. TestContainerManagerSecurity fails on Windows. Contributed by Chris Nauroth. (Revision 1459514) Result = SUCCESS hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1459514 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestContainerManagerSecurity.java TestContainerManagerSecurity fails on Windows - Key: YARN-488 URL: https://issues.apache.org/jira/browse/YARN-488 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: 3.0.0 Attachments: YARN-488.1.patch, YARN-488.2.patch These tests are failing to launch containers correctly when running on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-490) TestDistributedShell fails on Windows
[ https://issues.apache.org/jira/browse/YARN-490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609449#comment-13609449 ] Hitesh Shah commented on YARN-490: -- +1. Will commit shortly. TestDistributedShell fails on Windows - Key: YARN-490 URL: https://issues.apache.org/jira/browse/YARN-490 Project: Hadoop YARN Issue Type: Bug Components: applications/distributed-shell Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Labels: windows Attachments: YARN-490.1.patch There are a few platform-specific assumption in distributed shell (both main code and test code) that prevent it from working correctly on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-381) Improve FS docs
[ https://issues.apache.org/jira/browse/YARN-381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-381: Attachment: YARN-381.patch Improve FS docs --- Key: YARN-381 URL: https://issues.apache.org/jira/browse/YARN-381 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.0.0-alpha Reporter: Eli Collins Assignee: Sandy Ryza Priority: Minor Attachments: YARN-381.patch The MR2 FS docs could use some improvements. Configuration: - sizebasedweight - what is the size here? Total memory usage? Pool properties: - minResources - what does min amount of aggregate memory mean given that this is not a reservation? - maxResources - is this a hard limit? - weight: How is this ratio configured? Eg base is 1 and all weights are relative to that? - schedulingMode - what is the default? Is fifo pure fifo, eg waits until all tasks for the job are finished before launching the next job? There's no mention of ACLs, even though they're supported. See the CS docs for comparison. Also there are a couple typos worth fixing while we're at it, eg finish. apps to run Worth keeping in mind that some of these will need to be updated to reflect that resource calculators are now pluggable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-381) Improve FS docs
[ https://issues.apache.org/jira/browse/YARN-381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-381: Component/s: documentation Improve FS docs --- Key: YARN-381 URL: https://issues.apache.org/jira/browse/YARN-381 Project: Hadoop YARN Issue Type: Improvement Components: documentation Affects Versions: 2.0.0-alpha Reporter: Eli Collins Assignee: Sandy Ryza Priority: Minor Attachments: YARN-381.patch The MR2 FS docs could use some improvements. Configuration: - sizebasedweight - what is the size here? Total memory usage? Pool properties: - minResources - what does min amount of aggregate memory mean given that this is not a reservation? - maxResources - is this a hard limit? - weight: How is this ratio configured? Eg base is 1 and all weights are relative to that? - schedulingMode - what is the default? Is fifo pure fifo, eg waits until all tasks for the job are finished before launching the next job? There's no mention of ACLs, even though they're supported. See the CS docs for comparison. Also there are a couple typos worth fixing while we're at it, eg finish. apps to run Worth keeping in mind that some of these will need to be updated to reflect that resource calculators are now pluggable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-490) TestDistributedShell fails on Windows
[ https://issues.apache.org/jira/browse/YARN-490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609457#comment-13609457 ] Hudson commented on YARN-490: - Integrated in Hadoop-trunk-Commit #3502 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3502/]) YARN-490. TestDistributedShell fails on Windows. Contributed by Chris Nauroth. (Revision 1459520) Result = SUCCESS hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1459520 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java TestDistributedShell fails on Windows - Key: YARN-490 URL: https://issues.apache.org/jira/browse/YARN-490 Project: Hadoop YARN Issue Type: Bug Components: applications/distributed-shell Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Labels: windows Fix For: 3.0.0 Attachments: YARN-490.1.patch There are a few platform-specific assumption in distributed shell (both main code and test code) that prevent it from working correctly on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-491) TestContainerLogsPage fails on Windows
[
https://issues.apache.org/jira/browse/YARN-491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609472#comment-13609472
]
Hitesh Shah commented on YARN-491:
--
+1. Committing shortly.
TestContainerLogsPage fails on Windows
--
Key: YARN-491
URL: https://issues.apache.org/jira/browse/YARN-491
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: YARN-491.1.patch
{{TestContainerLogsPage}} contains some code for initializing a log directory
that doesn't work correctly on Windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-491) TestContainerLogsPage fails on Windows
[
https://issues.apache.org/jira/browse/YARN-491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609490#comment-13609490
]
Hudson commented on YARN-491:
-
Integrated in Hadoop-trunk-Commit #3503 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/3503/])
YARN-491. TestContainerLogsPage fails on Windows. Contributed by Chris
Nauroth. (Revision 1459526)
Result = SUCCESS
hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1459526
Files :
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestContainerLogsPage.java
TestContainerLogsPage fails on Windows
--
Key: YARN-491
URL: https://issues.apache.org/jira/browse/YARN-491
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Fix For: 3.0.0
Attachments: YARN-491.1.patch
{{TestContainerLogsPage}} contains some code for initializing a log directory
that doesn't work correctly on Windows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-109) .tmp file is not deleted for localized archives
[ https://issues.apache.org/jira/browse/YARN-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609495#comment-13609495 ] Mayank Bansal commented on YARN-109: [~revans2] Thanks for the comments and review. Updating the new patch. For adding the jar/tar/zip files in to repo I am not sure at this moment as i see in other cases it is also generating it on the fly. However I think your point is valid. I will recommend to do that in separate JIRA for everything to maintain consistency. Thoughts? Thanks, Mayank .tmp file is not deleted for localized archives --- Key: YARN-109 URL: https://issues.apache.org/jira/browse/YARN-109 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 0.23.3, 2.0.0-alpha Reporter: Jason Lowe Assignee: Mayank Bansal Attachments: YARN-109-trunk-1.patch, YARN-109-trunk-2.patch, YARN-109-trunk-3.patch, YARN-109-trunk.patch When archives are localized they are initially created as a .tmp file and unpacked from that file. However the .tmp file is not deleted afterwards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-109) .tmp file is not deleted for localized archives
[ https://issues.apache.org/jira/browse/YARN-109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated YARN-109: --- Attachment: YARN-109-trunk-4.patch .tmp file is not deleted for localized archives --- Key: YARN-109 URL: https://issues.apache.org/jira/browse/YARN-109 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 0.23.3, 2.0.0-alpha Reporter: Jason Lowe Assignee: Mayank Bansal Attachments: YARN-109-trunk-1.patch, YARN-109-trunk-2.patch, YARN-109-trunk-3.patch, YARN-109-trunk-4.patch, YARN-109-trunk.patch When archives are localized they are initially created as a .tmp file and unpacked from that file. However the .tmp file is not deleted afterwards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-109) .tmp file is not deleted for localized archives
[ https://issues.apache.org/jira/browse/YARN-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609509#comment-13609509 ] Robert Joseph Evans commented on YARN-109: -- That is fine with me. My concern was mostly with Windows support. tar, zip, jar, etc. should be there, but bash may not be. So if you want to file a new JIRA that is fine, if not you can just wait for windows support to be merged in and see if it breaks. .tmp file is not deleted for localized archives --- Key: YARN-109 URL: https://issues.apache.org/jira/browse/YARN-109 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 0.23.3, 2.0.0-alpha Reporter: Jason Lowe Assignee: Mayank Bansal Attachments: YARN-109-trunk-1.patch, YARN-109-trunk-2.patch, YARN-109-trunk-3.patch, YARN-109-trunk-4.patch, YARN-109-trunk.patch When archives are localized they are initially created as a .tmp file and unpacked from that file. However the .tmp file is not deleted afterwards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-378) ApplicationMaster retry times should be set by Client
[
https://issues.apache.org/jira/browse/YARN-378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609512#comment-13609512
]
Hadoop QA commented on YARN-378:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574871/YARN-378_MAPREDUCE-5062.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/557//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/557//console
This message is automatically generated.
ApplicationMaster retry times should be set by Client
-
Key: YARN-378
URL: https://issues.apache.org/jira/browse/YARN-378
Project: Hadoop YARN
Issue Type: Sub-task
Components: client, resourcemanager
Environment: suse
Reporter: xieguiming
Assignee: Zhijie Shen
Labels: usability
Attachments: YARN-378_10.patch, YARN-378_11.patch, YARN-378_1.patch,
YARN-378_2.patch, YARN-378_3.patch, YARN-378_4.patch, YARN-378_5.patch,
YARN-378_6.patch, YARN-378_6.patch, YARN-378_7.patch, YARN-378_8.patch,
YARN-378_9.patch, YARN-378_MAPREDUCE-5062.2.patch,
YARN-378_MAPREDUCE-5062.patch
We should support that different client or user have different
ApplicationMaster retry times. It also say that
yarn.resourcemanager.am.max-retries should be set by client.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-109) .tmp file is not deleted for localized archives
[
https://issues.apache.org/jira/browse/YARN-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609516#comment-13609516
]
Hadoop QA commented on YARN-109:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574885/YARN-109-trunk-4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:
org.apache.hadoop.yarn.util.TestFSDownload
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/559//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/559//console
This message is automatically generated.
.tmp file is not deleted for localized archives
---
Key: YARN-109
URL: https://issues.apache.org/jira/browse/YARN-109
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 0.23.3, 2.0.0-alpha
Reporter: Jason Lowe
Assignee: Mayank Bansal
Attachments: YARN-109-trunk-1.patch, YARN-109-trunk-2.patch,
YARN-109-trunk-3.patch, YARN-109-trunk-4.patch, YARN-109-trunk.patch
When archives are localized they are initially created as a .tmp file and
unpacked from that file. However the .tmp file is not deleted afterwards.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-439) Flatten NodeHeartbeatResponse
[ https://issues.apache.org/jira/browse/YARN-439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-439: --- Attachment: YARN-439.2.patch Flatten NodeHeartbeatResponse - Key: YARN-439 URL: https://issues.apache.org/jira/browse/YARN-439 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Xuan Gong Attachments: YARN-439.1.patch, YARN-439.2.patch NodeheartbeatResponse has another wrapper HeartbeatResponse under it, which can be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-439) Flatten NodeHeartbeatResponse
[ https://issues.apache.org/jira/browse/YARN-439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609533#comment-13609533 ] Xuan Gong commented on YARN-439: Recreate the patch based on latest trunk version Flatten NodeHeartbeatResponse - Key: YARN-439 URL: https://issues.apache.org/jira/browse/YARN-439 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Xuan Gong Attachments: YARN-439.1.patch, YARN-439.2.patch NodeheartbeatResponse has another wrapper HeartbeatResponse under it, which can be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-496) Fair scheduler configs are refreshed inconsistently in reinitialize
Sandy Ryza created YARN-496: --- Summary: Fair scheduler configs are refreshed inconsistently in reinitialize Key: YARN-496 URL: https://issues.apache.org/jira/browse/YARN-496 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Priority: Minor When FairScheduler#reinitialize is called, some of the scheduler-wide configs are refreshed and others aren't. They should all be refreshed. Ones that are refreshed: userAsDefaultQueue, nodeLocalityThreshold, rackLocalityThreshold, preemptionEnabled Ones that aren't: minimumAllocation, maximumAllocation, assignMultiple, maxAssign -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-109) .tmp file is not deleted for localized archives
[ https://issues.apache.org/jira/browse/YARN-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609548#comment-13609548 ] Mayank Bansal commented on YARN-109: [~revans2] Sure I will file a new JIRA for that. Thanks, Mayank .tmp file is not deleted for localized archives --- Key: YARN-109 URL: https://issues.apache.org/jira/browse/YARN-109 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 0.23.3, 2.0.0-alpha Reporter: Jason Lowe Assignee: Mayank Bansal Attachments: YARN-109-trunk-1.patch, YARN-109-trunk-2.patch, YARN-109-trunk-3.patch, YARN-109-trunk-4.patch, YARN-109-trunk.patch When archives are localized they are initially created as a .tmp file and unpacked from that file. However the .tmp file is not deleted afterwards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-109) .tmp file is not deleted for localized archives
[ https://issues.apache.org/jira/browse/YARN-109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated YARN-109: --- Attachment: YARN-109-trunk-5.patch .tmp file is not deleted for localized archives --- Key: YARN-109 URL: https://issues.apache.org/jira/browse/YARN-109 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 0.23.3, 2.0.0-alpha Reporter: Jason Lowe Assignee: Mayank Bansal Attachments: YARN-109-trunk-1.patch, YARN-109-trunk-2.patch, YARN-109-trunk-3.patch, YARN-109-trunk-4.patch, YARN-109-trunk-5.patch, YARN-109-trunk.patch When archives are localized they are initially created as a .tmp file and unpacked from that file. However the .tmp file is not deleted afterwards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-417) Add a poller that allows the AM to receive notifications when it is assigned containers
[
https://issues.apache.org/jira/browse/YARN-417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609553#comment-13609553
]
Hadoop QA commented on YARN-417:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574689/YARN-417-6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/560//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/560//console
This message is automatically generated.
Add a poller that allows the AM to receive notifications when it is assigned
containers
---
Key: YARN-417
URL: https://issues.apache.org/jira/browse/YARN-417
Project: Hadoop YARN
Issue Type: Sub-task
Components: api, applications
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: AMRMClientAsync-1.java, AMRMClientAsync.java,
YARN-417-1.patch, YARN-417-2.patch, YARN-417-3.patch, YARN-417-4.patch,
YARN-417-4.patch, YARN-417-5.patch, YARN-417-6.patch, YARN-417.patch,
YarnAppMaster.java, YarnAppMasterListener.java
Writing AMs would be easier for some if they did not have to handle
heartbeating to the RM on their own.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-209) Capacity scheduler can leave application in pending state
[ https://issues.apache.org/jira/browse/YARN-209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-209: - Attachment: YARN-209.2.patch Given YARN-474 is fixed, the patch can be used to test the scenario that the applications are submitted before NM are registered. Capacity scheduler can leave application in pending state - Key: YARN-209 URL: https://issues.apache.org/jira/browse/YARN-209 Project: Hadoop YARN Issue Type: Bug Reporter: Bikas Saha Assignee: Zhijie Shen Fix For: 3.0.0 Attachments: YARN-209.1.patch, YARN-209.2.patch, YARN-209-test.patch Say application A is submitted but at that time it does not meet the bar for activation because of resource limit settings for applications. After that if more hardware is added to the system and the application becomes valid it still remains in pending state, likely forever. This might be rare to hit in real life because enough NM's heartbeat to the RM before applications can get submitted. But a change in settings or heartbeat interval might make it easier to repro. In RM restart scenarios, this will likely hit more if its implemented by re-playing events and re-submitting applications to the scheduler before the RPC to NM's is activated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-417) Add a poller that allows the AM to receive notifications when it is assigned containers
[ https://issues.apache.org/jira/browse/YARN-417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609576#comment-13609576 ] Bikas Saha commented on YARN-417: - +1. The new locking also makes sure that the queue.put is outside the lock and wont get synch blocked upon a bounded queue. Thanks for being patient with my reviews! Add a poller that allows the AM to receive notifications when it is assigned containers --- Key: YARN-417 URL: https://issues.apache.org/jira/browse/YARN-417 Project: Hadoop YARN Issue Type: Sub-task Components: api, applications Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: AMRMClientAsync-1.java, AMRMClientAsync.java, YARN-417-1.patch, YARN-417-2.patch, YARN-417-3.patch, YARN-417-4.patch, YARN-417-4.patch, YARN-417-5.patch, YARN-417-6.patch, YARN-417.patch, YarnAppMaster.java, YarnAppMasterListener.java Writing AMs would be easier for some if they did not have to handle heartbeating to the RM on their own. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-439) Flatten NodeHeartbeatResponse
[
https://issues.apache.org/jira/browse/YARN-439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609577#comment-13609577
]
Hadoop QA commented on YARN-439:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574890/YARN-439.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 14 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 7
warning messages.
{color:red}-1 eclipse:eclipse{color}. The patch failed to build with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/561//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/561//console
This message is automatically generated.
Flatten NodeHeartbeatResponse
-
Key: YARN-439
URL: https://issues.apache.org/jira/browse/YARN-439
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Xuan Gong
Attachments: YARN-439.1.patch, YARN-439.2.patch
NodeheartbeatResponse has another wrapper HeartbeatResponse under it, which
can be removed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-496) Fair scheduler configs are refreshed inconsistently in reinitialize
[
https://issues.apache.org/jira/browse/YARN-496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609579#comment-13609579
]
Karthik Kambatla commented on YARN-496:
---
There seems to be unnecessary code duplication in reinitialize(). A cleaner
approach might be to move things around a little to avoid duplication.
{code}
// do common initialization stuff for both first time initialization and
subsequent reinitializations
if (first-time-initialization) {
// do first-time stuff
}
{code}
Fair scheduler configs are refreshed inconsistently in reinitialize
---
Key: YARN-496
URL: https://issues.apache.org/jira/browse/YARN-496
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Priority: Minor
Attachments: YARN-496.patch
When FairScheduler#reinitialize is called, some of the scheduler-wide configs
are refreshed and others aren't. They should all be refreshed.
Ones that are refreshed: userAsDefaultQueue, nodeLocalityThreshold,
rackLocalityThreshold, preemptionEnabled
Ones that aren't: minimumAllocation, maximumAllocation, assignMultiple,
maxAssign
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-109) .tmp file is not deleted for localized archives
[
https://issues.apache.org/jira/browse/YARN-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609581#comment-13609581
]
Hadoop QA commented on YARN-109:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574891/YARN-109-trunk-5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/563//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/563//console
This message is automatically generated.
.tmp file is not deleted for localized archives
---
Key: YARN-109
URL: https://issues.apache.org/jira/browse/YARN-109
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 0.23.3, 2.0.0-alpha
Reporter: Jason Lowe
Assignee: Mayank Bansal
Attachments: YARN-109-trunk-1.patch, YARN-109-trunk-2.patch,
YARN-109-trunk-3.patch, YARN-109-trunk-4.patch, YARN-109-trunk-5.patch,
YARN-109-trunk.patch
When archives are localized they are initially created as a .tmp file and
unpacked from that file. However the .tmp file is not deleted afterwards.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-417) Create AMRMClient wrapper that provides asynchronous callbacks
[ https://issues.apache.org/jira/browse/YARN-417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated YARN-417: Summary: Create AMRMClient wrapper that provides asynchronous callbacks (was: Add a poller that allows the AM to receive notifications when it is assigned containers) Create AMRMClient wrapper that provides asynchronous callbacks -- Key: YARN-417 URL: https://issues.apache.org/jira/browse/YARN-417 Project: Hadoop YARN Issue Type: Sub-task Components: api, applications Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: AMRMClientAsync-1.java, AMRMClientAsync.java, YARN-417-1.patch, YARN-417-2.patch, YARN-417-3.patch, YARN-417-4.patch, YARN-417-4.patch, YARN-417-5.patch, YARN-417-6.patch, YARN-417.patch, YarnAppMaster.java, YarnAppMasterListener.java Writing AMs would be easier for some if they did not have to handle heartbeating to the RM on their own. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-496) Fair scheduler configs are refreshed inconsistently in reinitialize
[
https://issues.apache.org/jira/browse/YARN-496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609582#comment-13609582
]
Hadoop QA commented on YARN-496:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574894/YARN-496.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/562//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/562//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/562//console
This message is automatically generated.
Fair scheduler configs are refreshed inconsistently in reinitialize
---
Key: YARN-496
URL: https://issues.apache.org/jira/browse/YARN-496
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Priority: Minor
Attachments: YARN-496.patch
When FairScheduler#reinitialize is called, some of the scheduler-wide configs
are refreshed and others aren't. They should all be refreshed.
Ones that are refreshed: userAsDefaultQueue, nodeLocalityThreshold,
rackLocalityThreshold, preemptionEnabled
Ones that aren't: minimumAllocation, maximumAllocation, assignMultiple,
maxAssign
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-109) .tmp file is not deleted for localized archives
[ https://issues.apache.org/jira/browse/YARN-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609589#comment-13609589 ] Mayank Bansal commented on YARN-109: [~revans2] Thanks for the review. Do you mind committing it as well? Thanks, Mayank .tmp file is not deleted for localized archives --- Key: YARN-109 URL: https://issues.apache.org/jira/browse/YARN-109 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 0.23.3, 2.0.0-alpha Reporter: Jason Lowe Assignee: Mayank Bansal Attachments: YARN-109-trunk-1.patch, YARN-109-trunk-2.patch, YARN-109-trunk-3.patch, YARN-109-trunk-4.patch, YARN-109-trunk-5.patch, YARN-109-trunk.patch When archives are localized they are initially created as a .tmp file and unpacked from that file. However the .tmp file is not deleted afterwards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-209) Capacity scheduler can leave application in pending state
[
https://issues.apache.org/jira/browse/YARN-209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609601#comment-13609601
]
Hadoop QA commented on YARN-209:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574896/YARN-209.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 eclipse:eclipse{color}. The patch failed to build with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/564//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/564//console
This message is automatically generated.
Capacity scheduler can leave application in pending state
-
Key: YARN-209
URL: https://issues.apache.org/jira/browse/YARN-209
Project: Hadoop YARN
Issue Type: Bug
Reporter: Bikas Saha
Assignee: Zhijie Shen
Fix For: 3.0.0
Attachments: YARN-209.1.patch, YARN-209.2.patch, YARN-209-test.patch
Say application A is submitted but at that time it does not meet the bar for
activation because of resource limit settings for applications. After that if
more hardware is added to the system and the application becomes valid it
still remains in pending state, likely forever.
This might be rare to hit in real life because enough NM's heartbeat to the
RM before applications can get submitted. But a change in settings or
heartbeat interval might make it easier to repro. In RM restart scenarios,
this will likely hit more if its implemented by re-playing events and
re-submitting applications to the scheduler before the RPC to NM's is
activated.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-439) Flatten NodeHeartbeatResponse
[ https://issues.apache.org/jira/browse/YARN-439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609609#comment-13609609 ] Siddharth Seth commented on YARN-439: - Xuan, NodeHeartbeatResponse also needs some cleanup (part of YARN-441) - to remove unnecessary methods. Also, we should rename getContainersToCleanupList to getContainersToCleanup, getApplicationsToCleanupList to getApplicationsToCleanup. Do you mind making these changes as part of this jira ? (likely a smaller patch) Otherwise we can create a separate jira. Flatten NodeHeartbeatResponse - Key: YARN-439 URL: https://issues.apache.org/jira/browse/YARN-439 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Xuan Gong Attachments: YARN-439.1.patch, YARN-439.2.patch NodeheartbeatResponse has another wrapper HeartbeatResponse under it, which can be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-71) Ensure/confirm that the NodeManager cleans up local-dirs on restart
[ https://issues.apache.org/jira/browse/YARN-71?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-71: -- Attachment: YARN-71.12.patch 1.Spy the deletionService to verify the deletion. 2.Use FileContext.listStatus to get all the file status 3. remove bash file, create files in fileCache instead Ensure/confirm that the NodeManager cleans up local-dirs on restart --- Key: YARN-71 URL: https://issues.apache.org/jira/browse/YARN-71 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Reporter: Vinod Kumar Vavilapalli Assignee: Xuan Gong Priority: Critical Attachments: YARN-71.10.patch, YARN-71.11.patch, YARN-71.12.patch, YARN-71.1.patch, YARN-71.2.patch, YARN-71.3.patch, YARN.71.4.patch, YARN-71.5.patch, YARN-71.6.patch, YARN-71.7.patch, YARN-71.8.patch, YARN-71.9.patch We have to make sure that NodeManagers cleanup their local files on restart. It may already be working like that in which case we should have tests validating this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-417) Create AMRMClient wrapper that provides asynchronous callbacks
[ https://issues.apache.org/jira/browse/YARN-417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609617#comment-13609617 ] Hudson commented on YARN-417: - Integrated in Hadoop-trunk-Commit #3506 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3506/]) YARN-417. Create AMRMClient wrapper that provides asynchronous callbacks. (Sandy Ryza via bikas) (Revision 1459555) Result = SUCCESS bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1459555 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java Create AMRMClient wrapper that provides asynchronous callbacks -- Key: YARN-417 URL: https://issues.apache.org/jira/browse/YARN-417 Project: Hadoop YARN Issue Type: Sub-task Components: api, applications Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: AMRMClientAsync-1.java, AMRMClientAsync.java, YARN-417-1.patch, YARN-417-2.patch, YARN-417-3.patch, YARN-417-4.patch, YARN-417-4.patch, YARN-417-5.patch, YARN-417-6.patch, YARN-417.patch, YarnAppMaster.java, YarnAppMasterListener.java Writing AMs would be easier for some if they did not have to handle heartbeating to the RM on their own. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-479) NM retry behavior for connection to RM should be similar for lost heartbeats
[ https://issues.apache.org/jira/browse/YARN-479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609633#comment-13609633 ] Xuan Gong commented on YARN-479: Oh. I got it. Do you think we still need a test case since testNMRegistration only covered part of it? For example, Test what happen if NM will never get a response back, etc. This behavior is almost the same as nm retry for connection to RM. And the retry behavior for connection to RM has already been covered by other test case. So, I am not sure whether we still need a new test case just for handling heartbeat lost. Other than that, I think the patch looks good. Some minor format issue need to be fixed, such as extra spaces. And this //Waiting for rmStartIntervalMS, RM will be started in testNMRegistration() can be removed. Re-phrase the error message and warning message, please. We are waiting for heartbeat response back here. NM retry behavior for connection to RM should be similar for lost heartbeats Key: YARN-479 URL: https://issues.apache.org/jira/browse/YARN-479 Project: Hadoop YARN Issue Type: Bug Reporter: Hitesh Shah Assignee: jian he Attachments: YARN-479.1.patch, YARN-479.2.patch Regardless of connection loss at the start or at an intermediate point, NM's retry behavior to the RM should follow the same flow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-479) NM retry behavior for connection to RM should be similar for lost heartbeats
[ https://issues.apache.org/jira/browse/YARN-479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609642#comment-13609642 ] jian he commented on YARN-479: -- thanks, Xuan. I'll update the patch NM retry behavior for connection to RM should be similar for lost heartbeats Key: YARN-479 URL: https://issues.apache.org/jira/browse/YARN-479 Project: Hadoop YARN Issue Type: Bug Reporter: Hitesh Shah Assignee: jian he Attachments: YARN-479.1.patch, YARN-479.2.patch Regardless of connection loss at the start or at an intermediate point, NM's retry behavior to the RM should follow the same flow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-71) Ensure/confirm that the NodeManager cleans up local-dirs on restart
[
https://issues.apache.org/jira/browse/YARN-71?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609643#comment-13609643
]
Hadoop QA commented on YARN-71:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574908/YARN-71.12.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/565//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/565//console
This message is automatically generated.
Ensure/confirm that the NodeManager cleans up local-dirs on restart
---
Key: YARN-71
URL: https://issues.apache.org/jira/browse/YARN-71
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Reporter: Vinod Kumar Vavilapalli
Assignee: Xuan Gong
Priority: Critical
Attachments: YARN-71.10.patch, YARN-71.11.patch, YARN-71.12.patch,
YARN-71.1.patch, YARN-71.2.patch, YARN-71.3.patch, YARN.71.4.patch,
YARN-71.5.patch, YARN-71.6.patch, YARN-71.7.patch, YARN-71.8.patch,
YARN-71.9.patch
We have to make sure that NodeManagers cleanup their local files on restart.
It may already be working like that in which case we should have tests
validating this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-417) Create AMRMClient wrapper that provides asynchronous callbacks
[ https://issues.apache.org/jira/browse/YARN-417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609649#comment-13609649 ] Sandy Ryza commented on YARN-417: - Thanks for the review and all the feedback! Create AMRMClient wrapper that provides asynchronous callbacks -- Key: YARN-417 URL: https://issues.apache.org/jira/browse/YARN-417 Project: Hadoop YARN Issue Type: Sub-task Components: api, applications Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: AMRMClientAsync-1.java, AMRMClientAsync.java, YARN-417-1.patch, YARN-417-2.patch, YARN-417-3.patch, YARN-417-4.patch, YARN-417-4.patch, YARN-417-5.patch, YARN-417-6.patch, YARN-417.patch, YarnAppMaster.java, YarnAppMasterListener.java Writing AMs would be easier for some if they did not have to handle heartbeating to the RM on their own. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-496) Fair scheduler configs are refreshed inconsistently in reinitialize
[ https://issues.apache.org/jira/browse/YARN-496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-496: Attachment: YARN-496.patch Fair scheduler configs are refreshed inconsistently in reinitialize --- Key: YARN-496 URL: https://issues.apache.org/jira/browse/YARN-496 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Priority: Minor Attachments: YARN-496.patch, YARN-496.patch When FairScheduler#reinitialize is called, some of the scheduler-wide configs are refreshed and others aren't. They should all be refreshed. Ones that are refreshed: userAsDefaultQueue, nodeLocalityThreshold, rackLocalityThreshold, preemptionEnabled Ones that aren't: minimumAllocation, maximumAllocation, assignMultiple, maxAssign -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-496) Fair scheduler configs are refreshed inconsistently in reinitialize
[ https://issues.apache.org/jira/browse/YARN-496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609655#comment-13609655 ] Sandy Ryza commented on YARN-496: - Updated patch does what Karthik requested. Fair scheduler configs are refreshed inconsistently in reinitialize --- Key: YARN-496 URL: https://issues.apache.org/jira/browse/YARN-496 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Priority: Minor Attachments: YARN-496.patch, YARN-496.patch When FairScheduler#reinitialize is called, some of the scheduler-wide configs are refreshed and others aren't. They should all be refreshed. Ones that are refreshed: userAsDefaultQueue, nodeLocalityThreshold, rackLocalityThreshold, preemptionEnabled Ones that aren't: minimumAllocation, maximumAllocation, assignMultiple, maxAssign -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-496) Fair scheduler configs are refreshed inconsistently in reinitialize
[
https://issues.apache.org/jira/browse/YARN-496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609687#comment-13609687
]
Hadoop QA commented on YARN-496:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574915/YARN-496.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/566//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/566//console
This message is automatically generated.
Fair scheduler configs are refreshed inconsistently in reinitialize
---
Key: YARN-496
URL: https://issues.apache.org/jira/browse/YARN-496
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Priority: Minor
Attachments: YARN-496.patch, YARN-496.patch
When FairScheduler#reinitialize is called, some of the scheduler-wide configs
are refreshed and others aren't. They should all be refreshed.
Ones that are refreshed: userAsDefaultQueue, nodeLocalityThreshold,
rackLocalityThreshold, preemptionEnabled
Ones that aren't: minimumAllocation, maximumAllocation, assignMultiple,
maxAssign
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-497) yarn unmanaged-am launcher jar does not define a main class in its manifest
Hitesh Shah created YARN-497: Summary: yarn unmanaged-am launcher jar does not define a main class in its manifest Key: YARN-497 URL: https://issues.apache.org/jira/browse/YARN-497 Project: Hadoop YARN Issue Type: Bug Reporter: Hitesh Shah Priority: Minor The jar should have a mainClass defined to make it easier to use with the hadoop jar command. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-498) Unmanaged AM launcher does not set Container Id into env for an AM
Hitesh Shah created YARN-498: Summary: Unmanaged AM launcher does not set Container Id into env for an AM Key: YARN-498 URL: https://issues.apache.org/jira/browse/YARN-498 Project: Hadoop YARN Issue Type: Bug Reporter: Hitesh Shah Currently, it only sets the app attempt id which is really not required as AMs are only expected to extract it from the container id. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-439) Flatten NodeHeartbeatResponse
[
https://issues.apache.org/jira/browse/YARN-439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609737#comment-13609737
]
Hadoop QA commented on YARN-439:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574919/YARN-439.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 14 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 7
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/567//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/567//console
This message is automatically generated.
Flatten NodeHeartbeatResponse
-
Key: YARN-439
URL: https://issues.apache.org/jira/browse/YARN-439
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Xuan Gong
Attachments: YARN-439.1.patch, YARN-439.2.patch, YARN-439.3.patch
NodeheartbeatResponse has another wrapper HeartbeatResponse under it, which
can be removed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-417) Create AMRMClient wrapper that provides asynchronous callbacks
[
https://issues.apache.org/jira/browse/YARN-417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609743#comment-13609743
]
Chris Nauroth commented on YARN-417:
Hi, Sandy. I am seeing {{TestUnmanagedAMLauncher}} hang when I try to run it,
and I think I've isolated it to this YARN-417 patch.
{{TestUnmanagedAMLauncher}} uses the {{ApplicationMaster}} from distributed
shell, so perhaps it's an unexpected side effect of the changes there. Do you
have any thoughts on this? Thanks!
Create AMRMClient wrapper that provides asynchronous callbacks
--
Key: YARN-417
URL: https://issues.apache.org/jira/browse/YARN-417
Project: Hadoop YARN
Issue Type: Sub-task
Components: api, applications
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: AMRMClientAsync-1.java, AMRMClientAsync.java,
YARN-417-1.patch, YARN-417-2.patch, YARN-417-3.patch, YARN-417-4.patch,
YARN-417-4.patch, YARN-417-5.patch, YARN-417-6.patch, YARN-417.patch,
YarnAppMaster.java, YarnAppMasterListener.java
Writing AMs would be easier for some if they did not have to handle
heartbeating to the RM on their own.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-417) Create AMRMClient wrapper that provides asynchronous callbacks
[ https://issues.apache.org/jira/browse/YARN-417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609779#comment-13609779 ] Sandy Ryza commented on YARN-417: - Hi Chris. I just tried running the test and I'm getting the hang too. I'll see if I can figure out what's causing it. Create AMRMClient wrapper that provides asynchronous callbacks -- Key: YARN-417 URL: https://issues.apache.org/jira/browse/YARN-417 Project: Hadoop YARN Issue Type: Sub-task Components: api, applications Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: AMRMClientAsync-1.java, AMRMClientAsync.java, YARN-417-1.patch, YARN-417-2.patch, YARN-417-3.patch, YARN-417-4.patch, YARN-417-4.patch, YARN-417-5.patch, YARN-417-6.patch, YARN-417.patch, YarnAppMaster.java, YarnAppMasterListener.java Writing AMs would be easier for some if they did not have to handle heartbeating to the RM on their own. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-495) Containers are not terminated when the NM is rebooted
[ https://issues.apache.org/jira/browse/YARN-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated YARN-495: Summary: Containers are not terminated when the NM is rebooted (was: Containers are not terminated when the NM is stopped) Containers are not terminated when the NM is rebooted - Key: YARN-495 URL: https://issues.apache.org/jira/browse/YARN-495 Project: Hadoop YARN Issue Type: Bug Reporter: jian he Assignee: jian he When a reboot command is sent from RM, the node manager doesn't clean up the containers while its stopping. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-417) Create AMRMClient wrapper that provides asynchronous callbacks
[ https://issues.apache.org/jira/browse/YARN-417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609801#comment-13609801 ] Sandy Ryza commented on YARN-417: - I think I've figured the problem. TestUnmanagedAMLauncher runs distributed shell with 0 containers. Prior to YARN-417, ApplicationMaster would check for being done at a regular interval. Now, using the AMRMClientAsync, it only checks on container completion, which never occurs because no containers are run. Should this be fixed in a separate JIRA or here? Create AMRMClient wrapper that provides asynchronous callbacks -- Key: YARN-417 URL: https://issues.apache.org/jira/browse/YARN-417 Project: Hadoop YARN Issue Type: Sub-task Components: api, applications Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: AMRMClientAsync-1.java, AMRMClientAsync.java, YARN-417-1.patch, YARN-417-2.patch, YARN-417-3.patch, YARN-417-4.patch, YARN-417-4.patch, YARN-417-5.patch, YARN-417-6.patch, YARN-417.patch, YarnAppMaster.java, YarnAppMasterListener.java Writing AMs would be easier for some if they did not have to handle heartbeating to the RM on their own. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (YARN-71) Ensure/confirm that the NodeManager cleans up local-dirs on restart
[ https://issues.apache.org/jira/browse/YARN-71?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609803#comment-13609803 ] Siddharth Seth edited comment on YARN-71 at 3/22/13 2:07 AM: - Couple more comments. Hopefully the last set. - //queue deletions here, rather than NM init? - this comment isn't required anymore. - The log messages in the delete method should not suppress the exception ( LOG.warn(Failed to delete localDir: + localDir, e) ; ) - There's a log message in the test which says named nobody - this likely needs to be user - Bunch of System.out, and printStackTrace() which need to be removed - ContainerManager.getContainerStatus does not imply much - ant state except non-final states will show up as RUNNING. What should be verified is the internal state of the Container - via container.getContainerState(). You'll need access to the nmcontext in the test to access this When testing, did you try different users with the LCE ? was (Author: sseth): Couple more comments. Hopefully the last set. - //queue deletions here, rather than NM init? - this comment isn't required anymore. - The log messages in the delete method should not suppress the exception ( LOG.warn(Failed to delete localDir: + localDir, e);) - There's a log message in the test which says named nobody - this likely needs to be user - Bunch of System.out, and printStackTrace() which need to be removed - ContainerManager.getContainerStatus does not imply much - ant state except non-final states will show up as RUNNING. What should be verified is the internal state of the Container - via container.getContainerState(). You'll need access to the nmcontext in the test to access this When testing, did you try different users with the LCE ? Ensure/confirm that the NodeManager cleans up local-dirs on restart --- Key: YARN-71 URL: https://issues.apache.org/jira/browse/YARN-71 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Reporter: Vinod Kumar Vavilapalli Assignee: Xuan Gong Priority: Critical Attachments: YARN-71.10.patch, YARN-71.11.patch, YARN-71.12.patch, YARN-71.1.patch, YARN-71.2.patch, YARN-71.3.patch, YARN.71.4.patch, YARN-71.5.patch, YARN-71.6.patch, YARN-71.7.patch, YARN-71.8.patch, YARN-71.9.patch We have to make sure that NodeManagers cleanup their local files on restart. It may already be working like that in which case we should have tests validating this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-479) NM retry behavior for connection to RM should be similar for lost heartbeats
[ https://issues.apache.org/jira/browse/YARN-479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jian he updated YARN-479: - Attachment: YARN-479.3.patch NM retry behavior for connection to RM should be similar for lost heartbeats Key: YARN-479 URL: https://issues.apache.org/jira/browse/YARN-479 Project: Hadoop YARN Issue Type: Bug Reporter: Hitesh Shah Assignee: jian he Attachments: YARN-479.1.patch, YARN-479.2.patch, YARN-479.3.patch Regardless of connection loss at the start or at an intermediate point, NM's retry behavior to the RM should follow the same flow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-479) NM retry behavior for connection to RM should be similar for lost heartbeats
[
https://issues.apache.org/jira/browse/YARN-479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609818#comment-13609818
]
Hadoop QA commented on YARN-479:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574940/YARN-479.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/568//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/568//console
This message is automatically generated.
NM retry behavior for connection to RM should be similar for lost heartbeats
Key: YARN-479
URL: https://issues.apache.org/jira/browse/YARN-479
Project: Hadoop YARN
Issue Type: Bug
Reporter: Hitesh Shah
Assignee: jian he
Attachments: YARN-479.1.patch, YARN-479.2.patch, YARN-479.3.patch
Regardless of connection loss at the start or at an intermediate point, NM's
retry behavior to the RM should follow the same flow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-497) yarn unmanaged-am launcher jar does not define a main class in its manifest
[ https://issues.apache.org/jira/browse/YARN-497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah reassigned YARN-497: Assignee: Hitesh Shah yarn unmanaged-am launcher jar does not define a main class in its manifest --- Key: YARN-497 URL: https://issues.apache.org/jira/browse/YARN-497 Project: Hadoop YARN Issue Type: Bug Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Minor Labels: usability The jar should have a mainClass defined to make it easier to use with the hadoop jar command. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-498) Unmanaged AM launcher does not set Container Id into env for an AM
[ https://issues.apache.org/jira/browse/YARN-498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah reassigned YARN-498: Assignee: Hitesh Shah Unmanaged AM launcher does not set Container Id into env for an AM -- Key: YARN-498 URL: https://issues.apache.org/jira/browse/YARN-498 Project: Hadoop YARN Issue Type: Bug Reporter: Hitesh Shah Assignee: Hitesh Shah Currently, it only sets the app attempt id which is really not required as AMs are only expected to extract it from the container id. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-439) Flatten NodeHeartbeatResponse
[ https://issues.apache.org/jira/browse/YARN-439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609827#comment-13609827 ] Siddharth Seth commented on YARN-439: - Looks good, except for the javadoc warnings. Flatten NodeHeartbeatResponse - Key: YARN-439 URL: https://issues.apache.org/jira/browse/YARN-439 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Xuan Gong Attachments: YARN-439.1.patch, YARN-439.2.patch, YARN-439.3.patch NodeheartbeatResponse has another wrapper HeartbeatResponse under it, which can be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-417) Create AMRMClient wrapper that provides asynchronous callbacks
[ https://issues.apache.org/jira/browse/YARN-417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609837#comment-13609837 ] Chris Nauroth commented on YARN-417: Thanks, Sandy. The explanation makes sense. It seems like this warrants a separate jira. Is it even valid to run the distributed shell AM with no containers? I'm not sure. Create AMRMClient wrapper that provides asynchronous callbacks -- Key: YARN-417 URL: https://issues.apache.org/jira/browse/YARN-417 Project: Hadoop YARN Issue Type: Sub-task Components: api, applications Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: AMRMClientAsync-1.java, AMRMClientAsync.java, YARN-417-1.patch, YARN-417-2.patch, YARN-417-3.patch, YARN-417-4.patch, YARN-417-4.patch, YARN-417-5.patch, YARN-417-6.patch, YARN-417.patch, YarnAppMaster.java, YarnAppMasterListener.java Writing AMs would be easier for some if they did not have to handle heartbeating to the RM on their own. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-499) On failure, include last n lines of container logs in diagnostics
Sandy Ryza created YARN-499: --- Summary: On failure, include last n lines of container logs in diagnostics Key: YARN-499 URL: https://issues.apache.org/jira/browse/YARN-499 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza When a container fails, the only way to diagnose it is to look at the logs. ContainerStatuses include a diagnostic string that is reported back to the resource manager by the node manager. Currently in MR2 I believe whatever is sent to the task's standard out is added to the diagnostics string, but for MR standard out is redirected to a file called stdout. In MR1, this string was populated with the last few lines of the task's stdout file, and got printed to the console, allowing for easy debugging. Handling this would help to soothe the infuriating problem of an AM dying for a mysterious reason before setting a tracking URL (MAPREDUCE-3688). This could be done in one of two ways. * Use tee to send MR's standard out to both the stdout file and standard out. This requires modifying ShellCmdExecutor to roll what it reads in, as we wouldn't want to be storing the entire task log in NM memory. * Read the task's log files. This would require standardizing or making the container log files configurable. Right now the log files are determined in userland and all that is YARN is aware of the log directory. Does this present any issues I'm not considering? If so it this might only be needed for AMs? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-499) On container failure, include last n lines of logs in diagnostics
[ https://issues.apache.org/jira/browse/YARN-499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-499: Summary: On container failure, include last n lines of logs in diagnostics (was: On failure, include last n lines of container logs in diagnostics) On container failure, include last n lines of logs in diagnostics - Key: YARN-499 URL: https://issues.apache.org/jira/browse/YARN-499 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza When a container fails, the only way to diagnose it is to look at the logs. ContainerStatuses include a diagnostic string that is reported back to the resource manager by the node manager. Currently in MR2 I believe whatever is sent to the task's standard out is added to the diagnostics string, but for MR standard out is redirected to a file called stdout. In MR1, this string was populated with the last few lines of the task's stdout file, and got printed to the console, allowing for easy debugging. Handling this would help to soothe the infuriating problem of an AM dying for a mysterious reason before setting a tracking URL (MAPREDUCE-3688). This could be done in one of two ways. * Use tee to send MR's standard out to both the stdout file and standard out. This requires modifying ShellCmdExecutor to roll what it reads in, as we wouldn't want to be storing the entire task log in NM memory. * Read the task's log files. This would require standardizing or making the container log files configurable. Right now the log files are determined in userland and all that is YARN is aware of the log directory. Does this present any issues I'm not considering? If so it this might only be needed for AMs? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-497) yarn unmanaged-am launcher jar does not define a main class in its manifest
[ https://issues.apache.org/jira/browse/YARN-497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated YARN-497: - Attachment: YARN-497.1.patch Trivial patch. yarn unmanaged-am launcher jar does not define a main class in its manifest --- Key: YARN-497 URL: https://issues.apache.org/jira/browse/YARN-497 Project: Hadoop YARN Issue Type: Bug Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Minor Labels: usability Attachments: YARN-497.1.patch The jar should have a mainClass defined to make it easier to use with the hadoop jar command. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-497) yarn unmanaged-am launcher jar does not define a main class in its manifest
[
https://issues.apache.org/jira/browse/YARN-497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609921#comment-13609921
]
Hadoop QA commented on YARN-497:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574954/YARN-497.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/569//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/569//console
This message is automatically generated.
yarn unmanaged-am launcher jar does not define a main class in its manifest
---
Key: YARN-497
URL: https://issues.apache.org/jira/browse/YARN-497
Project: Hadoop YARN
Issue Type: Bug
Reporter: Hitesh Shah
Assignee: Hitesh Shah
Priority: Minor
Labels: usability
Attachments: YARN-497.1.patch
The jar should have a mainClass defined to make it easier to use with the
hadoop jar command.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Moved] (YARN-500) ResourceManager webapp is using next port if configured port is already in use
[ https://issues.apache.org/jira/browse/YARN-500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishan Shetty moved MAPREDUCE-5091 to YARN-500: --- Component/s: (was: resourcemanager) resourcemanager Affects Version/s: (was: 2.0.2-alpha) (was: 2.0.1-alpha) 2.0.2-alpha 2.0.1-alpha Key: YARN-500 (was: MAPREDUCE-5091) Project: Hadoop YARN (was: Hadoop Map/Reduce) ResourceManager webapp is using next port if configured port is already in use -- Key: YARN-500 URL: https://issues.apache.org/jira/browse/YARN-500 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.0.1-alpha, 2.0.2-alpha Reporter: Nishan Shetty -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-500) ResourceManager webapp is using next port if configured port is already in use
[ https://issues.apache.org/jira/browse/YARN-500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609927#comment-13609927 ] Nishan Shetty commented on YARN-500: Same is for Nodemanager webapp ResourceManager webapp is using next port if configured port is already in use -- Key: YARN-500 URL: https://issues.apache.org/jira/browse/YARN-500 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.0.2-alpha, 2.0.1-alpha Reporter: Nishan Shetty -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-498) Unmanaged AM launcher does not set Container Id into env for an AM
[ https://issues.apache.org/jira/browse/YARN-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609932#comment-13609932 ] Hitesh Shah commented on YARN-498: -- Found other issues in the unmanaged am launcher. - More env vars are not set - It does not handle a failed AM properly - just hangs waiting for it to complete even after it has detected that the AM process has finished. Unmanaged AM launcher does not set Container Id into env for an AM -- Key: YARN-498 URL: https://issues.apache.org/jira/browse/YARN-498 Project: Hadoop YARN Issue Type: Bug Reporter: Hitesh Shah Assignee: Hitesh Shah Attachments: YARN-498.wip.patch Currently, it only sets the app attempt id which is really not required as AMs are only expected to extract it from the container id. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-498) Unmanaged AM launcher does not set Container Id into env for an AM
[ https://issues.apache.org/jira/browse/YARN-498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated YARN-498: - Attachment: YARN-498.wip.patch Attaching a wip patch. Tests pending. Unmanaged AM launcher does not set Container Id into env for an AM -- Key: YARN-498 URL: https://issues.apache.org/jira/browse/YARN-498 Project: Hadoop YARN Issue Type: Bug Reporter: Hitesh Shah Assignee: Hitesh Shah Attachments: YARN-498.wip.patch Currently, it only sets the app attempt id which is really not required as AMs are only expected to extract it from the container id. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
