[jira] [Updated] (YARN-779) AMRMClient should clean up dangling unsatisfied request
[
https://issues.apache.org/jira/browse/YARN-779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maysam Yabandeh updated YARN-779:
-
Attachment: YARN-779.patch
Fixed the test case with correct setting for relaxedLocality.
AMRMClient should clean up dangling unsatisfied request
---
Key: YARN-779
URL: https://issues.apache.org/jira/browse/YARN-779
Project: Hadoop YARN
Issue Type: Bug
Components: client
Affects Versions: 2.0.4-alpha
Reporter: Alejandro Abdelnur
Priority: Critical
Attachments: YARN-779.patch, YARN-779.patch
If an AMRMClient allocates a ContainerRequest for 10 containers in node1 or
node2 is placed (assuming a single rack) the resulting ResourceRequests will
be
{code}
location - containers
-
node1- 10
node2- 10
rack - 10
ANY - 10
{code}
Assuming 5 containers are allocated in node1 and 5 containers are allocated
in node2, the following ResourceRequests will be outstanding on the RM.
{code}
location - containers
-
node1- 5
node2- 5
{code}
If the AMMRClient does a new ContainerRequest allocation, this time for 5
containers in node3, the resulting outstanding ResourceRequests on the RM
will be:
{code}
location - containers
-
node1- 5
node2- 5
node3- 5
rack - 5
ANY - 5
{code}
At this point, the scheduler may assign 5 containers to node1 and it will
never assign the 5 containers node3 asked for.
AMRMClient should keep track of the outstanding allocations counts per
ContainerRequest and when gets to zero it should update the the RACK/ANY
decrementing the dangling requests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-779) AMRMClient should clean up dangling unsatisfied request
[
https://issues.apache.org/jira/browse/YARN-779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749341#comment-13749341
]
Maysam Yabandeh commented on YARN-779:
--
I am thinking perhaps we can solve the problem without needing a complete
change in the API. Since we are using Protocol Buffers, we can freely add new
fields to the message.
What we need is a way to express in a set of ResourceRequests the disjunction
between the requested containers in ContainerRequest. For that we can use a
locally unique resourceRequestId generated by the AMRMClientImpl.java. For
example if application requires one container in (node1 || node2),
#addContainerRequest decomposes it into two ResourceRequests but tagged with
the same resourceRequestId.
* ResourceRequest(node1, id1234);
* ResourceRequest(node2, id1234);
Later, when the ResourceManager services a ResourceRequest with ID id1234, it
can update all other corresponding ResourceRequests from the same application
with the same ID of id1234. Thanks to Protocol Buffers, there will be no
inconsistency between old/new clients with new/old servers.
Feedbacks are appreciated.
AMRMClient should clean up dangling unsatisfied request
---
Key: YARN-779
URL: https://issues.apache.org/jira/browse/YARN-779
Project: Hadoop YARN
Issue Type: Bug
Components: client
Affects Versions: 2.0.4-alpha
Reporter: Alejandro Abdelnur
Priority: Critical
Attachments: YARN-779.patch, YARN-779.patch
If an AMRMClient allocates a ContainerRequest for 10 containers in node1 or
node2 is placed (assuming a single rack) the resulting ResourceRequests will
be
{code}
location - containers
-
node1- 10
node2- 10
rack - 10
ANY - 10
{code}
Assuming 5 containers are allocated in node1 and 5 containers are allocated
in node2, the following ResourceRequests will be outstanding on the RM.
{code}
location - containers
-
node1- 5
node2- 5
{code}
If the AMMRClient does a new ContainerRequest allocation, this time for 5
containers in node3, the resulting outstanding ResourceRequests on the RM
will be:
{code}
location - containers
-
node1- 5
node2- 5
node3- 5
rack - 5
ANY - 5
{code}
At this point, the scheduler may assign 5 containers to node1 and it will
never assign the 5 containers node3 asked for.
AMRMClient should keep track of the outstanding allocations counts per
ContainerRequest and when gets to zero it should update the the RACK/ANY
decrementing the dangling requests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-905) Add state filters to nodes CLI
[ https://issues.apache.org/jira/browse/YARN-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749352#comment-13749352 ] Hudson commented on YARN-905: - FAILURE: Integrated in Hadoop-Yarn-trunk #311 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/311/]) YARN-905. Add state filters to nodes CLI (Wei Yan via Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1517083) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java Add state filters to nodes CLI -- Key: YARN-905 URL: https://issues.apache.org/jira/browse/YARN-905 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Wei Yan Attachments: Yarn-905.patch, YARN-905.patch, YARN-905.patch It would be helpful for the nodes CLI to have a node-states option that allows it to return nodes that are not just in the RUNNING state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1085) Yarn and MRv2 should do HTTP client authentication in kerberos setup.
[ https://issues.apache.org/jira/browse/YARN-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749355#comment-13749355 ] Hudson commented on YARN-1085: -- FAILURE: Integrated in Hadoop-Yarn-trunk #311 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/311/]) YARN-1085. Modified YARN and MR2 web-apps to do HTTP authentication in secure setup with kerberos. Contributed by Omkar Vinit Joshi. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1517101) * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryClientService.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java Yarn and MRv2 should do HTTP client authentication in kerberos setup. - Key: YARN-1085 URL: https://issues.apache.org/jira/browse/YARN-1085 Project: Hadoop YARN Issue Type: Task Components: nodemanager, resourcemanager Reporter: Jaimin D Jetly Assignee: Omkar Vinit Joshi Priority: Blocker Labels: security Fix For: 2.1.1-beta Attachments: YARN-1085.20130820.1.patch, YARN-1085.20130823.1.patch, YARN-1085.20130823.2.patch, YARN-1085.20130823.3.patch In kerberos setup it's expected for a http client to authenticate to kerberos before allowing user to browse any information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-707) Add user info in the YARN ClientToken
[ https://issues.apache.org/jira/browse/YARN-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749353#comment-13749353 ] Hudson commented on YARN-707: - FAILURE: Integrated in Hadoop-Yarn-trunk #311 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/311/]) Revert MAPREDUCE-5475 and YARN-707 (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1517097) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRClientService.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/client/ClientToAMTokenIdentifier.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestClientToAMTokens.java YARN-707. Add user info in the YARN ClientToken. Contributed by Vinod Kumar Vavilapalli (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1517073) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/client/ClientToAMTokenIdentifier.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestClientToAMTokens.java Add user info in the YARN ClientToken - Key: YARN-707 URL: https://issues.apache.org/jira/browse/YARN-707 Project: Hadoop YARN Issue Type: Improvement Reporter: Bikas Saha Assignee: Vinod Kumar Vavilapalli Fix For: 3.0.0, 2.1.1-beta Attachments: YARN-707-20130822.txt If user info is present in the client token then it can be used to do limited authz in the AM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-707) Add user info in the YARN ClientToken
[ https://issues.apache.org/jira/browse/YARN-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749381#comment-13749381 ] Hudson commented on YARN-707: - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1501 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1501/]) Revert MAPREDUCE-5475 and YARN-707 (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1517097) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRClientService.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/client/ClientToAMTokenIdentifier.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestClientToAMTokens.java YARN-707. Add user info in the YARN ClientToken. Contributed by Vinod Kumar Vavilapalli (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1517073) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/client/ClientToAMTokenIdentifier.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestClientToAMTokens.java Add user info in the YARN ClientToken - Key: YARN-707 URL: https://issues.apache.org/jira/browse/YARN-707 Project: Hadoop YARN Issue Type: Improvement Reporter: Bikas Saha Assignee: Vinod Kumar Vavilapalli Fix For: 3.0.0, 2.1.1-beta Attachments: YARN-707-20130822.txt If user info is present in the client token then it can be used to do limited authz in the AM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1085) Yarn and MRv2 should do HTTP client authentication in kerberos setup.
[ https://issues.apache.org/jira/browse/YARN-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749383#comment-13749383 ] Hudson commented on YARN-1085: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1501 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1501/]) YARN-1085. Modified YARN and MR2 web-apps to do HTTP authentication in secure setup with kerberos. Contributed by Omkar Vinit Joshi. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1517101) * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryClientService.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java Yarn and MRv2 should do HTTP client authentication in kerberos setup. - Key: YARN-1085 URL: https://issues.apache.org/jira/browse/YARN-1085 Project: Hadoop YARN Issue Type: Task Components: nodemanager, resourcemanager Reporter: Jaimin D Jetly Assignee: Omkar Vinit Joshi Priority: Blocker Labels: security Fix For: 2.1.1-beta Attachments: YARN-1085.20130820.1.patch, YARN-1085.20130823.1.patch, YARN-1085.20130823.2.patch, YARN-1085.20130823.3.patch In kerberos setup it's expected for a http client to authenticate to kerberos before allowing user to browse any information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-905) Add state filters to nodes CLI
[ https://issues.apache.org/jira/browse/YARN-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749380#comment-13749380 ] Hudson commented on YARN-905: - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1501 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1501/]) YARN-905. Add state filters to nodes CLI (Wei Yan via Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1517083) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java Add state filters to nodes CLI -- Key: YARN-905 URL: https://issues.apache.org/jira/browse/YARN-905 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Wei Yan Attachments: Yarn-905.patch, YARN-905.patch, YARN-905.patch It would be helpful for the nodes CLI to have a node-states option that allows it to return nodes that are not just in the RUNNING state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1067) AMRMClient heartbeat interval should not be static
[ https://issues.apache.org/jira/browse/YARN-1067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maysam Yabandeh updated YARN-1067: -- Attachment: YARN-1067.patch The attached patch takes the first stab. Comments are appreciated. AMRMClient heartbeat interval should not be static -- Key: YARN-1067 URL: https://issues.apache.org/jira/browse/YARN-1067 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.0.5-alpha Reporter: Siddharth Seth Attachments: YARN-1067.patch The heartbeat interval can be modified dynamically - more often when there are pending requests, and toned down when the heartbeat is solving no purpose other than a ping. There's a couple of jiras which are trying to change the scheduling loop - at which point this becomes useful. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-707) Add user info in the YARN ClientToken
[ https://issues.apache.org/jira/browse/YARN-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749397#comment-13749397 ] Hudson commented on YARN-707: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1528 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1528/]) Revert MAPREDUCE-5475 and YARN-707 (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1517097) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRClientService.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/client/ClientToAMTokenIdentifier.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestClientToAMTokens.java YARN-707. Add user info in the YARN ClientToken. Contributed by Vinod Kumar Vavilapalli (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1517073) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/client/ClientToAMTokenIdentifier.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestClientToAMTokens.java Add user info in the YARN ClientToken - Key: YARN-707 URL: https://issues.apache.org/jira/browse/YARN-707 Project: Hadoop YARN Issue Type: Improvement Reporter: Bikas Saha Assignee: Vinod Kumar Vavilapalli Fix For: 3.0.0, 2.1.1-beta Attachments: YARN-707-20130822.txt If user info is present in the client token then it can be used to do limited authz in the AM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-905) Add state filters to nodes CLI
[ https://issues.apache.org/jira/browse/YARN-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749396#comment-13749396 ] Hudson commented on YARN-905: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1528 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1528/]) YARN-905. Add state filters to nodes CLI (Wei Yan via Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1517083) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java Add state filters to nodes CLI -- Key: YARN-905 URL: https://issues.apache.org/jira/browse/YARN-905 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Wei Yan Attachments: Yarn-905.patch, YARN-905.patch, YARN-905.patch It would be helpful for the nodes CLI to have a node-states option that allows it to return nodes that are not just in the RUNNING state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1085) Yarn and MRv2 should do HTTP client authentication in kerberos setup.
[ https://issues.apache.org/jira/browse/YARN-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749399#comment-13749399 ] Hudson commented on YARN-1085: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1528 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1528/]) YARN-1085. Modified YARN and MR2 web-apps to do HTTP authentication in secure setup with kerberos. Contributed by Omkar Vinit Joshi. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1517101) * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryClientService.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java Yarn and MRv2 should do HTTP client authentication in kerberos setup. - Key: YARN-1085 URL: https://issues.apache.org/jira/browse/YARN-1085 Project: Hadoop YARN Issue Type: Task Components: nodemanager, resourcemanager Reporter: Jaimin D Jetly Assignee: Omkar Vinit Joshi Priority: Blocker Labels: security Fix For: 2.1.1-beta Attachments: YARN-1085.20130820.1.patch, YARN-1085.20130823.1.patch, YARN-1085.20130823.2.patch, YARN-1085.20130823.3.patch In kerberos setup it's expected for a http client to authenticate to kerberos before allowing user to browse any information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1094) RM restart throws Null pointer Exception in Secure Env
[ https://issues.apache.org/jira/browse/YARN-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1094: -- Attachment: YARN-1094-20130824.txt Here's a patch that fixes this bug. - Moved delegationTokenRenewer's start to be explicit and before the state-store starts. - Made GetDelegationTokenRequest.newInstance as static. This was a pre-existing bug! - Made fixes to consistently use RMDelegationTokenRenewer only in secure mode - Some cosmetic changes to call tokenRenewer as more specifically delegationTokenRenewer TestRMRestart.testDelegationTokenRestoredInDelegationTokenRenewer fails with the same NPE without the code changes and passes with. Also tested this on a single node secure setup where I first reproduced the NPE easily and verified that RM restart works as expected after the patch. RM restart throws Null pointer Exception in Secure Env -- Key: YARN-1094 URL: https://issues.apache.org/jira/browse/YARN-1094 Project: Hadoop YARN Issue Type: Bug Environment: secure env Reporter: yeshavora Assignee: Vinod Kumar Vavilapalli Priority: Blocker Attachments: YARN-1094-20130824.txt Enable rmrestart feature And restart Resorce Manager while a job is running. Resorce Manager fails to start with below error 2013-08-23 17:57:40,705 INFO resourcemanager.RMAppManager (RMAppManager.java:recover(370)) - Recovering application application_1377280618693_0001 2013-08-23 17:57:40,763 ERROR resourcemanager.ResourceManager (ResourceManager.java:serviceStart(617)) - Failed to load/recover state java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.setTimerForTokenRenewal(DelegationTokenRenewer.java:371) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addApplication(DelegationTokenRenewer.java:307) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:371) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:819) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:613) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:832) 2013-08-23 17:57:40,766 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1094) RM restart throws Null pointer Exception in Secure Env
[
https://issues.apache.org/jira/browse/YARN-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749466#comment-13749466
]
Hadoop QA commented on YARN-1094:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12599810/YARN-1094-20130824.1.txt
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1756//console
This message is automatically generated.
RM restart throws Null pointer Exception in Secure Env
--
Key: YARN-1094
URL: https://issues.apache.org/jira/browse/YARN-1094
Project: Hadoop YARN
Issue Type: Bug
Environment: secure env
Reporter: yeshavora
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Attachments: YARN-1094-20130824.1.txt, YARN-1094-20130824.txt
Enable rmrestart feature And restart Resorce Manager while a job is running.
Resorce Manager fails to start with below error
2013-08-23 17:57:40,705 INFO resourcemanager.RMAppManager
(RMAppManager.java:recover(370)) - Recovering application
application_1377280618693_0001
2013-08-23 17:57:40,763 ERROR resourcemanager.ResourceManager
(ResourceManager.java:serviceStart(617)) - Failed to load/recover state
java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.setTimerForTokenRenewal(DelegationTokenRenewer.java:371)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addApplication(DelegationTokenRenewer.java:307)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:371)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:819)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:613)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:832)
2013-08-23 17:57:40,766 INFO util.ExitUtil (ExitUtil.java:terminate(124)) -
Exiting with status 1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1094) RM restart throws Null pointer Exception in Secure Env
[ https://issues.apache.org/jira/browse/YARN-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1094: -- Attachment: YARN-1094-20130824.1.txt Correct patch. RM restart throws Null pointer Exception in Secure Env -- Key: YARN-1094 URL: https://issues.apache.org/jira/browse/YARN-1094 Project: Hadoop YARN Issue Type: Bug Environment: secure env Reporter: yeshavora Assignee: Vinod Kumar Vavilapalli Priority: Blocker Attachments: YARN-1094-20130824.1.txt, YARN-1094-20130824.txt Enable rmrestart feature And restart Resorce Manager while a job is running. Resorce Manager fails to start with below error 2013-08-23 17:57:40,705 INFO resourcemanager.RMAppManager (RMAppManager.java:recover(370)) - Recovering application application_1377280618693_0001 2013-08-23 17:57:40,763 ERROR resourcemanager.ResourceManager (ResourceManager.java:serviceStart(617)) - Failed to load/recover state java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.setTimerForTokenRenewal(DelegationTokenRenewer.java:371) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addApplication(DelegationTokenRenewer.java:307) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:371) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:819) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:613) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:832) 2013-08-23 17:57:40,766 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1074) Clean up YARN CLI app list to show only running apps.
[
https://issues.apache.org/jira/browse/YARN-1074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749471#comment-13749471
]
Vinod Kumar Vavilapalli commented on YARN-1074:
---
TestJHSSecurity seems to be caused by YARN-1085.
I'm checking this one in.
Clean up YARN CLI app list to show only running apps.
-
Key: YARN-1074
URL: https://issues.apache.org/jira/browse/YARN-1074
Project: Hadoop YARN
Issue Type: Improvement
Components: client
Reporter: Tassapol Athiapinya
Assignee: Xuan Gong
Attachments: YARN-1074.1.patch, YARN-1074.2.patch, YARN-1074.3.patch,
YARN-1074.4.patch, YARN-1074.5.patch, YARN-1074.6.patch, YARN-1074.7.patch,
YARN-1074.8.patch
Once a user brings up YARN daemon, runs jobs, jobs will stay in output
returned by $ yarn application -list even after jobs complete already. We
want YARN command line to clean up this list. Specifically, we want to remove
applications with FINISHED state(not Final-State) or KILLED state from the
result.
{code}
[user1@host1 ~]$ yarn application -list
Total Applications:150
Application-IdApplication-Name
Application-Type User Queue State
Final-State ProgressTracking-URL
application_1374638600275_0109 Sleep job
MAPREDUCEuser1 default KILLED
KILLED 100%host1:54059
application_1374638600275_0121 Sleep job
MAPREDUCEuser1 defaultFINISHED
SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0121
application_1374638600275_0020 Sleep job
MAPREDUCEuser1 defaultFINISHED
SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0020
application_1374638600275_0038 Sleep job
MAPREDUCEuser1 default
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-1074) Clean up YARN CLI app list to show only running apps.
[
https://issues.apache.org/jira/browse/YARN-1074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli resolved YARN-1074.
---
Resolution: Fixed
Fix Version/s: 2.1.1-beta
Hadoop Flags: Reviewed
Committed this to trunk, branch-2 and branch-2.1. Thanks Xuan!
Clean up YARN CLI app list to show only running apps.
-
Key: YARN-1074
URL: https://issues.apache.org/jira/browse/YARN-1074
Project: Hadoop YARN
Issue Type: Improvement
Components: client
Reporter: Tassapol Athiapinya
Assignee: Xuan Gong
Fix For: 2.1.1-beta
Attachments: YARN-1074.1.patch, YARN-1074.2.patch, YARN-1074.3.patch,
YARN-1074.4.patch, YARN-1074.5.patch, YARN-1074.6.patch, YARN-1074.7.patch,
YARN-1074.8.patch
Once a user brings up YARN daemon, runs jobs, jobs will stay in output
returned by $ yarn application -list even after jobs complete already. We
want YARN command line to clean up this list. Specifically, we want to remove
applications with FINISHED state(not Final-State) or KILLED state from the
result.
{code}
[user1@host1 ~]$ yarn application -list
Total Applications:150
Application-IdApplication-Name
Application-Type User Queue State
Final-State ProgressTracking-URL
application_1374638600275_0109 Sleep job
MAPREDUCEuser1 default KILLED
KILLED 100%host1:54059
application_1374638600275_0121 Sleep job
MAPREDUCEuser1 defaultFINISHED
SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0121
application_1374638600275_0020 Sleep job
MAPREDUCEuser1 defaultFINISHED
SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0020
application_1374638600275_0038 Sleep job
MAPREDUCEuser1 default
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1094) RM restart throws Null pointer Exception in Secure Env
[ https://issues.apache.org/jira/browse/YARN-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tassapol Athiapinya updated YARN-1094: -- Attachment: YARN-1094-20130824.1.txt Attaching YARN-1094-20130824.1.txt again (no changes) to kick off another Jenkins build RM restart throws Null pointer Exception in Secure Env -- Key: YARN-1094 URL: https://issues.apache.org/jira/browse/YARN-1094 Project: Hadoop YARN Issue Type: Bug Environment: secure env Reporter: yeshavora Assignee: Vinod Kumar Vavilapalli Priority: Blocker Attachments: YARN-1094-20130824.1.txt, YARN-1094-20130824.1.txt, YARN-1094-20130824.txt Enable rmrestart feature And restart Resorce Manager while a job is running. Resorce Manager fails to start with below error 2013-08-23 17:57:40,705 INFO resourcemanager.RMAppManager (RMAppManager.java:recover(370)) - Recovering application application_1377280618693_0001 2013-08-23 17:57:40,763 ERROR resourcemanager.ResourceManager (ResourceManager.java:serviceStart(617)) - Failed to load/recover state java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.setTimerForTokenRenewal(DelegationTokenRenewer.java:371) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addApplication(DelegationTokenRenewer.java:307) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:371) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:819) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:613) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:832) 2013-08-23 17:57:40,766 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1094) RM restart throws Null pointer Exception in Secure Env
[ https://issues.apache.org/jira/browse/YARN-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1094: -- Attachment: YARN-1094-20130824.2.txt Seems like a merge conflict. Uploading a new patch. RM restart throws Null pointer Exception in Secure Env -- Key: YARN-1094 URL: https://issues.apache.org/jira/browse/YARN-1094 Project: Hadoop YARN Issue Type: Bug Environment: secure env Reporter: yeshavora Assignee: Vinod Kumar Vavilapalli Priority: Blocker Attachments: YARN-1094-20130824.1.txt, YARN-1094-20130824.1.txt, YARN-1094-20130824.2.txt, YARN-1094-20130824.txt Enable rmrestart feature And restart Resorce Manager while a job is running. Resorce Manager fails to start with below error 2013-08-23 17:57:40,705 INFO resourcemanager.RMAppManager (RMAppManager.java:recover(370)) - Recovering application application_1377280618693_0001 2013-08-23 17:57:40,763 ERROR resourcemanager.ResourceManager (ResourceManager.java:serviceStart(617)) - Failed to load/recover state java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.setTimerForTokenRenewal(DelegationTokenRenewer.java:371) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addApplication(DelegationTokenRenewer.java:307) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:371) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:819) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:613) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:832) 2013-08-23 17:57:40,766 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1074) Clean up YARN CLI app list to show only running apps.
[
https://issues.apache.org/jira/browse/YARN-1074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749475#comment-13749475
]
Hudson commented on YARN-1074:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #4322 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/4322/])
YARN-1074. Cleaned up YARN CLI application list to only display running
applications by default. Contributed by Xuan Gong. (vinodkv:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1517196)
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetApplicationsRequest.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsRequestPBImpl.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
Clean up YARN CLI app list to show only running apps.
-
Key: YARN-1074
URL: https://issues.apache.org/jira/browse/YARN-1074
Project: Hadoop YARN
Issue Type: Improvement
Components: client
Reporter: Tassapol Athiapinya
Assignee: Xuan Gong
Fix For: 2.1.1-beta
Attachments: YARN-1074.1.patch, YARN-1074.2.patch, YARN-1074.3.patch,
YARN-1074.4.patch, YARN-1074.5.patch, YARN-1074.6.patch, YARN-1074.7.patch,
YARN-1074.8.patch
Once a user brings up YARN daemon, runs jobs, jobs will stay in output
returned by $ yarn application -list even after jobs complete already. We
want YARN command line to clean up this list. Specifically, we want to remove
applications with FINISHED state(not Final-State) or KILLED state from the
result.
{code}
[user1@host1 ~]$ yarn application -list
Total Applications:150
Application-IdApplication-Name
Application-Type User Queue State
Final-State ProgressTracking-URL
application_1374638600275_0109 Sleep job
MAPREDUCEuser1 default KILLED
KILLED 100%host1:54059
application_1374638600275_0121 Sleep job
MAPREDUCEuser1 defaultFINISHED
SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0121
application_1374638600275_0020 Sleep job
MAPREDUCEuser1 defaultFINISHED
SUCCEEDED 100% host1:19888/jobhistory/job/job_1374638600275_0020
application_1374638600275_0038 Sleep job
MAPREDUCEuser1 default
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1094) RM restart throws Null pointer Exception in Secure Env
[ https://issues.apache.org/jira/browse/YARN-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749479#comment-13749479 ] Jian He commented on YARN-1094: --- bq. TestRMRestart.testDelegationTokenRestoredInDelegationTokenRenewer fails with the same NPE without the code changes and passes with. my bad, should catch this in the first place One comment: addService(delegationTokenRenewer) is called in serviceInit, and then explicitly started in serviceStart, which may cause it start twice ? RM restart throws Null pointer Exception in Secure Env -- Key: YARN-1094 URL: https://issues.apache.org/jira/browse/YARN-1094 Project: Hadoop YARN Issue Type: Bug Environment: secure env Reporter: yeshavora Assignee: Vinod Kumar Vavilapalli Priority: Blocker Attachments: YARN-1094-20130824.1.txt, YARN-1094-20130824.1.txt, YARN-1094-20130824.2.txt, YARN-1094-20130824.txt Enable rmrestart feature And restart Resorce Manager while a job is running. Resorce Manager fails to start with below error 2013-08-23 17:57:40,705 INFO resourcemanager.RMAppManager (RMAppManager.java:recover(370)) - Recovering application application_1377280618693_0001 2013-08-23 17:57:40,763 ERROR resourcemanager.ResourceManager (ResourceManager.java:serviceStart(617)) - Failed to load/recover state java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.setTimerForTokenRenewal(DelegationTokenRenewer.java:371) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addApplication(DelegationTokenRenewer.java:307) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:371) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:819) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:613) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:832) 2013-08-23 17:57:40,766 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1094) RM restart throws Null pointer Exception in Secure Env
[
https://issues.apache.org/jira/browse/YARN-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749478#comment-13749478
]
Hadoop QA commented on YARN-1094:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12599812/YARN-1094-20130824.2.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1757//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1757//console
This message is automatically generated.
RM restart throws Null pointer Exception in Secure Env
--
Key: YARN-1094
URL: https://issues.apache.org/jira/browse/YARN-1094
Project: Hadoop YARN
Issue Type: Bug
Environment: secure env
Reporter: yeshavora
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Attachments: YARN-1094-20130824.1.txt, YARN-1094-20130824.1.txt,
YARN-1094-20130824.2.txt, YARN-1094-20130824.txt
Enable rmrestart feature And restart Resorce Manager while a job is running.
Resorce Manager fails to start with below error
2013-08-23 17:57:40,705 INFO resourcemanager.RMAppManager
(RMAppManager.java:recover(370)) - Recovering application
application_1377280618693_0001
2013-08-23 17:57:40,763 ERROR resourcemanager.ResourceManager
(ResourceManager.java:serviceStart(617)) - Failed to load/recover state
java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.setTimerForTokenRenewal(DelegationTokenRenewer.java:371)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addApplication(DelegationTokenRenewer.java:307)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:371)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:819)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:613)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:832)
2013-08-23 17:57:40,766 INFO util.ExitUtil (ExitUtil.java:terminate(124)) -
Exiting with status 1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1094) RM restart throws Null pointer Exception in Secure Env
[ https://issues.apache.org/jira/browse/YARN-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749494#comment-13749494 ] Vinod Kumar Vavilapalli commented on YARN-1094: --- bq. addService(delegationTokenRenewer) is called in serviceInit, and then explicitly started in serviceStart, which may cause it start twice ? Yes, I consciously left it like that. Service-start is idempotent, the second start is ignored. RM restart throws Null pointer Exception in Secure Env -- Key: YARN-1094 URL: https://issues.apache.org/jira/browse/YARN-1094 Project: Hadoop YARN Issue Type: Bug Environment: secure env Reporter: yeshavora Assignee: Vinod Kumar Vavilapalli Priority: Blocker Attachments: YARN-1094-20130824.1.txt, YARN-1094-20130824.1.txt, YARN-1094-20130824.2.txt, YARN-1094-20130824.txt Enable rmrestart feature And restart Resorce Manager while a job is running. Resorce Manager fails to start with below error 2013-08-23 17:57:40,705 INFO resourcemanager.RMAppManager (RMAppManager.java:recover(370)) - Recovering application application_1377280618693_0001 2013-08-23 17:57:40,763 ERROR resourcemanager.ResourceManager (ResourceManager.java:serviceStart(617)) - Failed to load/recover state java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.setTimerForTokenRenewal(DelegationTokenRenewer.java:371) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addApplication(DelegationTokenRenewer.java:307) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:371) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:819) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:613) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:832) 2013-08-23 17:57:40,766 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1094) RM restart throws Null pointer Exception in Secure Env
[ https://issues.apache.org/jira/browse/YARN-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749496#comment-13749496 ] Arun C Murthy commented on YARN-1094: - +1, lgtm RM restart throws Null pointer Exception in Secure Env -- Key: YARN-1094 URL: https://issues.apache.org/jira/browse/YARN-1094 Project: Hadoop YARN Issue Type: Bug Environment: secure env Reporter: yeshavora Assignee: Vinod Kumar Vavilapalli Priority: Blocker Attachments: YARN-1094-20130824.1.txt, YARN-1094-20130824.1.txt, YARN-1094-20130824.2.txt, YARN-1094-20130824.txt Enable rmrestart feature And restart Resorce Manager while a job is running. Resorce Manager fails to start with below error 2013-08-23 17:57:40,705 INFO resourcemanager.RMAppManager (RMAppManager.java:recover(370)) - Recovering application application_1377280618693_0001 2013-08-23 17:57:40,763 ERROR resourcemanager.ResourceManager (ResourceManager.java:serviceStart(617)) - Failed to load/recover state java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.setTimerForTokenRenewal(DelegationTokenRenewer.java:371) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addApplication(DelegationTokenRenewer.java:307) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:371) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:819) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:613) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:832) 2013-08-23 17:57:40,766 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1094) RM restart throws Null pointer Exception in Secure Env
[ https://issues.apache.org/jira/browse/YARN-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749497#comment-13749497 ] Jian He commented on YARN-1094: --- I see, +1, lgtm. RM restart throws Null pointer Exception in Secure Env -- Key: YARN-1094 URL: https://issues.apache.org/jira/browse/YARN-1094 Project: Hadoop YARN Issue Type: Bug Environment: secure env Reporter: yeshavora Assignee: Vinod Kumar Vavilapalli Priority: Blocker Attachments: YARN-1094-20130824.1.txt, YARN-1094-20130824.1.txt, YARN-1094-20130824.2.txt, YARN-1094-20130824.txt Enable rmrestart feature And restart Resorce Manager while a job is running. Resorce Manager fails to start with below error 2013-08-23 17:57:40,705 INFO resourcemanager.RMAppManager (RMAppManager.java:recover(370)) - Recovering application application_1377280618693_0001 2013-08-23 17:57:40,763 ERROR resourcemanager.ResourceManager (ResourceManager.java:serviceStart(617)) - Failed to load/recover state java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.setTimerForTokenRenewal(DelegationTokenRenewer.java:371) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addApplication(DelegationTokenRenewer.java:307) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:371) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:819) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:613) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:832) 2013-08-23 17:57:40,766 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1094) RM restart throws Null pointer Exception in Secure Env
[ https://issues.apache.org/jira/browse/YARN-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749499#comment-13749499 ] Hudson commented on YARN-1094: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4323 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4323/]) YARN-1094. Fixed a blocker with RM restart code because of which RM crashes when try to recover an existing app. Contributed by Vinod Kumar Vavilapalli. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1517215) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetDelegationTokenRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java RM restart throws Null pointer Exception in Secure Env -- Key: YARN-1094 URL: https://issues.apache.org/jira/browse/YARN-1094 Project: Hadoop YARN Issue Type: Bug Environment: secure env Reporter: yeshavora Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 2.1.1-beta Attachments: YARN-1094-20130824.1.txt, YARN-1094-20130824.1.txt, YARN-1094-20130824.2.txt, YARN-1094-20130824.txt Enable rmrestart feature And restart Resorce Manager while a job is running. Resorce Manager fails to start with below error 2013-08-23 17:57:40,705 INFO resourcemanager.RMAppManager (RMAppManager.java:recover(370)) - Recovering application application_1377280618693_0001 2013-08-23 17:57:40,763 ERROR resourcemanager.ResourceManager (ResourceManager.java:serviceStart(617)) - Failed to load/recover state java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.setTimerForTokenRenewal(DelegationTokenRenewer.java:371) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addApplication(DelegationTokenRenewer.java:307) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:371) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:819) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:613) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:832) 2013-08-23 17:57:40,766 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-540) Race condition causing RM to potentially relaunch already unregistered AMs on RM restart
[
https://issues.apache.org/jira/browse/YARN-540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749516#comment-13749516
]
Bikas Saha commented on YARN-540:
-
What will happen if the RM failed after deleting the app from the store but
before the app pulled that information from the RM? I think the RM is not going
to recognize that app after restarting and return exception to the
finishApplicationMaster response from the running app. The app will fail. And
then it will not be re-started by the RM.
Comments about the patch itself.
Why are we sleeping before checking the value?
{code}
+ do {
+response = scheduler.finishApplicationMaster(request);
+Thread.sleep(rmPollInterval);
+ } while (response.getIsRemovedFromRMStateSore());
{code}
The state transitions are asynchronous. We cannot expect to always find the app
in the FINISHING state.
{code}
+ if (RMAppState.FINISHING.equals(rmContext.getRMApps()
+.get(applicationAttemptId.getApplicationId()).getState())) {
+return FinishApplicationMasterResponse.newInstance(true);
+ }
{code}
Can the application finish on the RM (in between 2 finishApp() requests) such
that it never gets a true response?
RMAppEventType.ATTEMPT_FINISHING should be renamed to ATTEMPT_UNREGISTERED in a
different jira.
store.removeApplication() should be in the RMAppImpl transitions (AppRemoving
and FinalTransition) instead of ApplicationMasterService and RMAppManager.
Can we pick a name that does not expose class names and impl details? eg.
isUnregistered()
{code}
getIsRemovedFromRMStateSore()
{code}
Is this possible to avoid 2 round trips to store?
{code}
+if(!fs.exists(deletePath))
+ return;
if(!fs.delete(deletePath, true)) {
throw new Exception(Failed to delete + deletePath);
{code}
There is no need for multiple code paths/transitions. It should always go from
RUNNING-APP_REMOVING. Please look at NEW-NEW_SAVING. When recovery is not
enabled we use the NullRMStateStore to ensure that the main code path in the RM
remains the same.
{code}
+ private static final class RMAppFinishingOrRemovingTransition implements
+ MultipleArcTransitionRMAppImpl, RMAppEvent, RMAppState {
+@Override
+public RMAppState transition(RMAppImpl app, RMAppEvent event) {
+ boolean isRecoveryEnabled =
+ app.conf.getBoolean(YarnConfiguration.RECOVERY_ENABLED,
+YarnConfiguration.DEFAULT_RM_RECOVERY_ENABLED);
+ if (isRecoveryEnabled) {
+LOG.info(Removing application with id + app.applicationId);
+app.rmContext.getStateStore().removeApplication(app);
+return RMAppState.APP_REMOVING;
+ } else
+new RMAppFinishingTransition().transition(app, event);
+return RMAppState.FINISHING;
{code}
Why add the APP_?
{code}
RUNNING,
+ APP_REMOVING,
{code}
Race condition causing RM to potentially relaunch already unregistered AMs on
RM restart
Key: YARN-540
URL: https://issues.apache.org/jira/browse/YARN-540
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Jian He
Assignee: Jian He
Priority: Blocker
Attachments: YARN-540.1.patch, YARN-540.2.patch, YARN-540.patch,
YARN-540.patch
When job succeeds and successfully call finishApplicationMaster, RM shutdown
and restart-dispatcher is stopped before it can process REMOVE_APP event. The
next time RM comes back, it will reload the existing state files even though
the job is succeeded
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-879) Fix tests w.r.t o.a.h.y.server.resourcemanager.Application
[ https://issues.apache.org/jira/browse/YARN-879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-879: Summary: Fix tests w.r.t o.a.h.y.server.resourcemanager.Application (was: Fix NPE in test/o.a.h.y.server.resourcemanager.Application.getResources()) Fix tests w.r.t o.a.h.y.server.resourcemanager.Application -- Key: YARN-879 URL: https://issues.apache.org/jira/browse/YARN-879 Project: Hadoop YARN Issue Type: Bug Affects Versions: 3.0.0, 2.1.0-beta Reporter: Junping Du Assignee: Junping Du Attachments: YARN-879.patch, YARN-879-v2.patch getResources() will return a list of containers that allocated by RM. However, it is now return null directly. The worse thing is: if LOG.debug is enabled, then it will definitely cause NPE exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-771) AMRMClient support for resource blacklisting
[ https://issues.apache.org/jira/browse/YARN-771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-771: Attachment: YARN-771-v1.0.patch Upload the first patch. AMRMClient support for resource blacklisting - Key: YARN-771 URL: https://issues.apache.org/jira/browse/YARN-771 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Junping Du Attachments: YARN-771-v1.0.patch After YARN-750 AMRMClient should support blacklisting via the new YARN API's -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-957) Capacity Scheduler tries to reserve the memory more than what node manager reports.
[ https://issues.apache.org/jira/browse/YARN-957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749526#comment-13749526 ] Omkar Vinit Joshi commented on YARN-957: Thanks arun.. sure will separate and raise these issues separately. Reducing the scope of this ticket to only address node manager max resource check. I will upload the patch soon. Capacity Scheduler tries to reserve the memory more than what node manager reports. --- Key: YARN-957 URL: https://issues.apache.org/jira/browse/YARN-957 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: YARN-957-20130730.1.patch, YARN-957-20130730.2.patch, YARN-957-20130730.3.patch, YARN-957-20130731.1.patch I have 2 node managers. * one with 1024 MB memory.(nm1) * second with 2048 MB memory.(nm2) I am submitting simple map reduce application with 1 mapper and one reducer with 1024mb each. The steps to reproduce this are * stop nm2 with 2048MB memory.( This I am doing to make sure that this node's heartbeat doesn't reach RM first). * now submit application. As soon as it receives first node's (nm1) heartbeat it will try to reserve memory for AM-container (2048MB). However it has only 1024MB of memory. * now start nm2 with 2048 MB memory. It hangs forever... Ideally this has two potential issues. * It should not try to reserve memory on a node manager which is never going to give requested memory. i.e. Current max capability of node manager is 1024MB but 2048MB is reserved on it. But it still does that. * Say 2048MB is reserved on nm1 but nm2 comes back with 2048MB available memory. In this case if the original request was made without any locality then scheduler should unreserve memory on nm1 and allocate requested 2048MB container on nm2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
