[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13663846#comment-13663846 ] Vinod Kumar Vavilapalli commented on YARN-710: -- If you are looking for PB only serialization, why can't you directly use the generated proto records? ResourceManager for example uses them to persist data to RMStateStore. Don't see a need for adding this specifically to the records. If you are looking for generic serialization, I'd +1 Sid's proposal to do it as a library which converts to and fro the serialization format. We shouldn't be inventing serdes for this, no use-case per-se internal to YARN itself. Add to ser/deser methods to RecordFactory - Key: YARN-710 URL: https://issues.apache.org/jira/browse/YARN-710 Project: Hadoop YARN Issue Type: Bug Components: api Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: YARN-710.patch I order to do things like AMs failover and checkpointing I need to serialize app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. Because we are wrapping/hiding the PB implementation from the APIs, we are hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-617) In unsercure mode, AM can fake resource requirements
[ https://issues.apache.org/jira/browse/YARN-617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13663872#comment-13663872 ] Hudson commented on YARN-617: - Integrated in Hadoop-trunk-Commit #3774 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3774/]) MAPREDUCE-5261. Fix issues in TestRMContainerAllocator after YARN-617. Contributed by Omkar Vinit Joshi. (Revision 1485079) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1485079 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java In unsercure mode, AM can fake resource requirements - Key: YARN-617 URL: https://issues.apache.org/jira/browse/YARN-617 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Omkar Vinit Joshi Priority: Minor Fix For: 2.0.5-beta Attachments: YARN-617.20130501.1.patch, YARN-617.20130501.patch, YARN-617.20130502.patch, YARN-617-20130507.patch, YARN-617.20130508.patch, YARN-617-20130513.patch, YARN-617-20130515.patch, YARN-617-20130516.branch-2.patch, YARN-617-20130516.trunk.patch Without security, it is impossible to completely avoid AMs faking resources. We can at the least make it as difficult as possible by using the same container tokens and the RM-NM shared key mechanism over unauthenticated RM-NM channel. In the minimum, this will avoid accidental bugs in AMs in unsecure mode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-563) Add application type to ApplicationReport
[ https://issues.apache.org/jira/browse/YARN-563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13663887#comment-13663887 ] Mayank Bansal commented on YARN-563: [~vinodkv] Thanks for review. I am updating two patches, One without mapreduce changes for review and other for jenkins. I updated all your comments. I have tested this with one node cluster as well. Thanks, Mayank Add application type to ApplicationReport -- Key: YARN-563 URL: https://issues.apache.org/jira/browse/YARN-563 Project: Hadoop YARN Issue Type: Sub-task Reporter: Thomas Weise Assignee: Mayank Bansal Attachments: YARN-563-trunk-1.patch, YARN-563-trunk-2.patch, YARN-563-trunk-3.patch, YARN-563-trunk-4.patch, YARN-563-trunk-5.patch, YARN-563-trunk-6.patch, YARN-563-trunk-7.patch, YARN-563-trunk-8.patch, YARN-563-trunk-9-review.patch This field is needed to distinguish different types of applications (app master implementations). For example, we may run applications of type XYZ in a cluster alongside MR and would like to filter applications by type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-563) Add application type to ApplicationReport
[ https://issues.apache.org/jira/browse/YARN-563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated YARN-563: --- Attachment: YARN-563-trunk-9-review.patch Add application type to ApplicationReport -- Key: YARN-563 URL: https://issues.apache.org/jira/browse/YARN-563 Project: Hadoop YARN Issue Type: Sub-task Reporter: Thomas Weise Assignee: Mayank Bansal Attachments: YARN-563-trunk-1.patch, YARN-563-trunk-2.patch, YARN-563-trunk-3.patch, YARN-563-trunk-4.patch, YARN-563-trunk-5.patch, YARN-563-trunk-6.patch, YARN-563-trunk-7.patch, YARN-563-trunk-8.patch, YARN-563-trunk-9-review.patch This field is needed to distinguish different types of applications (app master implementations). For example, we may run applications of type XYZ in a cluster alongside MR and would like to filter applications by type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-563) Add application type to ApplicationReport
[ https://issues.apache.org/jira/browse/YARN-563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated YARN-563: --- Attachment: YARN-563-trunk-9-jenkins.patch Add application type to ApplicationReport -- Key: YARN-563 URL: https://issues.apache.org/jira/browse/YARN-563 Project: Hadoop YARN Issue Type: Sub-task Reporter: Thomas Weise Assignee: Mayank Bansal Attachments: YARN-563-trunk-1.patch, YARN-563-trunk-2.patch, YARN-563-trunk-3.patch, YARN-563-trunk-4.patch, YARN-563-trunk-5.patch, YARN-563-trunk-6.patch, YARN-563-trunk-7.patch, YARN-563-trunk-8.patch, YARN-563-trunk-9-jenkins.patch, YARN-563-trunk-9-review.patch This field is needed to distinguish different types of applications (app master implementations). For example, we may run applications of type XYZ in a cluster alongside MR and would like to filter applications by type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-563) Add application type to ApplicationReport
[
https://issues.apache.org/jira/browse/YARN-563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13663902#comment-13663902
]
Hadoop QA commented on YARN-563:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584265/YARN-563-trunk-9-jenkins.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/980//console
This message is automatically generated.
Add application type to ApplicationReport
--
Key: YARN-563
URL: https://issues.apache.org/jira/browse/YARN-563
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Thomas Weise
Assignee: Mayank Bansal
Attachments: YARN-563-trunk-1.patch, YARN-563-trunk-2.patch,
YARN-563-trunk-3.patch, YARN-563-trunk-4.patch, YARN-563-trunk-5.patch,
YARN-563-trunk-6.patch, YARN-563-trunk-7.patch, YARN-563-trunk-8.patch,
YARN-563-trunk-9-jenkins.patch, YARN-563-trunk-9-review.patch
This field is needed to distinguish different types of applications (app
master implementations). For example, we may run applications of type XYZ in
a cluster alongside MR and would like to filter applications by type.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[
https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13663910#comment-13663910
]
Zhijie Shen commented on YARN-422:
--
bq. There is no need to start NMCommunicator everytime for stop/getStatus? We
can cache the one's from startContainer() ?
In the previous discussion, you suggested:
{quote}
Among the 3 choices you listed in stopContainer(), we should adopt (1) for
now as that is what MR AM's ContainerLauncher. Primarily because of the
one-connection-per-container issue. This again can and should be changed post
YARN-613.
{quote}
In ContainerLauncherImpl, the proxy closes every time when a container is
started or stopped (getStatus is not used there).
bq. All the call-backs are blocking.
IMHO, the callbacks can be either blocking or non-blocking. It depends on the
implementation of the callback interface. To be non-blocking, the
implementation can similarly schedule an event when the callbacks are invoked,
and the event will be dispatched to a thread for processing.
Add NM client library
-
Key: YARN-422
URL: https://issues.apache.org/jira/browse/YARN-422
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Zhijie Shen
Attachments: AMNMClient_Defination.txt,
AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf,
YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch, YARN-422.4.patch,
YARN-422.5.patch, YARN-422.6.patch
Create a simple wrapper over the ContainerManager protocol to provide hide
the details of the protocol implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-563) Add application type to ApplicationReport
[ https://issues.apache.org/jira/browse/YARN-563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated YARN-563: --- Attachment: YARN-563-trunk-10-jenkins.patch Add application type to ApplicationReport -- Key: YARN-563 URL: https://issues.apache.org/jira/browse/YARN-563 Project: Hadoop YARN Issue Type: Sub-task Reporter: Thomas Weise Assignee: Mayank Bansal Attachments: YARN-563-trunk-10-jenkins.patch, YARN-563-trunk-10-review.patch, YARN-563-trunk-1.patch, YARN-563-trunk-2.patch, YARN-563-trunk-3.patch, YARN-563-trunk-4.patch, YARN-563-trunk-5.patch, YARN-563-trunk-6.patch, YARN-563-trunk-7.patch, YARN-563-trunk-8.patch, YARN-563-trunk-9-jenkins.patch, YARN-563-trunk-9-review.patch This field is needed to distinguish different types of applications (app master implementations). For example, we may run applications of type XYZ in a cluster alongside MR and would like to filter applications by type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-563) Add application type to ApplicationReport
[ https://issues.apache.org/jira/browse/YARN-563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated YARN-563: --- Attachment: YARN-563-trunk-10-review.patch Add application type to ApplicationReport -- Key: YARN-563 URL: https://issues.apache.org/jira/browse/YARN-563 Project: Hadoop YARN Issue Type: Sub-task Reporter: Thomas Weise Assignee: Mayank Bansal Attachments: YARN-563-trunk-10-jenkins.patch, YARN-563-trunk-10-review.patch, YARN-563-trunk-1.patch, YARN-563-trunk-2.patch, YARN-563-trunk-3.patch, YARN-563-trunk-4.patch, YARN-563-trunk-5.patch, YARN-563-trunk-6.patch, YARN-563-trunk-7.patch, YARN-563-trunk-8.patch, YARN-563-trunk-9-jenkins.patch, YARN-563-trunk-9-review.patch This field is needed to distinguish different types of applications (app master implementations). For example, we may run applications of type XYZ in a cluster alongside MR and would like to filter applications by type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-563) Add application type to ApplicationReport
[
https://issues.apache.org/jira/browse/YARN-563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13663963#comment-13663963
]
Hadoop QA commented on YARN-563:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584274/YARN-563-trunk-10-jenkins.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/981//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/981//console
This message is automatically generated.
Add application type to ApplicationReport
--
Key: YARN-563
URL: https://issues.apache.org/jira/browse/YARN-563
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Thomas Weise
Assignee: Mayank Bansal
Attachments: YARN-563-trunk-10-jenkins.patch,
YARN-563-trunk-10-review.patch, YARN-563-trunk-1.patch,
YARN-563-trunk-2.patch, YARN-563-trunk-3.patch, YARN-563-trunk-4.patch,
YARN-563-trunk-5.patch, YARN-563-trunk-6.patch, YARN-563-trunk-7.patch,
YARN-563-trunk-8.patch, YARN-563-trunk-9-jenkins.patch,
YARN-563-trunk-9-review.patch
This field is needed to distinguish different types of applications (app
master implementations). For example, we may run applications of type XYZ in
a cluster alongside MR and would like to filter applications by type.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-617) In unsercure mode, AM can fake resource requirements
[ https://issues.apache.org/jira/browse/YARN-617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13663994#comment-13663994 ] Hudson commented on YARN-617: - Integrated in Hadoop-Yarn-trunk #217 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/217/]) MAPREDUCE-5261. Fix issues in TestRMContainerAllocator after YARN-617. Contributed by Omkar Vinit Joshi. (Revision 1485079) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1485079 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java In unsercure mode, AM can fake resource requirements - Key: YARN-617 URL: https://issues.apache.org/jira/browse/YARN-617 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Omkar Vinit Joshi Priority: Minor Fix For: 2.0.5-beta Attachments: YARN-617.20130501.1.patch, YARN-617.20130501.patch, YARN-617.20130502.patch, YARN-617-20130507.patch, YARN-617.20130508.patch, YARN-617-20130513.patch, YARN-617-20130515.patch, YARN-617-20130516.branch-2.patch, YARN-617-20130516.trunk.patch Without security, it is impossible to completely avoid AMs faking resources. We can at the least make it as difficult as possible by using the same container tokens and the RM-NM shared key mechanism over unauthenticated RM-NM channel. In the minimum, this will avoid accidental bugs in AMs in unsecure mode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-706) Race Condition in TestFSDownload
[ https://issues.apache.org/jira/browse/YARN-706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13663995#comment-13663995 ] Hudson commented on YARN-706: - Integrated in Hadoop-Yarn-trunk #217 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/217/]) YARN-706. Fixed race conditions in TestFSDownload. Contributed by Zhijie Shen. (Revision 1485028) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1485028 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestFSDownload.java Race Condition in TestFSDownload Key: YARN-706 URL: https://issues.apache.org/jira/browse/YARN-706 Project: Hadoop YARN Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.0.5-beta Attachments: YARN-706.1.patch, YARN-706.2.patch, YARN-706.3.patch, YARN-706.4.patch See the test failure in YARN-695 https://builds.apache.org/job/PreCommit-YARN-Build/957//testReport/org.apache.hadoop.yarn.util/TestFSDownload/testDownloadPatternJar/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-563) Add application type to ApplicationReport
[ https://issues.apache.org/jira/browse/YARN-563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664068#comment-13664068 ] Thomas Graves commented on YARN-563: Thanks Mayank, can you please update the web services documentation also? Similar to http://hadoop.apache.org/docs/r2.0.4-alpha/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html its in ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm Add application type to ApplicationReport -- Key: YARN-563 URL: https://issues.apache.org/jira/browse/YARN-563 Project: Hadoop YARN Issue Type: Sub-task Reporter: Thomas Weise Assignee: Mayank Bansal Attachments: YARN-563-trunk-10-jenkins.patch, YARN-563-trunk-10-review.patch, YARN-563-trunk-1.patch, YARN-563-trunk-2.patch, YARN-563-trunk-3.patch, YARN-563-trunk-4.patch, YARN-563-trunk-5.patch, YARN-563-trunk-6.patch, YARN-563-trunk-7.patch, YARN-563-trunk-8.patch, YARN-563-trunk-9-jenkins.patch, YARN-563-trunk-9-review.patch This field is needed to distinguish different types of applications (app master implementations). For example, we may run applications of type XYZ in a cluster alongside MR and would like to filter applications by type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-706) Race Condition in TestFSDownload
[ https://issues.apache.org/jira/browse/YARN-706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664073#comment-13664073 ] Hudson commented on YARN-706: - Integrated in Hadoop-Hdfs-trunk #1406 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1406/]) YARN-706. Fixed race conditions in TestFSDownload. Contributed by Zhijie Shen. (Revision 1485028) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1485028 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestFSDownload.java Race Condition in TestFSDownload Key: YARN-706 URL: https://issues.apache.org/jira/browse/YARN-706 Project: Hadoop YARN Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.0.5-beta Attachments: YARN-706.1.patch, YARN-706.2.patch, YARN-706.3.patch, YARN-706.4.patch See the test failure in YARN-695 https://builds.apache.org/job/PreCommit-YARN-Build/957//testReport/org.apache.hadoop.yarn.util/TestFSDownload/testDownloadPatternJar/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-617) In unsercure mode, AM can fake resource requirements
[ https://issues.apache.org/jira/browse/YARN-617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664072#comment-13664072 ] Hudson commented on YARN-617: - Integrated in Hadoop-Hdfs-trunk #1406 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1406/]) MAPREDUCE-5261. Fix issues in TestRMContainerAllocator after YARN-617. Contributed by Omkar Vinit Joshi. (Revision 1485079) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1485079 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java In unsercure mode, AM can fake resource requirements - Key: YARN-617 URL: https://issues.apache.org/jira/browse/YARN-617 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Omkar Vinit Joshi Priority: Minor Fix For: 2.0.5-beta Attachments: YARN-617.20130501.1.patch, YARN-617.20130501.patch, YARN-617.20130502.patch, YARN-617-20130507.patch, YARN-617.20130508.patch, YARN-617-20130513.patch, YARN-617-20130515.patch, YARN-617-20130516.branch-2.patch, YARN-617-20130516.trunk.patch Without security, it is impossible to completely avoid AMs faking resources. We can at the least make it as difficult as possible by using the same container tokens and the RM-NM shared key mechanism over unauthenticated RM-NM channel. In the minimum, this will avoid accidental bugs in AMs in unsecure mode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-617) In unsercure mode, AM can fake resource requirements
[ https://issues.apache.org/jira/browse/YARN-617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664123#comment-13664123 ] Hudson commented on YARN-617: - Integrated in Hadoop-Mapreduce-trunk #1433 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1433/]) MAPREDUCE-5261. Fix issues in TestRMContainerAllocator after YARN-617. Contributed by Omkar Vinit Joshi. (Revision 1485079) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1485079 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java In unsercure mode, AM can fake resource requirements - Key: YARN-617 URL: https://issues.apache.org/jira/browse/YARN-617 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Omkar Vinit Joshi Priority: Minor Fix For: 2.0.5-beta Attachments: YARN-617.20130501.1.patch, YARN-617.20130501.patch, YARN-617.20130502.patch, YARN-617-20130507.patch, YARN-617.20130508.patch, YARN-617-20130513.patch, YARN-617-20130515.patch, YARN-617-20130516.branch-2.patch, YARN-617-20130516.trunk.patch Without security, it is impossible to completely avoid AMs faking resources. We can at the least make it as difficult as possible by using the same container tokens and the RM-NM shared key mechanism over unauthenticated RM-NM channel. In the minimum, this will avoid accidental bugs in AMs in unsecure mode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-706) Race Condition in TestFSDownload
[ https://issues.apache.org/jira/browse/YARN-706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664124#comment-13664124 ] Hudson commented on YARN-706: - Integrated in Hadoop-Mapreduce-trunk #1433 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1433/]) YARN-706. Fixed race conditions in TestFSDownload. Contributed by Zhijie Shen. (Revision 1485028) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1485028 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestFSDownload.java Race Condition in TestFSDownload Key: YARN-706 URL: https://issues.apache.org/jira/browse/YARN-706 Project: Hadoop YARN Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.0.5-beta Attachments: YARN-706.1.patch, YARN-706.2.patch, YARN-706.3.patch, YARN-706.4.patch See the test failure in YARN-695 https://builds.apache.org/job/PreCommit-YARN-Build/957//testReport/org.apache.hadoop.yarn.util/TestFSDownload/testDownloadPatternJar/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-715) TestDistributedShell and TestUnmanagedAMLauncher are failing
[
https://issues.apache.org/jira/browse/YARN-715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664256#comment-13664256
]
Chris Nauroth commented on YARN-715:
Is this a duplicate of YARN-699?
TestDistributedShell and TestUnmanagedAMLauncher are failing
Key: YARN-715
URL: https://issues.apache.org/jira/browse/YARN-715
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.0.4-alpha
Reporter: Siddharth Seth
Assignee: Omkar Vinit Joshi
Tests are timing out. Looks like this is related to YARN-617.
{code}
2013-05-21 17:40:23,693 ERROR [IPC Server handler 0 on 54024]
containermanager.ContainerManagerImpl
(ContainerManagerImpl.java:authorizeRequest(412)) - Unauthorized request to
start container.
Expected containerId: user Found: container_1369183214008_0001_01_01
2013-05-21 17:40:23,694 ERROR [IPC Server handler 0 on 54024]
security.UserGroupInformation (UserGroupInformation.java:doAs(1492)) -
PriviledgedActionException as:user (auth:SIMPLE) cause:org.apache.hado
Expected containerId: user Found: container_1369183214008_0001_01_01
2013-05-21 17:40:23,695 INFO [IPC Server handler 0 on 54024] ipc.Server
(Server.java:run(1864)) - IPC Server handler 0 on 54024, call
org.apache.hadoop.yarn.api.ContainerManagerPB.startContainer from 10.
Expected containerId: user Found: container_1369183214008_0001_01_01
org.apache.hadoop.yarn.exceptions.YarnRemoteException: Unauthorized request
to start container.
Expected containerId: user Found: container_1369183214008_0001_01_01
at org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:43)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.authorizeRequest(ContainerManagerImpl.java:413)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainer(ContainerManagerImpl.java:440)
at
org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagerPBServiceImpl.startContainer(ContainerManagerPBServiceImpl.java:72)
at
org.apache.hadoop.yarn.proto.ContainerManager$ContainerManagerService$2.callBlockingMethod(ContainerManager.java:83)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-718) Remove RemoteUGI.getRemoteUser check from startContainer in ContainerManagerImpl as this is no longer required
Omkar Vinit Joshi created YARN-718: -- Summary: Remove RemoteUGI.getRemoteUser check from startContainer in ContainerManagerImpl as this is no longer required Key: YARN-718 URL: https://issues.apache.org/jira/browse/YARN-718 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Earlier there was a check in startContainer which was validating that RemoteUGI.getRemoteUser is same as containerId. However this check is no longer required and it should be removed. YARN-699 and YARN-715 will get fixed with this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-715) TestDistributedShell and TestUnmanagedAMLauncher are failing
[
https://issues.apache.org/jira/browse/YARN-715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664270#comment-13664270
]
Hitesh Shah commented on YARN-715:
--
[~sseth] Given the tests are timing out instead of failing, is there another
underlying problem that the unauthorized access did not get reported back to
the AM via either the NM or RM such the app hung?
TestDistributedShell and TestUnmanagedAMLauncher are failing
Key: YARN-715
URL: https://issues.apache.org/jira/browse/YARN-715
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.0.4-alpha
Reporter: Siddharth Seth
Assignee: Omkar Vinit Joshi
Tests are timing out. Looks like this is related to YARN-617.
{code}
2013-05-21 17:40:23,693 ERROR [IPC Server handler 0 on 54024]
containermanager.ContainerManagerImpl
(ContainerManagerImpl.java:authorizeRequest(412)) - Unauthorized request to
start container.
Expected containerId: user Found: container_1369183214008_0001_01_01
2013-05-21 17:40:23,694 ERROR [IPC Server handler 0 on 54024]
security.UserGroupInformation (UserGroupInformation.java:doAs(1492)) -
PriviledgedActionException as:user (auth:SIMPLE) cause:org.apache.hado
Expected containerId: user Found: container_1369183214008_0001_01_01
2013-05-21 17:40:23,695 INFO [IPC Server handler 0 on 54024] ipc.Server
(Server.java:run(1864)) - IPC Server handler 0 on 54024, call
org.apache.hadoop.yarn.api.ContainerManagerPB.startContainer from 10.
Expected containerId: user Found: container_1369183214008_0001_01_01
org.apache.hadoop.yarn.exceptions.YarnRemoteException: Unauthorized request
to start container.
Expected containerId: user Found: container_1369183214008_0001_01_01
at org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:43)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.authorizeRequest(ContainerManagerImpl.java:413)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainer(ContainerManagerImpl.java:440)
at
org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagerPBServiceImpl.startContainer(ContainerManagerPBServiceImpl.java:72)
at
org.apache.hadoop.yarn.proto.ContainerManager$ContainerManagerService$2.callBlockingMethod(ContainerManager.java:83)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-699) TestUnmanagedAMLauncher fails with: Unauthorized request to start container
[
https://issues.apache.org/jira/browse/YARN-699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli resolved YARN-699.
--
Resolution: Duplicate
YARN-715 is fixing multiple issues. Closing this as duplicate.
TestUnmanagedAMLauncher fails with: Unauthorized request to start container
---
Key: YARN-699
URL: https://issues.apache.org/jira/browse/YARN-699
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Ivan Mitic
Assignee: Ivan Mitic
Just run into this. Looks like YARN-617 regressed TestUnmanagedAMLauncher.
From the test log:
{noformat}
2013-05-19 12:39:10,631 INFO distributedshell.ApplicationMaster
(ApplicationMaster.java:run(682)) - Setting up container launch container for
containerid=container_1368992334149_0001_01_01
2013-05-19 12:39:10,647 INFO distributedshell.ApplicationMaster
(ApplicationMaster.java:run(690)) - Setting user in ContainerLaunchContext
to: ivanmi
2013-05-19 12:39:10,678 ERROR containermanager.ContainerManagerImpl
(ContainerManagerImpl.java:authorizeRequest(412)) - Unauthorized request to
start container.
Expected containerId: ivanmi Found: container_1368992334149_0001_01_01
2013-05-19 12:39:10,678 ERROR security.UserGroupInformation
(UserGroupInformation.java:doAs(1492)) - PriviledgedActionException as:ivanmi
(auth:SIMPLE) cause:org.apache.hadoop.yarn.exceptions.YarnRemoteException:
Unauthorized request to start container.
Expected containerId: ivanmi Found: container_1368992334149_0001_01_01
2013-05-19 12:39:10,678 INFO ipc.Server (Server.java:run(1864)) - IPC Server
handler 5 on 49529, call
org.apache.hadoop.yarn.api.ContainerManagerPB.startContainer from
10.120.19.109:49566: error:
org.apache.hadoop.yarn.exceptions.YarnRemoteException: Unauthorized request
to start container.
Expected containerId: ivanmi Found: container_1368992334149_0001_01_01
org.apache.hadoop.yarn.exceptions.YarnRemoteException: Unauthorized request
to start container.
Expected containerId: ivanmi Found: container_1368992334149_0001_01_01
at
org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:43)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.authorizeRequest(ContainerManagerImpl.java:413)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainer(ContainerManagerImpl.java:440)
at
org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagerPBServiceImpl.startContainer(ContainerManagerPBServiceImpl.java:72)
at
org.apache.hadoop.yarn.proto.ContainerManager$ContainerManagerService$2.callBlockingMethod(ContainerManager.java:83)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1033)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1842)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1838)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1489)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1836)
2013-05-19 12:39:10,678 INFO distributedshell.ApplicationMaster
(ApplicationMaster.java:run(761)) - Start container failed for :,
containerId=container_1368992334149_0001_01_01
{noformat}
ContainerManagerImpl expected containerId to be equal to the remote UGI and
since this was not the case, failed the authorization:
{noformat}
Unauthorized request to start container.
Expected containerId: ivanmi Found: container_1368992334149_0001_01_01
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[
https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664333#comment-13664333
]
Bikas Saha commented on YARN-392:
-
Minor nits
You would want to put ResourceRequest#ANY to be clear what any means in the
javadoc.
Can anyRequest be null. Others are checked for null but this one isnt.
Typo in comment. Duplicate non-zero.
{code}
+return
+// There must be outstanding requests at the given priority:
+anyRequest.getNumContainers() 0
+// If locality relaxation is turned off at *-level, there must be a
non-zero
+// non-zero request for the node's rack:
{code}
Make it possible to specify hard locality constraints in resource requests
--
Key: YARN-392
URL: https://issues.apache.org/jira/browse/YARN-392
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Sandy Ryza
Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch,
YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch,
YARN-392-6.patch, YARN-392-7.patch, YARN-392.patch
Currently its not possible to specify scheduling requests for specific nodes
and nowhere else. The RM automatically relaxes locality to rack and * and
assigns non-specified machines to the app.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-715) TestDistributedShell and TestUnmanagedAMLauncher are failing
[
https://issues.apache.org/jira/browse/YARN-715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664338#comment-13664338
]
Hitesh Shah commented on YARN-715:
--
Seems like a bug in DSShell that does not handle failed container launches
properly.
TestDistributedShell and TestUnmanagedAMLauncher are failing
Key: YARN-715
URL: https://issues.apache.org/jira/browse/YARN-715
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.0.4-alpha
Reporter: Siddharth Seth
Assignee: Omkar Vinit Joshi
Tests are timing out. Looks like this is related to YARN-617.
{code}
2013-05-21 17:40:23,693 ERROR [IPC Server handler 0 on 54024]
containermanager.ContainerManagerImpl
(ContainerManagerImpl.java:authorizeRequest(412)) - Unauthorized request to
start container.
Expected containerId: user Found: container_1369183214008_0001_01_01
2013-05-21 17:40:23,694 ERROR [IPC Server handler 0 on 54024]
security.UserGroupInformation (UserGroupInformation.java:doAs(1492)) -
PriviledgedActionException as:user (auth:SIMPLE) cause:org.apache.hado
Expected containerId: user Found: container_1369183214008_0001_01_01
2013-05-21 17:40:23,695 INFO [IPC Server handler 0 on 54024] ipc.Server
(Server.java:run(1864)) - IPC Server handler 0 on 54024, call
org.apache.hadoop.yarn.api.ContainerManagerPB.startContainer from 10.
Expected containerId: user Found: container_1369183214008_0001_01_01
org.apache.hadoop.yarn.exceptions.YarnRemoteException: Unauthorized request
to start container.
Expected containerId: user Found: container_1369183214008_0001_01_01
at org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:43)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.authorizeRequest(ContainerManagerImpl.java:413)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainer(ContainerManagerImpl.java:440)
at
org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagerPBServiceImpl.startContainer(ContainerManagerPBServiceImpl.java:72)
at
org.apache.hadoop.yarn.proto.ContainerManager$ContainerManagerService$2.callBlockingMethod(ContainerManager.java:83)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664339#comment-13664339 ] Bikas Saha commented on YARN-392: - btw, what are the plans for the capacity scheduler? Make it possible to specify hard locality constraints in resource requests -- Key: YARN-392 URL: https://issues.apache.org/jira/browse/YARN-392 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Sandy Ryza Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch, YARN-392-6.patch, YARN-392-7.patch, YARN-392.patch Currently its not possible to specify scheduling requests for specific nodes and nowhere else. The RM automatically relaxes locality to rack and * and assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-715) TestDistributedShell and TestUnmanagedAMLauncher are failing
[
https://issues.apache.org/jira/browse/YARN-715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664360#comment-13664360
]
Chris Nauroth commented on YARN-715:
{quote}
Seems like a bug in DSShell that does not handle failed container launches
properly.
{quote}
Perhaps it's related to this comment on YARN-417:
https://issues.apache.org/jira/browse/YARN-417?focusedCommentId=13609801page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13609801
{quote}
Prior to YARN-417, ApplicationMaster would check for being done at a regular
interval. Now, using the AMRMClientAsync, it only checks on container
completion, which never occurs because no containers are run.
{quote}
Perhaps because of only checking on container completion, if no container ever
completes successfully, then the AM never knows to exit, and the process
appears to hang.
TestDistributedShell and TestUnmanagedAMLauncher are failing
Key: YARN-715
URL: https://issues.apache.org/jira/browse/YARN-715
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.0.4-alpha
Reporter: Siddharth Seth
Assignee: Omkar Vinit Joshi
Tests are timing out. Looks like this is related to YARN-617.
{code}
2013-05-21 17:40:23,693 ERROR [IPC Server handler 0 on 54024]
containermanager.ContainerManagerImpl
(ContainerManagerImpl.java:authorizeRequest(412)) - Unauthorized request to
start container.
Expected containerId: user Found: container_1369183214008_0001_01_01
2013-05-21 17:40:23,694 ERROR [IPC Server handler 0 on 54024]
security.UserGroupInformation (UserGroupInformation.java:doAs(1492)) -
PriviledgedActionException as:user (auth:SIMPLE) cause:org.apache.hado
Expected containerId: user Found: container_1369183214008_0001_01_01
2013-05-21 17:40:23,695 INFO [IPC Server handler 0 on 54024] ipc.Server
(Server.java:run(1864)) - IPC Server handler 0 on 54024, call
org.apache.hadoop.yarn.api.ContainerManagerPB.startContainer from 10.
Expected containerId: user Found: container_1369183214008_0001_01_01
org.apache.hadoop.yarn.exceptions.YarnRemoteException: Unauthorized request
to start container.
Expected containerId: user Found: container_1369183214008_0001_01_01
at org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:43)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.authorizeRequest(ContainerManagerImpl.java:413)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainer(ContainerManagerImpl.java:440)
at
org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagerPBServiceImpl.startContainer(ContainerManagerPBServiceImpl.java:72)
at
org.apache.hadoop.yarn.proto.ContainerManager$ContainerManagerService$2.callBlockingMethod(ContainerManager.java:83)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-563) Add application type to ApplicationReport
[ https://issues.apache.org/jira/browse/YARN-563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated YARN-563: --- Attachment: YARN-563-trunk-11-review.patch Thanks Thomas for review. Updated the patch. Thanks, Mayank Add application type to ApplicationReport -- Key: YARN-563 URL: https://issues.apache.org/jira/browse/YARN-563 Project: Hadoop YARN Issue Type: Sub-task Reporter: Thomas Weise Assignee: Mayank Bansal Attachments: YARN-563-trunk-10-jenkins.patch, YARN-563-trunk-10-review.patch, YARN-563-trunk-11-review.patch, YARN-563-trunk-1.patch, YARN-563-trunk-2.patch, YARN-563-trunk-3.patch, YARN-563-trunk-4.patch, YARN-563-trunk-5.patch, YARN-563-trunk-6.patch, YARN-563-trunk-7.patch, YARN-563-trunk-8.patch, YARN-563-trunk-9-jenkins.patch, YARN-563-trunk-9-review.patch This field is needed to distinguish different types of applications (app master implementations). For example, we may run applications of type XYZ in a cluster alongside MR and would like to filter applications by type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-563) Add application type to ApplicationReport
[ https://issues.apache.org/jira/browse/YARN-563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated YARN-563: --- Attachment: YARN-563-trunk-11-jenkins.patch Add application type to ApplicationReport -- Key: YARN-563 URL: https://issues.apache.org/jira/browse/YARN-563 Project: Hadoop YARN Issue Type: Sub-task Reporter: Thomas Weise Assignee: Mayank Bansal Attachments: YARN-563-trunk-10-jenkins.patch, YARN-563-trunk-10-review.patch, YARN-563-trunk-11-jenkins.patch, YARN-563-trunk-11-review.patch, YARN-563-trunk-1.patch, YARN-563-trunk-2.patch, YARN-563-trunk-3.patch, YARN-563-trunk-4.patch, YARN-563-trunk-5.patch, YARN-563-trunk-6.patch, YARN-563-trunk-7.patch, YARN-563-trunk-8.patch, YARN-563-trunk-9-jenkins.patch, YARN-563-trunk-9-review.patch This field is needed to distinguish different types of applications (app master implementations). For example, we may run applications of type XYZ in a cluster alongside MR and would like to filter applications by type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664384#comment-13664384 ] Siddharth Seth commented on YARN-710: - bq. Regardless of that, the ser/deser mechanism provided in this patch should do the real work for all records. Then it is matter of deciding for which ones we want to expose the ser/deser capabilities proxying to record factory implementation, right? Agreed. This can be the internal (private) implementation for whatever the public utility is that exposes ser/deser for various records. The internal implementation could then be used by RMStateStore as well. Add to ser/deser methods to RecordFactory - Key: YARN-710 URL: https://issues.apache.org/jira/browse/YARN-710 Project: Hadoop YARN Issue Type: Bug Components: api Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: YARN-710.patch I order to do things like AMs failover and checkpointing I need to serialize app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. Because we are wrapping/hiding the PB implementation from the APIs, we are hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664484#comment-13664484 ] Alejandro Abdelnur commented on YARN-710: - Vinod, Sid, given the discussion, are we OK with the current patch, and what is missing is what will be the public API for this? Add to ser/deser methods to RecordFactory - Key: YARN-710 URL: https://issues.apache.org/jira/browse/YARN-710 Project: Hadoop YARN Issue Type: Bug Components: api Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: YARN-710.patch I order to do things like AMs failover and checkpointing I need to serialize app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. Because we are wrapping/hiding the PB implementation from the APIs, we are hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-708) Move RecordFactory classes to hadoop-yarn-api, miscellaneous fixes to the interfaces
[ https://issues.apache.org/jira/browse/YARN-708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664501#comment-13664501 ] Vinod Kumar Vavilapalli commented on YARN-708: -- The patch looked daunting at first, but now after deeper looks, seems manageable. One thing that caught my attention is the configuration changes - the generic serialization config is removed, and we still have to set three configuration items. With little more work, we can have a single config item drive everything - essentially record serialization, server and client are all tied together. But perhaps in future. +1, will check it in if and when Jenkins is okay too. Move RecordFactory classes to hadoop-yarn-api, miscellaneous fixes to the interfaces Key: YARN-708 URL: https://issues.apache.org/jira/browse/YARN-708 Project: Hadoop YARN Issue Type: Task Affects Versions: 2.0.4-alpha Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: move_YARN_708.1.sh, YARN-708.1.txt, YARN-708.full.1.txt, YARN-708_postMvScript.1.txt This is required for additional changes in YARN-528. Some of the interfaces could use some cleanup as well - they shouldn't be declaring YarnException (Runtime) in their signature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664508#comment-13664508 ] Vinod Kumar Vavilapalli commented on YARN-710: -- FWIU, Just having the a public util class which selectively serializes/deserializes records is fine. No need for tagging records. We don't want to support compatibility of disk serialization of some records, for e.g. AllocateResponse. And as Sid says, Records/RecordFactory are all YARN private APIs. So, I'd ideally separate this serialization from YARN internal serialization, by creating a parallel factory class for reading and writing records. Instead if you just want to depend on existing stuff by adding methods to RecordFactory, sure. But like mentioned, it is going to be private and not supportable. Add to ser/deser methods to RecordFactory - Key: YARN-710 URL: https://issues.apache.org/jira/browse/YARN-710 Project: Hadoop YARN Issue Type: Bug Components: api Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: YARN-710.patch I order to do things like AMs failover and checkpointing I need to serialize app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. Because we are wrapping/hiding the PB implementation from the APIs, we are hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-708) Move RecordFactory classes to hadoop-yarn-api, miscellaneous fixes to the interfaces
[
https://issues.apache.org/jira/browse/YARN-708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664513#comment-13664513
]
Hadoop QA commented on YARN-708:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584066/YARN-708.full.1.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/983//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/983//console
This message is automatically generated.
Move RecordFactory classes to hadoop-yarn-api, miscellaneous fixes to the
interfaces
Key: YARN-708
URL: https://issues.apache.org/jira/browse/YARN-708
Project: Hadoop YARN
Issue Type: Task
Affects Versions: 2.0.4-alpha
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Attachments: move_YARN_708.1.sh, YARN-708.1.txt, YARN-708.full.1.txt,
YARN-708_postMvScript.1.txt
This is required for additional changes in YARN-528.
Some of the interfaces could use some cleanup as well - they shouldn't be
declaring YarnException (Runtime) in their signature.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664537#comment-13664537 ] Alejandro Abdelnur commented on YARN-710: - If we don't want to support compatibility of disk serialization for some records, how do you identify the supported ones without a tag interface? I would expect the write call to fail if it is not supported the the given record. I'm OK with having an additional public API class to do the ser/deser of 'supported' records. Still I think such class should delegate to the RecordFactory as the record factory knows about the underlaying implementation and can process the ser/deser leveraging the implementation. Add to ser/deser methods to RecordFactory - Key: YARN-710 URL: https://issues.apache.org/jira/browse/YARN-710 Project: Hadoop YARN Issue Type: Bug Components: api Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: YARN-710.patch I order to do things like AMs failover and checkpointing I need to serialize app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. Because we are wrapping/hiding the PB implementation from the APIs, we are hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-689) Add multiplier unit to resourcecapabilities
[ https://issues.apache.org/jira/browse/YARN-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664616#comment-13664616 ] Timothy St. Clair commented on YARN-689: Hi folks, +1 in agreement with [~tucu00], around resource requests. I'm not intimately familiar with the inner workings of YARN, but I have fair amount of experience with other schedulers. They typically get around this is through expression syntax/lang where the admin can define policies in order to tune to their environment workloads, where %quantization boundaries are ideal (e.g. best fit in [X] chunks, where [X] could be whole(MB) or fractional(CPU) units), with fragmentation being the biggest problem with this flexibility. Use Case: A request comes in for 20MB, .5CPUs(cpu_shares in cgroups), 1 booster_rock, and 3 GPUs. That request is then evaluated against an expression(min,max,whatever) during the activation time, which then splices the resource appropriately. Either way, this treads into a known space that exists around resource splicing, utilization, and such and such. Ref1: http://spinningmatt.wordpress.com/2012/11/13/no-longer-thinking-in-slots-thinking-in-aggregate-resources-and-consumption-policies/ Ref2: Every paper talk that Wilkes gives. Cheers, Tim Add multiplier unit to resourcecapabilities --- Key: YARN-689 URL: https://issues.apache.org/jira/browse/YARN-689 Project: Hadoop YARN Issue Type: Bug Components: api, scheduler Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: YARN-689.patch, YARN-689.patch, YARN-689.patch Currently we overloading the minimum resource value as the actual multiplier used by the scheduler. Today with a minimum memory set to 1GB, requests for 1.5GB are always translated to allocation of 2GB. We should decouple the minimum allocation from the multiplier. The multiplier should also be exposed to the client via the RegisterApplicationMasterResponse -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-708) Move RecordFactory classes to hadoop-yarn-api, miscellaneous fixes to the interfaces
[ https://issues.apache.org/jira/browse/YARN-708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664650#comment-13664650 ] Hudson commented on YARN-708: - Integrated in Hadoop-trunk-Commit #3777 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3777/]) YARN-708. Moved RecordFactory classes to hadoop-yarn-api, and put some miscellaneous fixes to the interfaces. Contributed by Siddharth Seth. (Revision 1485478) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1485478 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/YarnException.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/factories * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/factories/RecordFactory.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/factory * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/factory/providers * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/factory/providers/RecordFactoryProvider.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/util/Records.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/YarnException.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/RecordFactory.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/RpcClientFactory.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/RpcServerFactory.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/impl/pb/RecordFactoryPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/impl/pb/RpcClientFactoryPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/impl/pb/RpcServerFactoryPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factory/providers/RecordFactoryProvider.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factory/providers/RpcFactoryProvider.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Records.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRpcFactoryProvider.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/api/protocolrecords/impl/pb/TestPBLocalizerRPC.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/api/protocolrecords/impl/pb/TestPBRecordImpl.java Move RecordFactory classes to hadoop-yarn-api, miscellaneous fixes to the interfaces Key: YARN-708 URL: https://issues.apache.org/jira/browse/YARN-708 Project: Hadoop YARN Issue Type: Task Affects Versions: 2.0.4-alpha Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: 2.0.5-beta Attachments: move_YARN_708.1.sh, YARN-708.1.txt, YARN-708.full.1.txt, YARN-708_postMvScript.1.txt This is required for additional changes in YARN-528. Some of the interfaces could use some cleanup as well - they shouldn't be declaring YarnException (Runtime) in their signature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see:
[jira] [Commented] (YARN-708) Move RecordFactory classes to hadoop-yarn-api, miscellaneous fixes to the interfaces
[ https://issues.apache.org/jira/browse/YARN-708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664661#comment-13664661 ] Hudson commented on YARN-708: - Integrated in Hadoop-trunk-Commit #3778 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3778/]) YARN-708. Deleting files that were supposed to removed in the previous commit. (Revision 1485481) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1485481 Files : * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/package-info.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/YarnRemoteExceptionFactory.java Move RecordFactory classes to hadoop-yarn-api, miscellaneous fixes to the interfaces Key: YARN-708 URL: https://issues.apache.org/jira/browse/YARN-708 Project: Hadoop YARN Issue Type: Task Affects Versions: 2.0.4-alpha Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: 2.0.5-beta Attachments: move_YARN_708.1.sh, YARN-708.1.txt, YARN-708.full.1.txt, YARN-708_postMvScript.1.txt This is required for additional changes in YARN-528. Some of the interfaces could use some cleanup as well - they shouldn't be declaring YarnException (Runtime) in their signature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-711) Copy BuilderUtil methods into individual records
[ https://issues.apache.org/jira/browse/YARN-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-711: - Attachment: YARN-711.3.patch The new patch 1. attaches public/private annotations to the individual factory method 2. Removed ApplicationMaster Record and the PBImpl of that Copy BuilderUtil methods into individual records Key: YARN-711 URL: https://issues.apache.org/jira/browse/YARN-711 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Jian He Attachments: YARN-711.1.patch, YARN-711.2.patch, YARN-711.3.patch BuilderUtils is one giant utils class which has all the factory methods needed for creating records. It is painful for users to figure out how to create records. We are better off having the factories in each record, that way users can easily create records. As a first step, we should just copy all the factory methods into individual classes, deprecate BuilderUtils and then slowly move all code off BuilderUtils. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-710) Add to ser/deser methods to RecordFactory
[ https://issues.apache.org/jira/browse/YARN-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664665#comment-13664665 ] Siddharth Seth commented on YARN-710: - bq. bq. Vinod, Sid, given the discussion, are we OK with the current patch, and what is missing is what will be the public API for this? Yep. The public API is missing. Comments on the patch itself. - The new methods should be marked @Private - Should a runtime exception be thrown in case of a reflection error. Limit the scope of the IOException to errors reading from / writing to the stream. - I'm not sure if this is possible, but can the Proto class be figured out in a different manner - directly constructing the classname or from the return type of the getProto method, instead of instantiating *PBImpl and relying on what is returned by getProto. bq. If we don't want to support compatibility of disk serialization for some records, how do you identify the supported ones without a tag interface? I would expect the write call to fail if it is not supported the the given record. bq. I'm OK with having an additional public API class to do the ser/deser of 'supported' records. Still I think such class should delegate to the RecordFactory as the record factory knows about the underlaying implementation and can process the ser/deser leveraging the implementation. I don't think tagging is required for the public API. That can just be determined by what is exposed in the helper library. Like you said, this library can make use of the changes being added in this patch. Related, PB offers other means to create recrods - such as Builder.mergeDelimitedFrom. What the patch does is probably what it should be - i.e. leaves the responsibility of reading/writing multiple records to the user, but would like to here your thoughts on using mergeFrom / mergeDelimitedFrom. Alternately, and I'd like to know what others think, does it make sense for these methods to work with Protos directly. Something like ApplicationIdProto serialize(ApplicationId) ApplicationId deserialize(ApplicationIdProto) Add to ser/deser methods to RecordFactory - Key: YARN-710 URL: https://issues.apache.org/jira/browse/YARN-710 Project: Hadoop YARN Issue Type: Bug Components: api Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: YARN-710.patch I order to do things like AMs failover and checkpointing I need to serialize app IDs, app attempt IDs, containers and/or IDs, resource requests, etc. Because we are wrapping/hiding the PB implementation from the APIs, we are hiding the built in PB ser/deser capabilities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-711) Copy BuilderUtil methods into individual records
[
https://issues.apache.org/jira/browse/YARN-711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664699#comment-13664699
]
Hadoop QA commented on YARN-711:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584400/YARN-711.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/984//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/984//console
This message is automatically generated.
Copy BuilderUtil methods into individual records
Key: YARN-711
URL: https://issues.apache.org/jira/browse/YARN-711
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Jian He
Attachments: YARN-711.1.patch, YARN-711.2.patch, YARN-711.3.patch
BuilderUtils is one giant utils class which has all the factory methods
needed for creating records. It is painful for users to figure out how to
create records. We are better off having the factories in each record, that
way users can easily create records.
As a first step, we should just copy all the factory methods into individual
classes, deprecate BuilderUtils and then slowly move all code off
BuilderUtils.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-699) TestUnmanagedAMLauncher fails with: Unauthorized request to start container
[
https://issues.apache.org/jira/browse/YARN-699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664751#comment-13664751
]
Ivan Mitic commented on YARN-699:
-
bq. YARN-715 is fixing multiple issues. Closing this as duplicate.
Ah good, thanks Vinod!
TestUnmanagedAMLauncher fails with: Unauthorized request to start container
---
Key: YARN-699
URL: https://issues.apache.org/jira/browse/YARN-699
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Ivan Mitic
Assignee: Ivan Mitic
Just run into this. Looks like YARN-617 regressed TestUnmanagedAMLauncher.
From the test log:
{noformat}
2013-05-19 12:39:10,631 INFO distributedshell.ApplicationMaster
(ApplicationMaster.java:run(682)) - Setting up container launch container for
containerid=container_1368992334149_0001_01_01
2013-05-19 12:39:10,647 INFO distributedshell.ApplicationMaster
(ApplicationMaster.java:run(690)) - Setting user in ContainerLaunchContext
to: ivanmi
2013-05-19 12:39:10,678 ERROR containermanager.ContainerManagerImpl
(ContainerManagerImpl.java:authorizeRequest(412)) - Unauthorized request to
start container.
Expected containerId: ivanmi Found: container_1368992334149_0001_01_01
2013-05-19 12:39:10,678 ERROR security.UserGroupInformation
(UserGroupInformation.java:doAs(1492)) - PriviledgedActionException as:ivanmi
(auth:SIMPLE) cause:org.apache.hadoop.yarn.exceptions.YarnRemoteException:
Unauthorized request to start container.
Expected containerId: ivanmi Found: container_1368992334149_0001_01_01
2013-05-19 12:39:10,678 INFO ipc.Server (Server.java:run(1864)) - IPC Server
handler 5 on 49529, call
org.apache.hadoop.yarn.api.ContainerManagerPB.startContainer from
10.120.19.109:49566: error:
org.apache.hadoop.yarn.exceptions.YarnRemoteException: Unauthorized request
to start container.
Expected containerId: ivanmi Found: container_1368992334149_0001_01_01
org.apache.hadoop.yarn.exceptions.YarnRemoteException: Unauthorized request
to start container.
Expected containerId: ivanmi Found: container_1368992334149_0001_01_01
at
org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:43)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.authorizeRequest(ContainerManagerImpl.java:413)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainer(ContainerManagerImpl.java:440)
at
org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagerPBServiceImpl.startContainer(ContainerManagerPBServiceImpl.java:72)
at
org.apache.hadoop.yarn.proto.ContainerManager$ContainerManagerService$2.callBlockingMethod(ContainerManager.java:83)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1033)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1842)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1838)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1489)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1836)
2013-05-19 12:39:10,678 INFO distributedshell.ApplicationMaster
(ApplicationMaster.java:run(761)) - Start container failed for :,
containerId=container_1368992334149_0001_01_01
{noformat}
ContainerManagerImpl expected containerId to be equal to the remote UGI and
since this was not the case, failed the authorization:
{noformat}
Unauthorized request to start container.
Expected containerId: ivanmi Found: container_1368992334149_0001_01_01
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-715) TestDistributedShell and TestUnmanagedAMLauncher are failing
[
https://issues.apache.org/jira/browse/YARN-715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664753#comment-13664753
]
Ivan Mitic commented on YARN-715:
-
TestUnmanagedAMLauncher was not timing out for me, from a bit of debugging I am
relatively sure it started failing after YARN-617.
TestDistributedShell and TestUnmanagedAMLauncher are failing
Key: YARN-715
URL: https://issues.apache.org/jira/browse/YARN-715
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.0.4-alpha
Reporter: Siddharth Seth
Assignee: Omkar Vinit Joshi
Tests are timing out. Looks like this is related to YARN-617.
{code}
2013-05-21 17:40:23,693 ERROR [IPC Server handler 0 on 54024]
containermanager.ContainerManagerImpl
(ContainerManagerImpl.java:authorizeRequest(412)) - Unauthorized request to
start container.
Expected containerId: user Found: container_1369183214008_0001_01_01
2013-05-21 17:40:23,694 ERROR [IPC Server handler 0 on 54024]
security.UserGroupInformation (UserGroupInformation.java:doAs(1492)) -
PriviledgedActionException as:user (auth:SIMPLE) cause:org.apache.hado
Expected containerId: user Found: container_1369183214008_0001_01_01
2013-05-21 17:40:23,695 INFO [IPC Server handler 0 on 54024] ipc.Server
(Server.java:run(1864)) - IPC Server handler 0 on 54024, call
org.apache.hadoop.yarn.api.ContainerManagerPB.startContainer from 10.
Expected containerId: user Found: container_1369183214008_0001_01_01
org.apache.hadoop.yarn.exceptions.YarnRemoteException: Unauthorized request
to start container.
Expected containerId: user Found: container_1369183214008_0001_01_01
at org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:43)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.authorizeRequest(ContainerManagerImpl.java:413)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainer(ContainerManagerImpl.java:440)
at
org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagerPBServiceImpl.startContainer(ContainerManagerPBServiceImpl.java:72)
at
org.apache.hadoop.yarn.proto.ContainerManager$ContainerManagerService$2.callBlockingMethod(ContainerManager.java:83)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-715) TestDistributedShell and TestUnmanagedAMLauncher are failing
[
https://issues.apache.org/jira/browse/YARN-715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664757#comment-13664757
]
Ivan Mitic commented on YARN-715:
-
Correction to my above comment: -TestUnmanagedAMLauncher was not timing out for
me,- from a bit of debugging I am relatively sure TestUnmanagedAMLauncher
started failing after YARN-617.
TestDistributedShell and TestUnmanagedAMLauncher are failing
Key: YARN-715
URL: https://issues.apache.org/jira/browse/YARN-715
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.0.4-alpha
Reporter: Siddharth Seth
Assignee: Omkar Vinit Joshi
Tests are timing out. Looks like this is related to YARN-617.
{code}
2013-05-21 17:40:23,693 ERROR [IPC Server handler 0 on 54024]
containermanager.ContainerManagerImpl
(ContainerManagerImpl.java:authorizeRequest(412)) - Unauthorized request to
start container.
Expected containerId: user Found: container_1369183214008_0001_01_01
2013-05-21 17:40:23,694 ERROR [IPC Server handler 0 on 54024]
security.UserGroupInformation (UserGroupInformation.java:doAs(1492)) -
PriviledgedActionException as:user (auth:SIMPLE) cause:org.apache.hado
Expected containerId: user Found: container_1369183214008_0001_01_01
2013-05-21 17:40:23,695 INFO [IPC Server handler 0 on 54024] ipc.Server
(Server.java:run(1864)) - IPC Server handler 0 on 54024, call
org.apache.hadoop.yarn.api.ContainerManagerPB.startContainer from 10.
Expected containerId: user Found: container_1369183214008_0001_01_01
org.apache.hadoop.yarn.exceptions.YarnRemoteException: Unauthorized request
to start container.
Expected containerId: user Found: container_1369183214008_0001_01_01
at org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:43)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.authorizeRequest(ContainerManagerImpl.java:413)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainer(ContainerManagerImpl.java:440)
at
org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagerPBServiceImpl.startContainer(ContainerManagerPBServiceImpl.java:72)
at
org.apache.hadoop.yarn.proto.ContainerManager$ContainerManagerService$2.callBlockingMethod(ContainerManager.java:83)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-715) TestDistributedShell and TestUnmanagedAMLauncher are failing
[
https://issues.apache.org/jira/browse/YARN-715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli reassigned YARN-715:
Assignee: Vinod Kumar Vavilapalli (was: Omkar Vinit Joshi)
Let me try to fix this..
TestDistributedShell and TestUnmanagedAMLauncher are failing
Key: YARN-715
URL: https://issues.apache.org/jira/browse/YARN-715
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.0.4-alpha
Reporter: Siddharth Seth
Assignee: Vinod Kumar Vavilapalli
Tests are timing out. Looks like this is related to YARN-617.
{code}
2013-05-21 17:40:23,693 ERROR [IPC Server handler 0 on 54024]
containermanager.ContainerManagerImpl
(ContainerManagerImpl.java:authorizeRequest(412)) - Unauthorized request to
start container.
Expected containerId: user Found: container_1369183214008_0001_01_01
2013-05-21 17:40:23,694 ERROR [IPC Server handler 0 on 54024]
security.UserGroupInformation (UserGroupInformation.java:doAs(1492)) -
PriviledgedActionException as:user (auth:SIMPLE) cause:org.apache.hado
Expected containerId: user Found: container_1369183214008_0001_01_01
2013-05-21 17:40:23,695 INFO [IPC Server handler 0 on 54024] ipc.Server
(Server.java:run(1864)) - IPC Server handler 0 on 54024, call
org.apache.hadoop.yarn.api.ContainerManagerPB.startContainer from 10.
Expected containerId: user Found: container_1369183214008_0001_01_01
org.apache.hadoop.yarn.exceptions.YarnRemoteException: Unauthorized request
to start container.
Expected containerId: user Found: container_1369183214008_0001_01_01
at org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:43)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.authorizeRequest(ContainerManagerImpl.java:413)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainer(ContainerManagerImpl.java:440)
at
org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagerPBServiceImpl.startContainer(ContainerManagerPBServiceImpl.java:72)
at
org.apache.hadoop.yarn.proto.ContainerManager$ContainerManagerService$2.callBlockingMethod(ContainerManager.java:83)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-664) throw InvalidRequestException for requests with different capabilities at the same priority
[ https://issues.apache.org/jira/browse/YARN-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664760#comment-13664760 ] Arun C Murthy commented on YARN-664: +1 for fixing this. This is, potentially, a very bad bug. I'm not convinced about YARN-314, so for now, at least, we should throw an exception rather than cause inconsistencies. throw InvalidRequestException for requests with different capabilities at the same priority --- Key: YARN-664 URL: https://issues.apache.org/jira/browse/YARN-664 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager, scheduler Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Nothing stops an application from submitting a request with priority=1, location=*, memory=1024 and a request with priority=1, location=rack1, memory=2048. However, this does not make sense under the request model and can cause bad things to happen in the scheduler. It should be possible to detect this at AMRM heartbeat time and throw an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-711) Copy BuilderUtil methods into individual records
[ https://issues.apache.org/jira/browse/YARN-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-711: - Attachment: YARN-711.4.patch new patch fixed last comments. Tokens are handled in YARN-717 Copy BuilderUtil methods into individual records Key: YARN-711 URL: https://issues.apache.org/jira/browse/YARN-711 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Jian He Attachments: YARN-711.1.patch, YARN-711.2.patch, YARN-711.3.patch, YARN-711.4.patch BuilderUtils is one giant utils class which has all the factory methods needed for creating records. It is painful for users to figure out how to create records. We are better off having the factories in each record, that way users can easily create records. As a first step, we should just copy all the factory methods into individual classes, deprecate BuilderUtils and then slowly move all code off BuilderUtils. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-717) Copy BuilderUtil methods into token-related records
[ https://issues.apache.org/jira/browse/YARN-717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-717: - Attachment: YARN-717.1.patch This patch removes clienToken/ContainerToken/DelegationToken interface, and let *PBImpl only extends TokenPBImpl, also make Token abstract class. Copy BuilderUtil methods into token-related records --- Key: YARN-717 URL: https://issues.apache.org/jira/browse/YARN-717 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Attachments: YARN-717.1.patch This is separated from YARN-711,as after changing yarn.api.token from interface to abstract class, eg: ClientTokenPBImpl has to extend two classes: both TokenPBImpl and ClientToken abstract class, which is not allowed in JAVA. We may remove the ClientToken/ContainerToken/DelegationToken interface and just use the common Token interface -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-711) Copy BuilderUtil methods into individual records
[ https://issues.apache.org/jira/browse/YARN-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-711: - Attachment: YARN-711.5.patch fix some annotations and applicationReport factory method, unnecessary diff Copy BuilderUtil methods into individual records Key: YARN-711 URL: https://issues.apache.org/jira/browse/YARN-711 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Jian He Attachments: YARN-711.1.patch, YARN-711.2.patch, YARN-711.3.patch, YARN-711.4.patch, YARN-711.5.patch BuilderUtils is one giant utils class which has all the factory methods needed for creating records. It is painful for users to figure out how to create records. We are better off having the factories in each record, that way users can easily create records. As a first step, we should just copy all the factory methods into individual classes, deprecate BuilderUtils and then slowly move all code off BuilderUtils. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-422) Add NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-422: - Attachment: YARN-422.8.patch Thank Vinod for the comments! I've addressed most of them. WRT the question: bq. What if a container is already stopped is stopped again? In the updated patch, stopping a container when it is not started or scheduled to start will throw the exception immediately. bq. In fact, you can use a simple state-machine and explicit transitions instead of the field states and the conditional logic. State machine is good idea, but it seems not to be a quick solution. How about we confirming the API, and improving the implementation later? Add NM client library - Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: AMNMClient_Defination.txt, AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf, YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch, YARN-422.4.patch, YARN-422.5.patch, YARN-422.6.patch, YARN-422.8.patch Create a simple wrapper over the ContainerManager protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[
https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664807#comment-13664807
]
Hadoop QA commented on YARN-422:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584430/YARN-422.8.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/986//console
This message is automatically generated.
Add NM client library
-
Key: YARN-422
URL: https://issues.apache.org/jira/browse/YARN-422
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Zhijie Shen
Attachments: AMNMClient_Defination.txt,
AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf,
YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch, YARN-422.4.patch,
YARN-422.5.patch, YARN-422.6.patch, YARN-422.8.patch
Create a simple wrapper over the ContainerManager protocol to provide hide
the details of the protocol implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-664) throw InvalidRequestException for requests with different capabilities at the same priority
[ https://issues.apache.org/jira/browse/YARN-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664808#comment-13664808 ] Bikas Saha commented on YARN-664: - I don't think that this is invalid as far as the API is concerned. There might be reasons in the current implementations of the schedulers that may make supporting this prohibitive, and so it was not done. Say an application is running 2 concurrent but unrelated items that should not be blocked by each other due to performance. Both have different resources sizes. If they are separated by priority then the lower priority item will not get any containers until all the containers for the higher priority item have been allocated. This is by definition of priority. That may lead to poor performance. So the app would want to run both at the same priority and expect the RM to allocate containers to both wherever resources are free to match them. The most simple solution for the RM that I can think off the top of my head is to always use the largest resource size. It may lead to fragmentation but is safe. Currently, the last resource size is used. This means that if the client asks for 1GB and 2GB and the 1GB happens to be the last one set by the RM then the RM will return 2 1GB containers, which is wrong. If we choose to use the largest value as a solution, then we should close this as invalid and use YARN-314 to implement that solution. throw InvalidRequestException for requests with different capabilities at the same priority --- Key: YARN-664 URL: https://issues.apache.org/jira/browse/YARN-664 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager, scheduler Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Nothing stops an application from submitting a request with priority=1, location=*, memory=1024 and a request with priority=1, location=rack1, memory=2048. However, this does not make sense under the request model and can cause bad things to happen in the scheduler. It should be possible to detect this at AMRM heartbeat time and throw an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-716) Make ApplicationID immutable
[
https://issues.apache.org/jira/browse/YARN-716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664814#comment-13664814
]
Hadoop QA commented on YARN-716:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584229/YARN-716.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 24 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.mapreduce.v2.app.webapp.TestAppController
org.apache.hadoop.mapreduce.TestTypeConverter
org.apache.hadoop.mapreduce.v2.util.TestMRApps
org.apache.hadoop.mapred.TestResourceMgrDelegate
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/985//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/985//console
This message is automatically generated.
Make ApplicationID immutable
Key: YARN-716
URL: https://issues.apache.org/jira/browse/YARN-716
Project: Hadoop YARN
Issue Type: Task
Affects Versions: 2.0.4-alpha
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Attachments: YARN-716.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-715) TestDistributedShell and TestUnmanagedAMLauncher are failing
[
https://issues.apache.org/jira/browse/YARN-715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated YARN-715:
-
Attachment: YARN-715-20130522.txt
This should fix it. The tests pass for me now with the patch.
TestDistributedShell and TestUnmanagedAMLauncher are failing
Key: YARN-715
URL: https://issues.apache.org/jira/browse/YARN-715
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.0.4-alpha
Reporter: Siddharth Seth
Assignee: Vinod Kumar Vavilapalli
Attachments: YARN-715-20130522.txt
Tests are timing out. Looks like this is related to YARN-617.
{code}
2013-05-21 17:40:23,693 ERROR [IPC Server handler 0 on 54024]
containermanager.ContainerManagerImpl
(ContainerManagerImpl.java:authorizeRequest(412)) - Unauthorized request to
start container.
Expected containerId: user Found: container_1369183214008_0001_01_01
2013-05-21 17:40:23,694 ERROR [IPC Server handler 0 on 54024]
security.UserGroupInformation (UserGroupInformation.java:doAs(1492)) -
PriviledgedActionException as:user (auth:SIMPLE) cause:org.apache.hado
Expected containerId: user Found: container_1369183214008_0001_01_01
2013-05-21 17:40:23,695 INFO [IPC Server handler 0 on 54024] ipc.Server
(Server.java:run(1864)) - IPC Server handler 0 on 54024, call
org.apache.hadoop.yarn.api.ContainerManagerPB.startContainer from 10.
Expected containerId: user Found: container_1369183214008_0001_01_01
org.apache.hadoop.yarn.exceptions.YarnRemoteException: Unauthorized request
to start container.
Expected containerId: user Found: container_1369183214008_0001_01_01
at org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:43)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.authorizeRequest(ContainerManagerImpl.java:413)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainer(ContainerManagerImpl.java:440)
at
org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagerPBServiceImpl.startContainer(ContainerManagerPBServiceImpl.java:72)
at
org.apache.hadoop.yarn.proto.ContainerManager$ContainerManagerService$2.callBlockingMethod(ContainerManager.java:83)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-715) TestDistributedShell and TestUnmanagedAMLauncher are failing
[
https://issues.apache.org/jira/browse/YARN-715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664822#comment-13664822
]
Hadoop QA commented on YARN-715:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584433/YARN-715-20130522.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/987//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/987//console
This message is automatically generated.
TestDistributedShell and TestUnmanagedAMLauncher are failing
Key: YARN-715
URL: https://issues.apache.org/jira/browse/YARN-715
Project: Hadoop YARN
Issue Type: Bug
Reporter: Siddharth Seth
Assignee: Vinod Kumar Vavilapalli
Attachments: YARN-715-20130522.txt
Tests are timing out. Looks like this is related to YARN-617.
{code}
2013-05-21 17:40:23,693 ERROR [IPC Server handler 0 on 54024]
containermanager.ContainerManagerImpl
(ContainerManagerImpl.java:authorizeRequest(412)) - Unauthorized request to
start container.
Expected containerId: user Found: container_1369183214008_0001_01_01
2013-05-21 17:40:23,694 ERROR [IPC Server handler 0 on 54024]
security.UserGroupInformation (UserGroupInformation.java:doAs(1492)) -
PriviledgedActionException as:user (auth:SIMPLE) cause:org.apache.hado
Expected containerId: user Found: container_1369183214008_0001_01_01
2013-05-21 17:40:23,695 INFO [IPC Server handler 0 on 54024] ipc.Server
(Server.java:run(1864)) - IPC Server handler 0 on 54024, call
org.apache.hadoop.yarn.api.ContainerManagerPB.startContainer from 10.
Expected containerId: user Found: container_1369183214008_0001_01_01
org.apache.hadoop.yarn.exceptions.YarnRemoteException: Unauthorized request
to start container.
Expected containerId: user Found: container_1369183214008_0001_01_01
at org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:43)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.authorizeRequest(ContainerManagerImpl.java:413)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainer(ContainerManagerImpl.java:440)
at
org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagerPBServiceImpl.startContainer(ContainerManagerPBServiceImpl.java:72)
at
org.apache.hadoop.yarn.proto.ContainerManager$ContainerManagerService$2.callBlockingMethod(ContainerManager.java:83)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[
https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664844#comment-13664844
]
Zhijie Shen commented on YARN-422:
--
It seems that smart-apply-patch.sh doesn't accept the following section:
{code}
diff --git
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index f9b017d..4271d72 100644
---
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -381,6 +381,14 @@
public static final String NM_LOG_DIRS = NM_PREFIX + log-dirs;
public static final String DEFAULT_NM_LOG_DIRS = /tmp/logs;
+ /**
+ * Max number of threads in NMClientAsync to process container management
+ * events
+ */
+ public static final String NM_CLIENT_ASYNC_THREAD_POOL_MAX_SIZE =
+ YARN_PREFIX + client.nodemanager-client-async.thread-pool-max-size;
+ public static final int DEFAULT_NM_CLIENT_ASYNC_THREAD_POOL_MAX_SIZE = 500;
+
/** Interval at which the delayed token removal thread runs */
public static final String RM_DELAYED_DELEGATION_TOKEN_REMOVAL_INTERVAL_MS =
RM_PREFIX + delayed.delegation-token.removal-interval-ms;
{code}
Add NM client library
-
Key: YARN-422
URL: https://issues.apache.org/jira/browse/YARN-422
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Zhijie Shen
Attachments: AMNMClient_Defination.txt,
AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf,
YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch, YARN-422.4.patch,
YARN-422.5.patch, YARN-422.6.patch, YARN-422.8.patch
Create a simple wrapper over the ContainerManager protocol to provide hide
the details of the protocol implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-422) Add NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-422: - Attachment: YARN-422.9.patch Updated the patch against the latest trunk Add NM client library - Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: AMNMClient_Defination.txt, AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf, YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch, YARN-422.4.patch, YARN-422.5.patch, YARN-422.6.patch, YARN-422.8.patch, YARN-422.9.patch Create a simple wrapper over the ContainerManager protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[
https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664876#comment-13664876
]
Hadoop QA commented on YARN-422:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584440/YARN-422.9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/988//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/988//console
This message is automatically generated.
Add NM client library
-
Key: YARN-422
URL: https://issues.apache.org/jira/browse/YARN-422
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Zhijie Shen
Attachments: AMNMClient_Defination.txt,
AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf,
YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch, YARN-422.4.patch,
YARN-422.5.patch, YARN-422.6.patch, YARN-422.8.patch, YARN-422.9.patch
Create a simple wrapper over the ContainerManager protocol to provide hide
the details of the protocol implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-715) TestDistributedShell and TestUnmanagedAMLauncher are failing
[
https://issues.apache.org/jira/browse/YARN-715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664883#comment-13664883
]
Siddharth Seth commented on YARN-715:
-
+1. Thanks Vinod.
TestDistributedShell and TestUnmanagedAMLauncher are failing
Key: YARN-715
URL: https://issues.apache.org/jira/browse/YARN-715
Project: Hadoop YARN
Issue Type: Bug
Reporter: Siddharth Seth
Assignee: Vinod Kumar Vavilapalli
Attachments: YARN-715-20130522.txt
Tests are timing out. Looks like this is related to YARN-617.
{code}
2013-05-21 17:40:23,693 ERROR [IPC Server handler 0 on 54024]
containermanager.ContainerManagerImpl
(ContainerManagerImpl.java:authorizeRequest(412)) - Unauthorized request to
start container.
Expected containerId: user Found: container_1369183214008_0001_01_01
2013-05-21 17:40:23,694 ERROR [IPC Server handler 0 on 54024]
security.UserGroupInformation (UserGroupInformation.java:doAs(1492)) -
PriviledgedActionException as:user (auth:SIMPLE) cause:org.apache.hado
Expected containerId: user Found: container_1369183214008_0001_01_01
2013-05-21 17:40:23,695 INFO [IPC Server handler 0 on 54024] ipc.Server
(Server.java:run(1864)) - IPC Server handler 0 on 54024, call
org.apache.hadoop.yarn.api.ContainerManagerPB.startContainer from 10.
Expected containerId: user Found: container_1369183214008_0001_01_01
org.apache.hadoop.yarn.exceptions.YarnRemoteException: Unauthorized request
to start container.
Expected containerId: user Found: container_1369183214008_0001_01_01
at org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:43)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.authorizeRequest(ContainerManagerImpl.java:413)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainer(ContainerManagerImpl.java:440)
at
org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagerPBServiceImpl.startContainer(ContainerManagerPBServiceImpl.java:72)
at
org.apache.hadoop.yarn.proto.ContainerManager$ContainerManagerService$2.callBlockingMethod(ContainerManager.java:83)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
