[jira] [Commented] (YARN-422) Add NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13669145#comment-13669145 ] Hudson commented on YARN-422: - Integrated in Hadoop-Yarn-trunk #224 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/224/]) YARN-422. Add a NM Client library to help application-writers. Contributed by Zhijie Shen. (Revision 1487184) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1487184 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/NMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/NMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/NMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestNMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestNMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml Add NM client library - Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Fix For: 2.0.5-beta Attachments: AMNMClient_Defination.txt, AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf, YARN-422.10.patch, YARN-422.11.patch, YARN-422.12.patch, YARN-422.13.patch, YARN-422.14.patch, YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch, YARN-422.4.patch, YARN-422.5.patch, YARN-422.6.patch, YARN-422.8.patch, YARN-422.9.patch, YARN-422-javadoc-fixes.txt Create a simple wrapper over the ContainerManager protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13669217#comment-13669217 ] Hudson commented on YARN-422: - Integrated in Hadoop-Hdfs-trunk #1414 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1414/]) YARN-422. Add a NM Client library to help application-writers. Contributed by Zhijie Shen. (Revision 1487184) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1487184 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/NMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/NMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/NMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestNMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestNMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml Add NM client library - Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Fix For: 2.0.5-beta Attachments: AMNMClient_Defination.txt, AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf, YARN-422.10.patch, YARN-422.11.patch, YARN-422.12.patch, YARN-422.13.patch, YARN-422.14.patch, YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch, YARN-422.4.patch, YARN-422.5.patch, YARN-422.6.patch, YARN-422.8.patch, YARN-422.9.patch, YARN-422-javadoc-fixes.txt Create a simple wrapper over the ContainerManager protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13669271#comment-13669271 ] Hudson commented on YARN-422: - Integrated in Hadoop-Mapreduce-trunk #1440 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1440/]) YARN-422. Add a NM Client library to help application-writers. Contributed by Zhijie Shen. (Revision 1487184) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1487184 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/NMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/NMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/NMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestNMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestNMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml Add NM client library - Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Fix For: 2.0.5-beta Attachments: AMNMClient_Defination.txt, AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf, YARN-422.10.patch, YARN-422.11.patch, YARN-422.12.patch, YARN-422.13.patch, YARN-422.14.patch, YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch, YARN-422.4.patch, YARN-422.5.patch, YARN-422.6.patch, YARN-422.8.patch, YARN-422.9.patch, YARN-422-javadoc-fixes.txt Create a simple wrapper over the ContainerManager protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[
https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13668845#comment-13668845
]
Hadoop QA commented on YARN-422:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585122/YARN-422.14.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1017//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1017//console
This message is automatically generated.
Add NM client library
-
Key: YARN-422
URL: https://issues.apache.org/jira/browse/YARN-422
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Zhijie Shen
Attachments: AMNMClient_Defination.txt,
AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf,
YARN-422.10.patch, YARN-422.11.patch, YARN-422.12.patch, YARN-422.13.patch,
YARN-422.14.patch, YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch,
YARN-422.4.patch, YARN-422.5.patch, YARN-422.6.patch, YARN-422.8.patch,
YARN-422.9.patch, YARN-422-javadoc-fixes.txt
Create a simple wrapper over the ContainerManager protocol to provide hide
the details of the protocol implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13668894#comment-13668894 ] Vinod Kumar Vavilapalli commented on YARN-422: -- Looks good, +1. Checking this in. Add NM client library - Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: AMNMClient_Defination.txt, AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf, YARN-422.10.patch, YARN-422.11.patch, YARN-422.12.patch, YARN-422.13.patch, YARN-422.14.patch, YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch, YARN-422.4.patch, YARN-422.5.patch, YARN-422.6.patch, YARN-422.8.patch, YARN-422.9.patch, YARN-422-javadoc-fixes.txt Create a simple wrapper over the ContainerManager protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[
https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666916#comment-13666916
]
Hadoop QA commented on YARN-422:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584807/YARN-422.12.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1007//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/1007//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-client.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1007//console
This message is automatically generated.
Add NM client library
-
Key: YARN-422
URL: https://issues.apache.org/jira/browse/YARN-422
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Zhijie Shen
Attachments: AMNMClient_Defination.txt,
AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf,
YARN-422.10.patch, YARN-422.11.patch, YARN-422.12.patch, YARN-422.1.patch,
YARN-422.2.patch, YARN-422.3.patch, YARN-422.4.patch, YARN-422.5.patch,
YARN-422.6.patch, YARN-422.8.patch, YARN-422.9.patch,
YARN-422-javadoc-fixes.txt
Create a simple wrapper over the ContainerManager protocol to provide hide
the details of the protocol implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[
https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13663910#comment-13663910
]
Zhijie Shen commented on YARN-422:
--
bq. There is no need to start NMCommunicator everytime for stop/getStatus? We
can cache the one's from startContainer() ?
In the previous discussion, you suggested:
{quote}
Among the 3 choices you listed in stopContainer(), we should adopt (1) for
now as that is what MR AM's ContainerLauncher. Primarily because of the
one-connection-per-container issue. This again can and should be changed post
YARN-613.
{quote}
In ContainerLauncherImpl, the proxy closes every time when a container is
started or stopped (getStatus is not used there).
bq. All the call-backs are blocking.
IMHO, the callbacks can be either blocking or non-blocking. It depends on the
implementation of the callback interface. To be non-blocking, the
implementation can similarly schedule an event when the callbacks are invoked,
and the event will be dispatched to a thread for processing.
Add NM client library
-
Key: YARN-422
URL: https://issues.apache.org/jira/browse/YARN-422
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Zhijie Shen
Attachments: AMNMClient_Defination.txt,
AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf,
YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch, YARN-422.4.patch,
YARN-422.5.patch, YARN-422.6.patch
Create a simple wrapper over the ContainerManager protocol to provide hide
the details of the protocol implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[
https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664807#comment-13664807
]
Hadoop QA commented on YARN-422:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584430/YARN-422.8.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/986//console
This message is automatically generated.
Add NM client library
-
Key: YARN-422
URL: https://issues.apache.org/jira/browse/YARN-422
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Zhijie Shen
Attachments: AMNMClient_Defination.txt,
AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf,
YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch, YARN-422.4.patch,
YARN-422.5.patch, YARN-422.6.patch, YARN-422.8.patch
Create a simple wrapper over the ContainerManager protocol to provide hide
the details of the protocol implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[
https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664844#comment-13664844
]
Zhijie Shen commented on YARN-422:
--
It seems that smart-apply-patch.sh doesn't accept the following section:
{code}
diff --git
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index f9b017d..4271d72 100644
---
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -381,6 +381,14 @@
public static final String NM_LOG_DIRS = NM_PREFIX + log-dirs;
public static final String DEFAULT_NM_LOG_DIRS = /tmp/logs;
+ /**
+ * Max number of threads in NMClientAsync to process container management
+ * events
+ */
+ public static final String NM_CLIENT_ASYNC_THREAD_POOL_MAX_SIZE =
+ YARN_PREFIX + client.nodemanager-client-async.thread-pool-max-size;
+ public static final int DEFAULT_NM_CLIENT_ASYNC_THREAD_POOL_MAX_SIZE = 500;
+
/** Interval at which the delayed token removal thread runs */
public static final String RM_DELAYED_DELEGATION_TOKEN_REMOVAL_INTERVAL_MS =
RM_PREFIX + delayed.delegation-token.removal-interval-ms;
{code}
Add NM client library
-
Key: YARN-422
URL: https://issues.apache.org/jira/browse/YARN-422
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Zhijie Shen
Attachments: AMNMClient_Defination.txt,
AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf,
YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch, YARN-422.4.patch,
YARN-422.5.patch, YARN-422.6.patch, YARN-422.8.patch
Create a simple wrapper over the ContainerManager protocol to provide hide
the details of the protocol implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[
https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664876#comment-13664876
]
Hadoop QA commented on YARN-422:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584440/YARN-422.9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/988//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/988//console
This message is automatically generated.
Add NM client library
-
Key: YARN-422
URL: https://issues.apache.org/jira/browse/YARN-422
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Zhijie Shen
Attachments: AMNMClient_Defination.txt,
AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf,
YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch, YARN-422.4.patch,
YARN-422.5.patch, YARN-422.6.patch, YARN-422.8.patch, YARN-422.9.patch
Create a simple wrapper over the ContainerManager protocol to provide hide
the details of the protocol implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662538#comment-13662538 ] Vinod Kumar Vavilapalli commented on YARN-422: -- YARN-613 isn't progressing as fast as I'd like. I am tending to push this before YARN-613. Will do a quick review. This is all new code, so can push this in and then fix it after YARN-613 is done w.r.t connection management. Add NM client library - Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: AMNMClient_Defination.txt, AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf, YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch, YARN-422.4.patch, YARN-422.5.patch, YARN-422.6.patch Create a simple wrapper over the ContainerManager protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[
https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662547#comment-13662547
]
Hadoop QA commented on YARN-422:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583928/YARN-422.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/966//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/966//console
This message is automatically generated.
Add NM client library
-
Key: YARN-422
URL: https://issues.apache.org/jira/browse/YARN-422
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Zhijie Shen
Attachments: AMNMClient_Defination.txt,
AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf,
YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch, YARN-422.4.patch,
YARN-422.5.patch, YARN-422.6.patch
Create a simple wrapper over the ContainerManager protocol to provide hide
the details of the protocol implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[
https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656127#comment-13656127
]
Zhijie Shen commented on YARN-422:
--
{quote}
Why is TestAMRMClient being removed?
{quote}
TestAMRMClient is renamed as TestAMClients, because it not only tests
AMRMClient but also tests NMClient. The reason why they need to be done
together, because in the minicluster, AMRMClient has to be used to allocate
container, then the container can be started through NMClient. Otherwise, if we
create TestNMClient independently, we still need to duplicate the AMRMClient
part.
Add NM client library
-
Key: YARN-422
URL: https://issues.apache.org/jira/browse/YARN-422
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Zhijie Shen
Attachments: AMNMClient_Defination.txt,
AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf,
YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch, YARN-422.4.patch,
YARN-422.5.patch
Create a simple wrapper over the ContainerManager protocol to provide hide
the details of the protocol implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[
https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13655689#comment-13655689
]
Bikas Saha commented on YARN-422:
-
Rename onExceptionRaisedWhenStartingContainer() to onStartContainerError()?
What do you say?
The comment should not say 10 since the value can change with time. Is the
initial value checked to be less than the MAX specified?
{code}
+// Start with a default core-pool size of 10 and change it dynamically.
+threadPool = new ThreadPoolExecutor(INITIAL_THREAD_POOL_SIZE,
+Integer.MAX_VALUE, 1, TimeUnit.HOURS,
{code}
Improve grammar?
{code}
+ // See if we need up the pool size only if haven't reached the
+ // maximum limit yet.
{code}
Should the boolean flag set/get be part of NMClient interface itself?
{code}
+if (!(client instanceof NMClientImpl) ||
+((NMClientImpl) client).stopAllRunningContainersOnStoppingEnabled())
{code}
Why is TestAMRMClient being removed?
We need to double check the synchronization/thread safety of this class. Lots
of objects and threads. Can you please document the expected locking order?
Add NM client library
-
Key: YARN-422
URL: https://issues.apache.org/jira/browse/YARN-422
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Zhijie Shen
Attachments: AMNMClient_Defination.txt,
AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf,
YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch, YARN-422.4.patch,
YARN-422.5.patch
Create a simple wrapper over the ContainerManager protocol to provide hide
the details of the protocol implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[
https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13651429#comment-13651429
]
Hadoop QA commented on YARN-422:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582193/YARN-422.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/890//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/890//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-client.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/890//console
This message is automatically generated.
Add NM client library
-
Key: YARN-422
URL: https://issues.apache.org/jira/browse/YARN-422
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Zhijie Shen
Attachments: AMNMClient_Defination.txt,
AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf,
YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch, YARN-422.4.patch
Create a simple wrapper over the ContainerManager protocol to provide hide
the details of the protocol implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[
https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13651455#comment-13651455
]
Hadoop QA commented on YARN-422:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582206/YARN-422.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/892//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/892//console
This message is automatically generated.
Add NM client library
-
Key: YARN-422
URL: https://issues.apache.org/jira/browse/YARN-422
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Zhijie Shen
Attachments: AMNMClient_Defination.txt,
AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf,
YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch, YARN-422.4.patch,
YARN-422.5.patch
Create a simple wrapper over the ContainerManager protocol to provide hide
the details of the protocol implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13649809#comment-13649809 ] Zhijie Shen commented on YARN-422: -- Like ContainerLauncher, I think NMClient is supposed to start when AM starts and stop when AM stops. IMHO, no matter how long a container keeps running, it should be stopped when the AM who starts it stops. In addition, NMClient would be the only gate to access NodeManager (I mean it's not good to use NMClient and the raw RPC proxy simultaneously), such that if NMClient doesn't stop the running containers when NMClient stops (or AM exits), these containers may be not stoppable. Semantically, it is not good that AM has already stopped, while the forked containers keep running. Add NM client library - Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: AMNMClient_Defination.txt, AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf, YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch Create a simple wrapper over the ContainerManager protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13649869#comment-13649869 ] Bikas Saha commented on YARN-422: - I am not contending that the behavior is wrong. I am suggesting that it should be optional with the default set to true. If needed, it can be overridden to not do so. e.g. there is a pending change in AMRMClient to not unregister with the RM when the RM sends a reboot command. That is the correct behavior but it will have an override option because sometimes apps may actually want to unregister (race between reboot and app completion). Add NM client library - Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: AMNMClient_Defination.txt, AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf, YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch Create a simple wrapper over the ContainerManager protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13649899#comment-13649899 ] Zhijie Shen commented on YARN-422: -- Well, giving users the choice to disable the feature sounds reasonable. I'll add the setting for it. Moreover, let me add some javadoc to describe this part. Add NM client library - Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: AMNMClient_Defination.txt, AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf, YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch Create a simple wrapper over the ContainerManager protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13650182#comment-13650182 ] Zhijie Shen commented on YARN-422: -- When I investigated using NMClient in M/R AM (see MAPREDUCE-5203), I found one limitation of the current design. Note that NMClientAsync doesn't execute the start/stop/query RPC calls immediately when the corresponding APIs are called. Instead, an event is scheduled to start a thread later. The users of NMClientAsync cannot define some logic to be run immediately before the RPC calls. For example, in ContainerLauncherImpl, the container state must be checked right before the PRC calls in launch() and kill(). To be logically correct, this logic cannot be move up to the place where the event is scheduled. Therefore, it's useful to let users define what to do immediately before the three RPC calls. I proposed to add three more APIs in CallbackHandler, and insert the hoots immediately before the three RPC calls, respectively. Add NM client library - Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: AMNMClient_Defination.txt, AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf, YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch Create a simple wrapper over the ContainerManager protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13650190#comment-13650190 ] Bikas Saha commented on YARN-422: - That might be going down a slippery slope. IMO, while its a good idea to allow MR app to use this library, it may not be a good idea for the library to adhere to every idiosyncrasy of the MR app. The MR app has its own development legacy and it may not be possible to retro-fit a client onto it without major surgery on both sides. Thoughts? Add NM client library - Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: AMNMClient_Defination.txt, AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf, YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch Create a simple wrapper over the ContainerManager protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13650194#comment-13650194 ] Vinod Kumar Vavilapalli commented on YARN-422: -- bq. Is it necessary for the library to stop all containers before stopping itself? I think the callers should have the option of stopping together with killing all containers or stopping without killing all containers. bq. Note that NMClientAsync doesn't execute the start/stop/query RPC calls immediately when the corresponding APIs are called. Instead, an event is scheduled to start a thread later. We don't need more handlers. That code is written to make sure that, for e.g., start doesn't happen on a container that is already asked to be stopped. We should just pull those changes into your library, they aren't arbitrary code that needs plugin points. Add NM client library - Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: AMNMClient_Defination.txt, AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf, YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch Create a simple wrapper over the ContainerManager protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[
https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13650370#comment-13650370
]
Hadoop QA commented on YARN-422:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12581791/YARN-422.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/874//console
This message is automatically generated.
Add NM client library
-
Key: YARN-422
URL: https://issues.apache.org/jira/browse/YARN-422
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Zhijie Shen
Attachments: AMNMClient_Defination.txt,
AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf,
YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch
Create a simple wrapper over the ContainerManager protocol to provide hide
the details of the protocol implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13649119#comment-13649119 ] Bikas Saha commented on YARN-422: - Is it necessary for the library to stop all containers before stopping itself? I dont think its the semantics of the protocol and should not be enforced by the library. I can easily see cases in which clients can start a bunch of long running containers and go away. Add NM client library - Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: AMNMClient_Defination.txt, AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf, YARN-422.1.patch, YARN-422.2.patch, YARN-422.3.patch Create a simple wrapper over the ContainerManager protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
