[jira] [Commented] (YARN-852) TestAggregatedLogFormat.testContainerLogsFileAccess fails on Windows
[
https://issues.apache.org/jira/browse/YARN-852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687702#comment-13687702
]
Hadoop QA commented on YARN-852:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588544/YARN-852-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1346//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1346//console
This message is automatically generated.
> TestAggregatedLogFormat.testContainerLogsFileAccess fails on Windows
>
>
> Key: YARN-852
> URL: https://issues.apache.org/jira/browse/YARN-852
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Chuan Liu
>Assignee: Chuan Liu
>Priority: Minor
> Attachments: YARN-852-trunk.patch, YARN-852-trunk.patch
>
>
> The YARN unit test case fails on Windows when comparing expected message with
> log message in the file. The expected message constructed in the test case
> has two problems: 1) it uses Path.separator to concatenate path string.
> Path.separator is always a forward slash, which does not match the backslash
> used in the log message. 2) On Windows, the default file owner is
> Administrators group if the file is created by an Administrators user. The
> test expect the user to be the current user.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-553) Have YarnClient generate a directly usable ApplicationSubmissionContext
[
https://issues.apache.org/jira/browse/YARN-553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687708#comment-13687708
]
Hudson commented on YARN-553:
-
Integrated in Hadoop-trunk-Commit #3979 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/3979/])
YARN-553. Replaced YarnClient.getNewApplication with
YarnClient.createApplication which provides a directly usable
ApplicationSubmissionContext to simplify the api. Contributed by Karthik
Kambatla. (Revision 1494476)
Result = SUCCESS
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494476
Files :
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClientApplication.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestNMClient.java
> Have YarnClient generate a directly usable ApplicationSubmissionContext
> ---
>
> Key: YARN-553
> URL: https://issues.apache.org/jira/browse/YARN-553
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
>Affects Versions: 2.0.3-alpha
>Reporter: Harsh J
>Assignee: Karthik Kambatla
>Priority: Minor
> Fix For: 2.1.0-beta
>
> Attachments: yarn-553-1.patch, yarn-553-2.patch, yarn-553-3.patch,
> yarn-553-4.patch, yarn-553-5.patch, yarn-553-6.patch, yarn-553-7.patch
>
>
> Right now, we're doing multiple steps to create a relevant
> ApplicationSubmissionContext for a pre-received GetNewApplicationResponse.
> {code}
> GetNewApplicationResponse newApp = yarnClient.getNewApplication();
> ApplicationId appId = newApp.getApplicationId();
> ApplicationSubmissionContext appContext =
> Records.newRecord(ApplicationSubmissionContext.class);
> appContext.setApplicationId(appId);
> {code}
> A simplified way may be to have the GetNewApplicationResponse itself provide
> a helper method that builds a usable ApplicationSubmissionContext for us.
> Something like:
> {code}
> GetNewApplicationResponse newApp = yarnClient.getNewApplication();
> ApplicationSubmissionContext appContext =
> newApp.generateApplicationSubmissionContext();
> {code}
> [The above method can also take an arg for the container launch spec, or
> perhaps pre-load defaults like min-resource, etc. in the returned object,
> aside of just associating the application ID automatically.]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-425) coverage fix for yarn api
[ https://issues.apache.org/jira/browse/YARN-425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Gorshkov updated YARN-425: -- Attachment: YARN-425-trunk-v1.patch YARN-425-branch-2-v1.patch YARN-425-branch-0.23-v1.patch > coverage fix for yarn api > - > > Key: YARN-425 > URL: https://issues.apache.org/jira/browse/YARN-425 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0, 0.23.7, 2.1.0-beta >Reporter: Aleksey Gorshkov >Assignee: Aleksey Gorshkov > Attachments: YARN-425-branch-0.23-d.patch, > YARN-425-branch-0.23.patch, YARN-425-branch-0.23-v1.patch, > YARN-425-branch-2-b.patch, YARN-425-branch-2-c.patch, > YARN-425-branch-2.patch, YARN-425-branch-2-v1.patch, YARN-425-trunk-a.patch, > YARN-425-trunk-b.patch, YARN-425-trunk-c.patch, YARN-425-trunk-d.patch, > YARN-425-trunk.patch, YARN-425-trunk-v1.patch > > > coverage fix for yarn api > patch YARN-425-trunk-a.patch for trunk > patch YARN-425-branch-2.patch for branch-2 > patch YARN-425-branch-0.23.patch for branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-425) coverage fix for yarn api
[ https://issues.apache.org/jira/browse/YARN-425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687715#comment-13687715 ] Aleksey Gorshkov commented on YARN-425: --- patches were changed. patch YARN-425-trunk-v1.patch for trunk patch YARN-425-branch-2-v1.patch for branch-2 patch YARN-425-branch-0.23-v1.patch for branch-0.23 > coverage fix for yarn api > - > > Key: YARN-425 > URL: https://issues.apache.org/jira/browse/YARN-425 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0, 0.23.7, 2.1.0-beta >Reporter: Aleksey Gorshkov >Assignee: Aleksey Gorshkov > Attachments: YARN-425-branch-0.23-d.patch, > YARN-425-branch-0.23.patch, YARN-425-branch-0.23-v1.patch, > YARN-425-branch-2-b.patch, YARN-425-branch-2-c.patch, > YARN-425-branch-2.patch, YARN-425-branch-2-v1.patch, YARN-425-trunk-a.patch, > YARN-425-trunk-b.patch, YARN-425-trunk-c.patch, YARN-425-trunk-d.patch, > YARN-425-trunk.patch, YARN-425-trunk-v1.patch > > > coverage fix for yarn api > patch YARN-425-trunk-a.patch for trunk > patch YARN-425-branch-2.patch for branch-2 > patch YARN-425-branch-0.23.patch for branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-425) coverage fix for yarn api
[
https://issues.apache.org/jira/browse/YARN-425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687731#comment-13687731
]
Hadoop QA commented on YARN-425:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588555/YARN-425-trunk-v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1348//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1348//console
This message is automatically generated.
> coverage fix for yarn api
> -
>
> Key: YARN-425
> URL: https://issues.apache.org/jira/browse/YARN-425
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.0.0, 0.23.7, 2.1.0-beta
>Reporter: Aleksey Gorshkov
>Assignee: Aleksey Gorshkov
> Attachments: YARN-425-branch-0.23-d.patch,
> YARN-425-branch-0.23.patch, YARN-425-branch-0.23-v1.patch,
> YARN-425-branch-2-b.patch, YARN-425-branch-2-c.patch,
> YARN-425-branch-2.patch, YARN-425-branch-2-v1.patch, YARN-425-trunk-a.patch,
> YARN-425-trunk-b.patch, YARN-425-trunk-c.patch, YARN-425-trunk-d.patch,
> YARN-425-trunk.patch, YARN-425-trunk-v1.patch
>
>
> coverage fix for yarn api
> patch YARN-425-trunk-a.patch for trunk
> patch YARN-425-branch-2.patch for branch-2
> patch YARN-425-branch-0.23.patch for branch-0.23
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-553) Have YarnClient generate a directly usable ApplicationSubmissionContext
[
https://issues.apache.org/jira/browse/YARN-553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687736#comment-13687736
]
Hadoop QA commented on YARN-553:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588549/yarn-553-7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1345//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1345//console
This message is automatically generated.
> Have YarnClient generate a directly usable ApplicationSubmissionContext
> ---
>
> Key: YARN-553
> URL: https://issues.apache.org/jira/browse/YARN-553
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
>Affects Versions: 2.0.3-alpha
>Reporter: Harsh J
>Assignee: Karthik Kambatla
>Priority: Minor
> Fix For: 2.1.0-beta
>
> Attachments: yarn-553-1.patch, yarn-553-2.patch, yarn-553-3.patch,
> yarn-553-4.patch, yarn-553-5.patch, yarn-553-6.patch, yarn-553-7.patch
>
>
> Right now, we're doing multiple steps to create a relevant
> ApplicationSubmissionContext for a pre-received GetNewApplicationResponse.
> {code}
> GetNewApplicationResponse newApp = yarnClient.getNewApplication();
> ApplicationId appId = newApp.getApplicationId();
> ApplicationSubmissionContext appContext =
> Records.newRecord(ApplicationSubmissionContext.class);
> appContext.setApplicationId(appId);
> {code}
> A simplified way may be to have the GetNewApplicationResponse itself provide
> a helper method that builds a usable ApplicationSubmissionContext for us.
> Something like:
> {code}
> GetNewApplicationResponse newApp = yarnClient.getNewApplication();
> ApplicationSubmissionContext appContext =
> newApp.generateApplicationSubmissionContext();
> {code}
> [The above method can also take an arg for the container launch spec, or
> perhaps pre-load defaults like min-resource, etc. in the returned object,
> aside of just associating the application ID automatically.]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-791) Ensure that RM RPC APIs that return nodes are consistent with /nodes REST API
[ https://issues.apache.org/jira/browse/YARN-791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-791: Attachment: YARN-791-4.patch > Ensure that RM RPC APIs that return nodes are consistent with /nodes REST API > - > > Key: YARN-791 > URL: https://issues.apache.org/jira/browse/YARN-791 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, resourcemanager >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-791-1.patch, YARN-791-2.patch, YARN-791-3.patch, > YARN-791-4.patch, YARN-791.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-791) Ensure that RM RPC APIs that return nodes are consistent with /nodes REST API
[ https://issues.apache.org/jira/browse/YARN-791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687802#comment-13687802 ] Sandy Ryza commented on YARN-791: - Attaching patch that makes both the REST API and the RPC API accept multiple node states. > Ensure that RM RPC APIs that return nodes are consistent with /nodes REST API > - > > Key: YARN-791 > URL: https://issues.apache.org/jira/browse/YARN-791 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, resourcemanager >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-791-1.patch, YARN-791-2.patch, YARN-791-3.patch, > YARN-791-4.patch, YARN-791.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-791) Ensure that RM RPC APIs that return nodes are consistent with /nodes REST API
[
https://issues.apache.org/jira/browse/YARN-791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687838#comment-13687838
]
Hadoop QA commented on YARN-791:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588564/YARN-791-4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.client.cli.TestYarnCLI
org.apache.hadoop.yarn.client.api.impl.TestNMClient
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1349//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1349//console
This message is automatically generated.
> Ensure that RM RPC APIs that return nodes are consistent with /nodes REST API
> -
>
> Key: YARN-791
> URL: https://issues.apache.org/jira/browse/YARN-791
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, resourcemanager
>Affects Versions: 2.0.4-alpha
>Reporter: Sandy Ryza
>Assignee: Sandy Ryza
> Attachments: YARN-791-1.patch, YARN-791-2.patch, YARN-791-3.patch,
> YARN-791-4.patch, YARN-791.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-848) Nodemanager does not register with RM using the fully qualified hostname
[ https://issues.apache.org/jira/browse/YARN-848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687847#comment-13687847 ] Hudson commented on YARN-848: - Integrated in Hadoop-Yarn-trunk #245 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/245/]) YARN-848. Fix NodeManager to register with RM using the fully qualified hostname. Contributed by Hitesh Shah. (Revision 1494385) Result = FAILURE sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494385 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java > Nodemanager does not register with RM using the fully qualified hostname > > > Key: YARN-848 > URL: https://issues.apache.org/jira/browse/YARN-848 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah > Fix For: 2.1.0-beta > > Attachments: YARN-848.1.patch, YARN-848.3.patch > > > If the hostname is misconfigured to not be fully qualified ( i.e. hostname > returns foo and hostname -f returns foo.bar.xyz ), the NM ends up registering > with the RM using only "foo". This can create problems if DNS cannot resolve > the hostname properly. > Furthermore, HDFS uses fully qualified hostnames which can end up affecting > locality matches when allocating containers based on block locations. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-850) Rename getClusterAvailableResources to getAvailableResources in AMRMClients
[ https://issues.apache.org/jira/browse/YARN-850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687850#comment-13687850 ] Hudson commented on YARN-850: - Integrated in Hadoop-Yarn-trunk #245 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/245/]) YARN-850. Rename getClusterAvailableResources to getAvailableResources in AMRMClients (Jian He via bikas) (Revision 1494309) Result = FAILURE bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494309 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/AMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/impl/AMRMClientAsyncImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/async/impl/TestAMRMClientAsync.java > Rename getClusterAvailableResources to getAvailableResources in AMRMClients > --- > > Key: YARN-850 > URL: https://issues.apache.org/jira/browse/YARN-850 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.1.0-beta >Reporter: Jian He >Assignee: Jian He > Fix For: 2.1.0-beta > > Attachments: YARN-850.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-694) Start using NMTokens to authenticate all communication with NM
[ https://issues.apache.org/jira/browse/YARN-694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687851#comment-13687851 ] Hudson commented on YARN-694: - Integrated in Hadoop-Yarn-trunk #245 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/245/]) YARN-694. Starting to use NMTokens to authenticate all communication with NodeManagers. Contributed by Omkar Vinit Joshi. (Revision 1494369) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494369 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFail.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/NMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/NMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/impl/NMClientAsyncImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/ContainerManagementProtocolProxy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/NMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/async/impl/TestNMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestNMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ContainerManagerSecurityInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/NMTokenIdentifier.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/NMTokenSelector.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/BaseNMTokenSecretManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop
[jira] [Commented] (YARN-553) Have YarnClient generate a directly usable ApplicationSubmissionContext
[
https://issues.apache.org/jira/browse/YARN-553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687848#comment-13687848
]
Hudson commented on YARN-553:
-
Integrated in Hadoop-Yarn-trunk #245 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/245/])
YARN-553. Replaced YarnClient.getNewApplication with
YarnClient.createApplication which provides a directly usable
ApplicationSubmissionContext to simplify the api. Contributed by Karthik
Kambatla. (Revision 1494476)
Result = FAILURE
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494476
Files :
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClientApplication.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestNMClient.java
> Have YarnClient generate a directly usable ApplicationSubmissionContext
> ---
>
> Key: YARN-553
> URL: https://issues.apache.org/jira/browse/YARN-553
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
>Affects Versions: 2.0.3-alpha
>Reporter: Harsh J
>Assignee: Karthik Kambatla
>Priority: Minor
> Fix For: 2.1.0-beta
>
> Attachments: yarn-553-1.patch, yarn-553-2.patch, yarn-553-3.patch,
> yarn-553-4.patch, yarn-553-5.patch, yarn-553-6.patch, yarn-553-7.patch
>
>
> Right now, we're doing multiple steps to create a relevant
> ApplicationSubmissionContext for a pre-received GetNewApplicationResponse.
> {code}
> GetNewApplicationResponse newApp = yarnClient.getNewApplication();
> ApplicationId appId = newApp.getApplicationId();
> ApplicationSubmissionContext appContext =
> Records.newRecord(ApplicationSubmissionContext.class);
> appContext.setApplicationId(appId);
> {code}
> A simplified way may be to have the GetNewApplicationResponse itself provide
> a helper method that builds a usable ApplicationSubmissionContext for us.
> Something like:
> {code}
> GetNewApplicationResponse newApp = yarnClient.getNewApplication();
> ApplicationSubmissionContext appContext =
> newApp.generateApplicationSubmissionContext();
> {code}
> [The above method can also take an arg for the container launch spec, or
> perhaps pre-load defaults like min-resource, etc. in the returned object,
> aside of just associating the application ID automatically.]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-850) Rename getClusterAvailableResources to getAvailableResources in AMRMClients
[ https://issues.apache.org/jira/browse/YARN-850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687951#comment-13687951 ] Hudson commented on YARN-850: - Integrated in Hadoop-Hdfs-trunk #1435 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1435/]) YARN-850. Rename getClusterAvailableResources to getAvailableResources in AMRMClients (Jian He via bikas) (Revision 1494309) Result = FAILURE bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494309 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/AMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/impl/AMRMClientAsyncImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/async/impl/TestAMRMClientAsync.java > Rename getClusterAvailableResources to getAvailableResources in AMRMClients > --- > > Key: YARN-850 > URL: https://issues.apache.org/jira/browse/YARN-850 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.1.0-beta >Reporter: Jian He >Assignee: Jian He > Fix For: 2.1.0-beta > > Attachments: YARN-850.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-553) Have YarnClient generate a directly usable ApplicationSubmissionContext
[
https://issues.apache.org/jira/browse/YARN-553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687949#comment-13687949
]
Hudson commented on YARN-553:
-
Integrated in Hadoop-Hdfs-trunk #1435 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1435/])
YARN-553. Replaced YarnClient.getNewApplication with
YarnClient.createApplication which provides a directly usable
ApplicationSubmissionContext to simplify the api. Contributed by Karthik
Kambatla. (Revision 1494476)
Result = FAILURE
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494476
Files :
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClientApplication.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestNMClient.java
> Have YarnClient generate a directly usable ApplicationSubmissionContext
> ---
>
> Key: YARN-553
> URL: https://issues.apache.org/jira/browse/YARN-553
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
>Affects Versions: 2.0.3-alpha
>Reporter: Harsh J
>Assignee: Karthik Kambatla
>Priority: Minor
> Fix For: 2.1.0-beta
>
> Attachments: yarn-553-1.patch, yarn-553-2.patch, yarn-553-3.patch,
> yarn-553-4.patch, yarn-553-5.patch, yarn-553-6.patch, yarn-553-7.patch
>
>
> Right now, we're doing multiple steps to create a relevant
> ApplicationSubmissionContext for a pre-received GetNewApplicationResponse.
> {code}
> GetNewApplicationResponse newApp = yarnClient.getNewApplication();
> ApplicationId appId = newApp.getApplicationId();
> ApplicationSubmissionContext appContext =
> Records.newRecord(ApplicationSubmissionContext.class);
> appContext.setApplicationId(appId);
> {code}
> A simplified way may be to have the GetNewApplicationResponse itself provide
> a helper method that builds a usable ApplicationSubmissionContext for us.
> Something like:
> {code}
> GetNewApplicationResponse newApp = yarnClient.getNewApplication();
> ApplicationSubmissionContext appContext =
> newApp.generateApplicationSubmissionContext();
> {code}
> [The above method can also take an arg for the container launch spec, or
> perhaps pre-load defaults like min-resource, etc. in the returned object,
> aside of just associating the application ID automatically.]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-848) Nodemanager does not register with RM using the fully qualified hostname
[ https://issues.apache.org/jira/browse/YARN-848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687948#comment-13687948 ] Hudson commented on YARN-848: - Integrated in Hadoop-Hdfs-trunk #1435 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1435/]) YARN-848. Fix NodeManager to register with RM using the fully qualified hostname. Contributed by Hitesh Shah. (Revision 1494385) Result = FAILURE sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494385 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java > Nodemanager does not register with RM using the fully qualified hostname > > > Key: YARN-848 > URL: https://issues.apache.org/jira/browse/YARN-848 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah > Fix For: 2.1.0-beta > > Attachments: YARN-848.1.patch, YARN-848.3.patch > > > If the hostname is misconfigured to not be fully qualified ( i.e. hostname > returns foo and hostname -f returns foo.bar.xyz ), the NM ends up registering > with the RM using only "foo". This can create problems if DNS cannot resolve > the hostname properly. > Furthermore, HDFS uses fully qualified hostnames which can end up affecting > locality matches when allocating containers based on block locations. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-694) Start using NMTokens to authenticate all communication with NM
[ https://issues.apache.org/jira/browse/YARN-694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687952#comment-13687952 ] Hudson commented on YARN-694: - Integrated in Hadoop-Hdfs-trunk #1435 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1435/]) YARN-694. Starting to use NMTokens to authenticate all communication with NodeManagers. Contributed by Omkar Vinit Joshi. (Revision 1494369) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494369 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFail.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/NMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/NMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/impl/NMClientAsyncImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/ContainerManagementProtocolProxy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/NMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/async/impl/TestNMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestNMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ContainerManagerSecurityInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/NMTokenIdentifier.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/NMTokenSelector.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/BaseNMTokenSecretManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hado
[jira] [Created] (YARN-853) maximum-am-resource-percent doesn't work consistently with refreshQueues
Devaraj K created YARN-853: -- Summary: maximum-am-resource-percent doesn't work consistently with refreshQueues Key: YARN-853 URL: https://issues.apache.org/jira/browse/YARN-853 Project: Hadoop YARN Issue Type: Bug Components: capacityscheduler Affects Versions: 3.0.0, 2.1.0-beta, 2.0.5-alpha Reporter: Devaraj K Assignee: Devaraj K If we update yarn.scheduler.capacity.maximum-am-resource-percent / yarn.scheduler.capacity..maximum-am-resource-percent configuration and then do the refreshNodes, it uses the new config value to calculate Max Active Applications and Max Active Application Per User. If we add new node after issuing 'rmadmin -refreshQueues' command, it uses the old maximum-am-resource-percent config value to calculate Max Active Applications and Max Active Application Per User. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-694) Start using NMTokens to authenticate all communication with NM
[ https://issues.apache.org/jira/browse/YARN-694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687997#comment-13687997 ] Hudson commented on YARN-694: - Integrated in Hadoop-Mapreduce-trunk #1462 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1462/]) YARN-694. Starting to use NMTokens to authenticate all communication with NodeManagers. Contributed by Omkar Vinit Joshi. (Revision 1494369) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494369 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFail.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/NMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/NMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/impl/NMClientAsyncImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/ContainerManagementProtocolProxy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/NMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/async/impl/TestNMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestNMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ContainerManagerSecurityInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/NMTokenIdentifier.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/NMTokenSelector.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/BaseNMTokenSecretManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-s
[jira] [Commented] (YARN-850) Rename getClusterAvailableResources to getAvailableResources in AMRMClients
[ https://issues.apache.org/jira/browse/YARN-850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687996#comment-13687996 ] Hudson commented on YARN-850: - Integrated in Hadoop-Mapreduce-trunk #1462 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1462/]) YARN-850. Rename getClusterAvailableResources to getAvailableResources in AMRMClients (Jian He via bikas) (Revision 1494309) Result = FAILURE bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494309 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/AMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/impl/AMRMClientAsyncImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/async/impl/TestAMRMClientAsync.java > Rename getClusterAvailableResources to getAvailableResources in AMRMClients > --- > > Key: YARN-850 > URL: https://issues.apache.org/jira/browse/YARN-850 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.1.0-beta >Reporter: Jian He >Assignee: Jian He > Fix For: 2.1.0-beta > > Attachments: YARN-850.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-848) Nodemanager does not register with RM using the fully qualified hostname
[ https://issues.apache.org/jira/browse/YARN-848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687993#comment-13687993 ] Hudson commented on YARN-848: - Integrated in Hadoop-Mapreduce-trunk #1462 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1462/]) YARN-848. Fix NodeManager to register with RM using the fully qualified hostname. Contributed by Hitesh Shah. (Revision 1494385) Result = FAILURE sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494385 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java > Nodemanager does not register with RM using the fully qualified hostname > > > Key: YARN-848 > URL: https://issues.apache.org/jira/browse/YARN-848 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah > Fix For: 2.1.0-beta > > Attachments: YARN-848.1.patch, YARN-848.3.patch > > > If the hostname is misconfigured to not be fully qualified ( i.e. hostname > returns foo and hostname -f returns foo.bar.xyz ), the NM ends up registering > with the RM using only "foo". This can create problems if DNS cannot resolve > the hostname properly. > Furthermore, HDFS uses fully qualified hostnames which can end up affecting > locality matches when allocating containers based on block locations. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-553) Have YarnClient generate a directly usable ApplicationSubmissionContext
[
https://issues.apache.org/jira/browse/YARN-553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687994#comment-13687994
]
Hudson commented on YARN-553:
-
Integrated in Hadoop-Mapreduce-trunk #1462 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1462/])
YARN-553. Replaced YarnClient.getNewApplication with
YarnClient.createApplication which provides a directly usable
ApplicationSubmissionContext to simplify the api. Contributed by Karthik
Kambatla. (Revision 1494476)
Result = FAILURE
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494476
Files :
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClientApplication.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestNMClient.java
> Have YarnClient generate a directly usable ApplicationSubmissionContext
> ---
>
> Key: YARN-553
> URL: https://issues.apache.org/jira/browse/YARN-553
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
>Affects Versions: 2.0.3-alpha
>Reporter: Harsh J
>Assignee: Karthik Kambatla
>Priority: Minor
> Fix For: 2.1.0-beta
>
> Attachments: yarn-553-1.patch, yarn-553-2.patch, yarn-553-3.patch,
> yarn-553-4.patch, yarn-553-5.patch, yarn-553-6.patch, yarn-553-7.patch
>
>
> Right now, we're doing multiple steps to create a relevant
> ApplicationSubmissionContext for a pre-received GetNewApplicationResponse.
> {code}
> GetNewApplicationResponse newApp = yarnClient.getNewApplication();
> ApplicationId appId = newApp.getApplicationId();
> ApplicationSubmissionContext appContext =
> Records.newRecord(ApplicationSubmissionContext.class);
> appContext.setApplicationId(appId);
> {code}
> A simplified way may be to have the GetNewApplicationResponse itself provide
> a helper method that builds a usable ApplicationSubmissionContext for us.
> Something like:
> {code}
> GetNewApplicationResponse newApp = yarnClient.getNewApplication();
> ApplicationSubmissionContext appContext =
> newApp.generateApplicationSubmissionContext();
> {code}
> [The above method can also take an arg for the container launch spec, or
> perhaps pre-load defaults like min-resource, etc. in the returned object,
> aside of just associating the application ID automatically.]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-694) Start using NMTokens to authenticate all communication with NM
[ https://issues.apache.org/jira/browse/YARN-694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688083#comment-13688083 ] Jason Lowe commented on YARN-694: - This broke TestContainerLauncherImpl -- it's now hanging and failing builds. Jenkins gave a +1 to the patch because it ignores build failures, search for FAIL in the console output at https://builds.apache.org/job/PreCommit-YARN-Build/1334//console. As an aside, sure wish someone would commit HADOOP-9583 or an equivalent fix so we can avoid accidentally checking in test timeouts that Jenkins should have caught. Please reopen and fix or I can file a new JIRA if desired. While we're at it, since this test in particular has failed a number of times, we should add a timeout argument to the @Test annotation to help avoid build failures if/when the test breaks in the future. > Start using NMTokens to authenticate all communication with NM > -- > > Key: YARN-694 > URL: https://issues.apache.org/jira/browse/YARN-694 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi > Fix For: 2.1.0-beta > > Attachments: YARN-694-20130613.patch, YARN-694-20130617.1.patch, > YARN-694-20130617.2.patch, YARN-694-20130617.patch, > YARN-694-20130618.1.patch, YARN-694-20130618.2.patch, > YARN-694-20130618.3.patch, YARN-694-20130618.4.patch, > YARN-694-20130618.5.patch, YARN-694-20130618.patch.branch-2, > YARN-694-20130618.patch.yarn-694-branch-2.1-beta > > > AM uses the NMToken to authenticate all the AM-NM communication. > NM will validate NMToken in below manner > * If NMToken is using current or previous master key then the NMToken is > valid. In this case it will update its cache with this key corresponding to > appId. > * If NMToken is using the master key which is present in NM's cache > corresponding to AM's appId then it will be validated based on this. > * If NMToken is invalid then NM will reject AM calls. > Modification for ContainerToken > * At present RPC validates AM-NM communication based on ContainerToken. It > will be replaced with NMToken. Also now onwards AM will use NMToken per NM > (replacing earlier behavior of ContainerToken per container per NM). > * startContainer in case of Secured environment is using ContainerToken from > UGI YARN-617; however after this it will use it from the payload (Container). > * ContainerToken will exist and it will only be used to validate the AM's > container start request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-791) Ensure that RM RPC APIs that return nodes are consistent with /nodes REST API
[ https://issues.apache.org/jira/browse/YARN-791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688182#comment-13688182 ] Alejandro Abdelnur commented on YARN-791: - I think we should have consistency between the HTTP and JAVA API (and at ProtoBuffer if possible): Over HTTP if we don't specify a state we get RUNNING only and we can specify multiple state values separated by commas Over Java we can achieve the exact same behavior by using a a var arg (State ... states), if called with () would be RUNNING only. And, as ProtoBuffer is taking a list, so if the list is empty we get RUNNING. > Ensure that RM RPC APIs that return nodes are consistent with /nodes REST API > - > > Key: YARN-791 > URL: https://issues.apache.org/jira/browse/YARN-791 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, resourcemanager >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-791-1.patch, YARN-791-2.patch, YARN-791-3.patch, > YARN-791-4.patch, YARN-791.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688189#comment-13688189 ] Karthik Kambatla commented on YARN-736: --- Thanks Sandy. Yes, it should be okay to handle #2 in a separate JIRA. Nits: # AppSchedulable#getMaxShare(): instead of creating a resource with max values every time, we should create a Resource object on the first call and return it from then on. # There seem to be several constructors for FakeSchedulable. Can we git rid of any that are not used anywhere. Ignore if all of them are used. > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-727) ClientRMProtocol.getAllApplications should accept ApplicationType as a parameter
[ https://issues.apache.org/jira/browse/YARN-727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-727: --- Attachment: YARN-727.14.patch 1. getApplications API accept a set of appTypes as a parameter 2. parse the command line -list --appTypes=, to display application reports for different appTypes 3. we get null value if we parse commandLine -list, so at ApplicationCLI.java we still check null condition > ClientRMProtocol.getAllApplications should accept ApplicationType as a > parameter > > > Key: YARN-727 > URL: https://issues.apache.org/jira/browse/YARN-727 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.1.0-beta >Reporter: Siddharth Seth >Assignee: Xuan Gong > Attachments: YARN-727.10.patch, YARN-727.11.patch, YARN-727.12.patch, > YARN-727.13.patch, YARN-727.14.patch, YARN-727.1.patch, YARN-727.2.patch, > YARN-727.3.patch, YARN-727.4.patch, YARN-727.5.patch, YARN-727.6.patch, > YARN-727.7.patch, YARN-727.8.patch, YARN-727.9.patch > > > Now that an ApplicationType is registered on ApplicationSubmission, > getAllApplications should be able to use this string to query for a specific > application type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-851) Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently.
[ https://issues.apache.org/jira/browse/YARN-851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688200#comment-13688200 ] Bikas Saha commented on YARN-851: - If an interface method is not going to be used then we are better of not exposing it. > Share NMTokens using NMTokenCache (api-based) instead of memory based > approach which is used currently. > --- > > Key: YARN-851 > URL: https://issues.apache.org/jira/browse/YARN-851 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi > Attachments: YARN-851-20130618.patch > > > It is a follow up ticket for YARN-694. Changing the way NMTokens are shared. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-851) Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently.
[
https://issues.apache.org/jira/browse/YARN-851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688199#comment-13688199
]
Bikas Saha commented on YARN-851:
-
This loop changes doesnt quite look the same as earlier. The old code is
looping until the token identifier matches. The token would get updated when
the map would get updated. The new code will also loop until the token
identifier matches but the token object reference is not being updated. So it
the identifier does not match then the loop will not exit. Which one is correct?
{code}
-
+Token token = NMTokenCache.getNMToken(containerManagerBindAddr);
+
+if (token == null) {
+ throw new InvalidToken("No NMToken sent for "
+ + containerManagerBindAddr);
+}
+
while (proxy != null
-&& !proxy.token.getIdentifier().equals(
-nmTokens.get(containerManagerBindAddr).getIdentifier())) {
+&& !proxy.token.getIdentifier().equals(token.getIdentifier())) {
{code}
Why are tokens being removed from the TokenCache in the test?
{code}
- nodeI = receivedNMTokens.keySet().iterator();
- while (nodeI.hasNext()) {
-nmTokens.remove(nodeI.next());
+NMTokenCache.removeNMToken(nodeID);
+receivedNMTokens.put(nodeID, token.getToken());
{code}
Overall comment of NMTokenCache is that its mixing up static and singleton for
full functionality. If it has an internal private single object then that
object should take care of any required synchronization etc and not the static
class method. Alternatively, one can make the internal map a private class
object and not have an internal new NMTokenCache object.
We might be better off using a concurrent map implementation instead of coarse
synchronization on the NMTokenCache method level.
Are these API meant to be public and for users? What scenario? Typo in comment.
{code}
+ /**
+ * Removes NMToken for specified node manger
+ * @param nodeAddr node address (host:port)
+ */
+ public static synchronized void removeNMToken(String nodeAddr) {
+instance.nmTokens.remove(nodeAddr);
+ }
+
+ /**
+ * It will remove all the nm tokens from its cache
+ */
+ public static synchronized void clearCache() {
+instance.nmTokens.clear();
+ }
{code}
Is this meant to be public for users? It should probably just return a java
unmodifiable map wrapped around the internal and mention in the java doc that
it returns an unmodifiable map. Unless its really required to create a snapshot
of the token cache at that point in time.
{code}
+ /**
+ * It returns all NMTokens present in cache.
+ */
+ public static synchronized Map getAllNMTokens() {
+Map nmTokens = new HashMap();
+for (String key : instance.nmTokens.keySet()) {
+ nmTokens.put(key, instance.nmTokens.get(key));
+}
+return nmTokens;
+ }
{code}
> Share NMTokens using NMTokenCache (api-based) instead of memory based
> approach which is used currently.
> ---
>
> Key: YARN-851
> URL: https://issues.apache.org/jira/browse/YARN-851
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Omkar Vinit Joshi
>Assignee: Omkar Vinit Joshi
> Attachments: YARN-851-20130618.patch
>
>
> It is a follow up ticket for YARN-694. Changing the way NMTokens are shared.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688204#comment-13688204 ] Sandy Ryza commented on YARN-736: - All the FakeSchedulable constructors appear to be used. Uploaded a patch in which AppSchedulable#getMaxShare does not create a new Resource each time. > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736-3.patch, > YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-736: Attachment: YARN-736-3.patch > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736-3.patch, > YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-569) CapacityScheduler: support for preemption (using a capacity monitor)
[ https://issues.apache.org/jira/browse/YARN-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688233#comment-13688233 ] Bikas Saha commented on YARN-569: - bq. You're saying the block updating the responseMap probably belongs just before the return? That makes sense, though I haven't traced it explicitly. Yes. > CapacityScheduler: support for preemption (using a capacity monitor) > > > Key: YARN-569 > URL: https://issues.apache.org/jira/browse/YARN-569 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: 3queues.pdf, CapScheduler_with_preemption.pdf, > preemption.2.patch, YARN-569.1.patch, YARN-569.2.patch, YARN-569.3.patch, > YARN-569.4.patch, YARN-569.5.patch, YARN-569.6.patch, YARN-569.8.patch, > YARN-569.patch, YARN-569.patch > > > There is a tension between the fast-pace reactive role of the > CapacityScheduler, which needs to respond quickly to > applications resource requests, and node updates, and the more introspective, > time-based considerations > needed to observe and correct for capacity balance. To this purpose we opted > instead of hacking the delicate > mechanisms of the CapacityScheduler directly to add support for preemption by > means of a "Capacity Monitor", > which can be run optionally as a separate service (much like the > NMLivelinessMonitor). > The capacity monitor (similarly to equivalent functionalities in the fairness > scheduler) operates running on intervals > (e.g., every 3 seconds), observe the state of the assignment of resources to > queues from the capacity scheduler, > performs off-line computation to determine if preemption is needed, and how > best to "edit" the current schedule to > improve capacity, and generates events that produce four possible actions: > # Container de-reservations > # Resource-based preemptions > # Container-based preemptions > # Container killing > The actions listed above are progressively more costly, and it is up to the > policy to use them as desired to achieve the rebalancing goals. > Note that due to the "lag" in the effect of these actions the policy should > operate at the macroscopic level (e.g., preempt tens of containers > from a queue) and not trying to tightly and consistently micromanage > container allocations. > - Preemption policy (ProportionalCapacityPreemptionPolicy): > - > Preemption policies are by design pluggable, in the following we present an > initial policy (ProportionalCapacityPreemptionPolicy) we have been > experimenting with. The ProportionalCapacityPreemptionPolicy behaves as > follows: > # it gathers from the scheduler the state of the queues, in particular, their > current capacity, guaranteed capacity and pending requests (*) > # if there are pending requests from queues that are under capacity it > computes a new ideal balanced state (**) > # it computes the set of preemptions needed to repair the current schedule > and achieve capacity balance (accounting for natural completion rates, and > respecting bounds on the amount of preemption we allow for each round) > # it selects which applications to preempt from each over-capacity queue (the > last one in the FIFO order) > # it remove reservations from the most recently assigned app until the amount > of resource to reclaim is obtained, or until no more reservations exits > # (if not enough) it issues preemptions for containers from the same > applications (reverse chronological order, last assigned container first) > again until necessary or until no containers except the AM container are left, > # (if not enough) it moves onto unreserve and preempt from the next > application. > # containers that have been asked to preempt are tracked across executions. > If a containers is among the one to be preempted for more than a certain > time, the container is moved in a the list of containers to be forcibly > killed. > Notes: > (*) at the moment, in order to avoid double-counting of the requests, we only > look at the "ANY" part of pending resource requests, which means we might not > preempt on behalf of AMs that ask only for specific locations but not any. > (**) The ideal balance state is one in which each queue has at least its > guaranteed capacity, and the spare capacity is distributed among queues (that > wants some) as a weighted fair share. Where the weighting is based on the > guaranteed capacity of a queue, and the function runs to a fix point. > Tunables of the ProportionalCapacityPreemptionPolicy: > # observe-only mode (i.e., log the actions it would take, but behave as > read-only) > # how frequently to run the policy > # how lo
[jira] [Commented] (YARN-569) CapacityScheduler: support for preemption (using a capacity monitor)
[ https://issues.apache.org/jira/browse/YARN-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688237#comment-13688237 ] Bikas Saha commented on YARN-569: - Since configuring this involves more than 1 config file, as long as its clear which file to change for which config, its all good. > CapacityScheduler: support for preemption (using a capacity monitor) > > > Key: YARN-569 > URL: https://issues.apache.org/jira/browse/YARN-569 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: 3queues.pdf, CapScheduler_with_preemption.pdf, > preemption.2.patch, YARN-569.1.patch, YARN-569.2.patch, YARN-569.3.patch, > YARN-569.4.patch, YARN-569.5.patch, YARN-569.6.patch, YARN-569.8.patch, > YARN-569.patch, YARN-569.patch > > > There is a tension between the fast-pace reactive role of the > CapacityScheduler, which needs to respond quickly to > applications resource requests, and node updates, and the more introspective, > time-based considerations > needed to observe and correct for capacity balance. To this purpose we opted > instead of hacking the delicate > mechanisms of the CapacityScheduler directly to add support for preemption by > means of a "Capacity Monitor", > which can be run optionally as a separate service (much like the > NMLivelinessMonitor). > The capacity monitor (similarly to equivalent functionalities in the fairness > scheduler) operates running on intervals > (e.g., every 3 seconds), observe the state of the assignment of resources to > queues from the capacity scheduler, > performs off-line computation to determine if preemption is needed, and how > best to "edit" the current schedule to > improve capacity, and generates events that produce four possible actions: > # Container de-reservations > # Resource-based preemptions > # Container-based preemptions > # Container killing > The actions listed above are progressively more costly, and it is up to the > policy to use them as desired to achieve the rebalancing goals. > Note that due to the "lag" in the effect of these actions the policy should > operate at the macroscopic level (e.g., preempt tens of containers > from a queue) and not trying to tightly and consistently micromanage > container allocations. > - Preemption policy (ProportionalCapacityPreemptionPolicy): > - > Preemption policies are by design pluggable, in the following we present an > initial policy (ProportionalCapacityPreemptionPolicy) we have been > experimenting with. The ProportionalCapacityPreemptionPolicy behaves as > follows: > # it gathers from the scheduler the state of the queues, in particular, their > current capacity, guaranteed capacity and pending requests (*) > # if there are pending requests from queues that are under capacity it > computes a new ideal balanced state (**) > # it computes the set of preemptions needed to repair the current schedule > and achieve capacity balance (accounting for natural completion rates, and > respecting bounds on the amount of preemption we allow for each round) > # it selects which applications to preempt from each over-capacity queue (the > last one in the FIFO order) > # it remove reservations from the most recently assigned app until the amount > of resource to reclaim is obtained, or until no more reservations exits > # (if not enough) it issues preemptions for containers from the same > applications (reverse chronological order, last assigned container first) > again until necessary or until no containers except the AM container are left, > # (if not enough) it moves onto unreserve and preempt from the next > application. > # containers that have been asked to preempt are tracked across executions. > If a containers is among the one to be preempted for more than a certain > time, the container is moved in a the list of containers to be forcibly > killed. > Notes: > (*) at the moment, in order to avoid double-counting of the requests, we only > look at the "ANY" part of pending resource requests, which means we might not > preempt on behalf of AMs that ask only for specific locations but not any. > (**) The ideal balance state is one in which each queue has at least its > guaranteed capacity, and the spare capacity is distributed among queues (that > wants some) as a weighted fair share. Where the weighting is based on the > guaranteed capacity of a queue, and the function runs to a fix point. > Tunables of the ProportionalCapacityPreemptionPolicy: > # observe-only mode (i.e., log the actions it would take, but behave as > read-only) > # how frequently to run the policy > # how long to wait between preem
[jira] [Updated] (YARN-597) TestFSDownload fails on Windows because of dependencies on tar/gzip/jar tools
[
https://issues.apache.org/jira/browse/YARN-597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated YARN-597:
---
Fix Version/s: 2.1.0-beta
Hadoop Flags: Reviewed
I merged this to branch-2 and branch-2.1-beta and updated CHANGES.txt in trunk
to move attribution from the trunk section to release 2.1.0-beta.
> TestFSDownload fails on Windows because of dependencies on tar/gzip/jar tools
> -
>
> Key: YARN-597
> URL: https://issues.apache.org/jira/browse/YARN-597
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ivan Mitic
>Assignee: Ivan Mitic
> Fix For: 3.0.0, 2.1.0-beta
>
> Attachments: YARN-597.patch
>
>
> {{testDownloadArchive}}, {{testDownloadPatternJar}} and
> {{testDownloadArchiveZip}} fail with the similar Shell ExitCodeException:
> {code}
> testDownloadArchiveZip(org.apache.hadoop.yarn.util.TestFSDownload) Time
> elapsed: 480 sec <<< ERROR!
> org.apache.hadoop.util.Shell$ExitCodeException: bash: line 0: cd:
> /D:/svn/t/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/TestFSDownload:
> No such file or directory
> gzip: 1: No such file or directory
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:377)
> at org.apache.hadoop.util.Shell.run(Shell.java:292)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:497)
> at
> org.apache.hadoop.yarn.util.TestFSDownload.createZipFile(TestFSDownload.java:225)
> at
> org.apache.hadoop.yarn.util.TestFSDownload.testDownloadArchiveZip(TestFSDownload.java:503)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[
https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688244#comment-13688244
]
Hadoop QA commented on YARN-736:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588643/YARN-736-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1351//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1351//console
This message is automatically generated.
> Add a multi-resource fair sharing metric
>
>
> Key: YARN-736
> URL: https://issues.apache.org/jira/browse/YARN-736
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler
>Affects Versions: 2.0.4-alpha
>Reporter: Sandy Ryza
>Assignee: Sandy Ryza
> Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736-3.patch,
> YARN-736.patch
>
>
> Currently, at a regular interval, the fair scheduler computes a fair memory
> share for each queue and application inside it. This fair share is not used
> for scheduling decisions, but is displayed in the web UI, exposed as a
> metric, and used for preemption decisions.
> With DRF and multi-resource scheduling, assigning a memory share as the fair
> share metric to every queue no longer makes sense. It's not obvious what the
> replacement should be, but probably something like fractional fairness within
> a queue, or distance from an ideal cluster state.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-597) TestFSDownload fails on Windows because of dependencies on tar/gzip/jar tools
[
https://issues.apache.org/jira/browse/YARN-597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688253#comment-13688253
]
Hudson commented on YARN-597:
-
Integrated in Hadoop-trunk-Commit #3983 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/3983/])
YARN-597. Change attribution of YARN-597 from trunk to release 2.1.0-beta
in CHANGES.txt. (cnauroth) (Revision 1494717)
Result = SUCCESS
cnauroth :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494717
Files :
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
> TestFSDownload fails on Windows because of dependencies on tar/gzip/jar tools
> -
>
> Key: YARN-597
> URL: https://issues.apache.org/jira/browse/YARN-597
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ivan Mitic
>Assignee: Ivan Mitic
> Fix For: 3.0.0, 2.1.0-beta
>
> Attachments: YARN-597.patch
>
>
> {{testDownloadArchive}}, {{testDownloadPatternJar}} and
> {{testDownloadArchiveZip}} fail with the similar Shell ExitCodeException:
> {code}
> testDownloadArchiveZip(org.apache.hadoop.yarn.util.TestFSDownload) Time
> elapsed: 480 sec <<< ERROR!
> org.apache.hadoop.util.Shell$ExitCodeException: bash: line 0: cd:
> /D:/svn/t/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/TestFSDownload:
> No such file or directory
> gzip: 1: No such file or directory
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:377)
> at org.apache.hadoop.util.Shell.run(Shell.java:292)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:497)
> at
> org.apache.hadoop.yarn.util.TestFSDownload.createZipFile(TestFSDownload.java:225)
> at
> org.apache.hadoop.yarn.util.TestFSDownload.testDownloadArchiveZip(TestFSDownload.java:503)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688254#comment-13688254 ] Karthik Kambatla commented on YARN-736: --- Having an UNBOUNDED resource in Resources can be handy, even though having a local copy in AppSchedulable should be enough for this JIRA. +1 to the yarn-736-3.patch. > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736-3.patch, > YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-852) TestAggregatedLogFormat.testContainerLogsFileAccess fails on Windows
[ https://issues.apache.org/jira/browse/YARN-852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated YARN-852: --- Target Version/s: 3.0.0, 2.1.0-beta Hadoop Flags: Reviewed +1 for the patch. I verified the test on Mac and Windows. Thanks, Chuan! I'll commit this shortly. > TestAggregatedLogFormat.testContainerLogsFileAccess fails on Windows > > > Key: YARN-852 > URL: https://issues.apache.org/jira/browse/YARN-852 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Chuan Liu >Assignee: Chuan Liu >Priority: Minor > Attachments: YARN-852-trunk.patch, YARN-852-trunk.patch > > > The YARN unit test case fails on Windows when comparing expected message with > log message in the file. The expected message constructed in the test case > has two problems: 1) it uses Path.separator to concatenate path string. > Path.separator is always a forward slash, which does not match the backslash > used in the log message. 2) On Windows, the default file owner is > Administrators group if the file is created by an Administrators user. The > test expect the user to be the current user. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-569) CapacityScheduler: support for preemption (using a capacity monitor)
[ https://issues.apache.org/jira/browse/YARN-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688272#comment-13688272 ] Bikas Saha commented on YARN-569: - One other thing to check would be if the preemption policy will use refreshed values when the capacity scheduler config is refreshed on the fly. Looks like cloneQueues() will take the absolute used and guaranteed numbers on every clone. So we should be good wrt that. Would be good to check other values the policy looks at. Noticed formatting issues with spaces in the patch. eg. cloneQueues() > CapacityScheduler: support for preemption (using a capacity monitor) > > > Key: YARN-569 > URL: https://issues.apache.org/jira/browse/YARN-569 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: 3queues.pdf, CapScheduler_with_preemption.pdf, > preemption.2.patch, YARN-569.1.patch, YARN-569.2.patch, YARN-569.3.patch, > YARN-569.4.patch, YARN-569.5.patch, YARN-569.6.patch, YARN-569.8.patch, > YARN-569.patch, YARN-569.patch > > > There is a tension between the fast-pace reactive role of the > CapacityScheduler, which needs to respond quickly to > applications resource requests, and node updates, and the more introspective, > time-based considerations > needed to observe and correct for capacity balance. To this purpose we opted > instead of hacking the delicate > mechanisms of the CapacityScheduler directly to add support for preemption by > means of a "Capacity Monitor", > which can be run optionally as a separate service (much like the > NMLivelinessMonitor). > The capacity monitor (similarly to equivalent functionalities in the fairness > scheduler) operates running on intervals > (e.g., every 3 seconds), observe the state of the assignment of resources to > queues from the capacity scheduler, > performs off-line computation to determine if preemption is needed, and how > best to "edit" the current schedule to > improve capacity, and generates events that produce four possible actions: > # Container de-reservations > # Resource-based preemptions > # Container-based preemptions > # Container killing > The actions listed above are progressively more costly, and it is up to the > policy to use them as desired to achieve the rebalancing goals. > Note that due to the "lag" in the effect of these actions the policy should > operate at the macroscopic level (e.g., preempt tens of containers > from a queue) and not trying to tightly and consistently micromanage > container allocations. > - Preemption policy (ProportionalCapacityPreemptionPolicy): > - > Preemption policies are by design pluggable, in the following we present an > initial policy (ProportionalCapacityPreemptionPolicy) we have been > experimenting with. The ProportionalCapacityPreemptionPolicy behaves as > follows: > # it gathers from the scheduler the state of the queues, in particular, their > current capacity, guaranteed capacity and pending requests (*) > # if there are pending requests from queues that are under capacity it > computes a new ideal balanced state (**) > # it computes the set of preemptions needed to repair the current schedule > and achieve capacity balance (accounting for natural completion rates, and > respecting bounds on the amount of preemption we allow for each round) > # it selects which applications to preempt from each over-capacity queue (the > last one in the FIFO order) > # it remove reservations from the most recently assigned app until the amount > of resource to reclaim is obtained, or until no more reservations exits > # (if not enough) it issues preemptions for containers from the same > applications (reverse chronological order, last assigned container first) > again until necessary or until no containers except the AM container are left, > # (if not enough) it moves onto unreserve and preempt from the next > application. > # containers that have been asked to preempt are tracked across executions. > If a containers is among the one to be preempted for more than a certain > time, the container is moved in a the list of containers to be forcibly > killed. > Notes: > (*) at the moment, in order to avoid double-counting of the requests, we only > look at the "ANY" part of pending resource requests, which means we might not > preempt on behalf of AMs that ask only for specific locations but not any. > (**) The ideal balance state is one in which each queue has at least its > guaranteed capacity, and the spare capacity is distributed among queues (that > wants some) as a weighted fair share. Where the weighting is based on the > guaranteed capacity of a
[jira] [Commented] (YARN-852) TestAggregatedLogFormat.testContainerLogsFileAccess fails on Windows
[ https://issues.apache.org/jira/browse/YARN-852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688315#comment-13688315 ] Hudson commented on YARN-852: - Integrated in Hadoop-trunk-Commit #3984 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3984/]) YARN-852. TestAggregatedLogFormat.testContainerLogsFileAccess fails on Windows. Contributed by Chuan Liu. (Revision 1494733) Result = SUCCESS cnauroth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494733 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java > TestAggregatedLogFormat.testContainerLogsFileAccess fails on Windows > > > Key: YARN-852 > URL: https://issues.apache.org/jira/browse/YARN-852 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Chuan Liu >Assignee: Chuan Liu >Priority: Minor > Attachments: YARN-852-trunk.patch, YARN-852-trunk.patch > > > The YARN unit test case fails on Windows when comparing expected message with > log message in the file. The expected message constructed in the test case > has two problems: 1) it uses Path.separator to concatenate path string. > Path.separator is always a forward slash, which does not match the backslash > used in the log message. 2) On Windows, the default file owner is > Administrators group if the file is created by an Administrators user. The > test expect the user to be the current user. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-694) Start using NMTokens to authenticate all communication with NM
[ https://issues.apache.org/jira/browse/YARN-694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688334#comment-13688334 ] Vinod Kumar Vavilapalli commented on YARN-694: -- I'm opening a new ticket for the test. Will also take care of HADOOP-9583. > Start using NMTokens to authenticate all communication with NM > -- > > Key: YARN-694 > URL: https://issues.apache.org/jira/browse/YARN-694 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi > Fix For: 2.1.0-beta > > Attachments: YARN-694-20130613.patch, YARN-694-20130617.1.patch, > YARN-694-20130617.2.patch, YARN-694-20130617.patch, > YARN-694-20130618.1.patch, YARN-694-20130618.2.patch, > YARN-694-20130618.3.patch, YARN-694-20130618.4.patch, > YARN-694-20130618.5.patch, YARN-694-20130618.patch.branch-2, > YARN-694-20130618.patch.yarn-694-branch-2.1-beta > > > AM uses the NMToken to authenticate all the AM-NM communication. > NM will validate NMToken in below manner > * If NMToken is using current or previous master key then the NMToken is > valid. In this case it will update its cache with this key corresponding to > appId. > * If NMToken is using the master key which is present in NM's cache > corresponding to AM's appId then it will be validated based on this. > * If NMToken is invalid then NM will reject AM calls. > Modification for ContainerToken > * At present RPC validates AM-NM communication based on ContainerToken. It > will be replaced with NMToken. Also now onwards AM will use NMToken per NM > (replacing earlier behavior of ContainerToken per container per NM). > * startContainer in case of Secured environment is using ContainerToken from > UGI YARN-617; however after this it will use it from the payload (Container). > * ContainerToken will exist and it will only be used to validate the AM's > container start request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-727) ClientRMProtocol.getAllApplications should accept ApplicationType as a parameter
[
https://issues.apache.org/jira/browse/YARN-727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688338#comment-13688338
]
Hadoop QA commented on YARN-727:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588638/YARN-727.14.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1350//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1350//console
This message is automatically generated.
> ClientRMProtocol.getAllApplications should accept ApplicationType as a
> parameter
>
>
> Key: YARN-727
> URL: https://issues.apache.org/jira/browse/YARN-727
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.1.0-beta
>Reporter: Siddharth Seth
>Assignee: Xuan Gong
> Attachments: YARN-727.10.patch, YARN-727.11.patch, YARN-727.12.patch,
> YARN-727.13.patch, YARN-727.14.patch, YARN-727.1.patch, YARN-727.2.patch,
> YARN-727.3.patch, YARN-727.4.patch, YARN-727.5.patch, YARN-727.6.patch,
> YARN-727.7.patch, YARN-727.8.patch, YARN-727.9.patch
>
>
> Now that an ApplicationType is registered on ApplicationSubmission,
> getAllApplications should be able to use this string to query for a specific
> application type.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-694) Start using NMTokens to authenticate all communication with NM
[ https://issues.apache.org/jira/browse/YARN-694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688375#comment-13688375 ] Vinod Kumar Vavilapalli commented on YARN-694: -- bq. I'm opening a new ticket for the test. MAPREDUCE-5334. > Start using NMTokens to authenticate all communication with NM > -- > > Key: YARN-694 > URL: https://issues.apache.org/jira/browse/YARN-694 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi > Fix For: 2.1.0-beta > > Attachments: YARN-694-20130613.patch, YARN-694-20130617.1.patch, > YARN-694-20130617.2.patch, YARN-694-20130617.patch, > YARN-694-20130618.1.patch, YARN-694-20130618.2.patch, > YARN-694-20130618.3.patch, YARN-694-20130618.4.patch, > YARN-694-20130618.5.patch, YARN-694-20130618.patch.branch-2, > YARN-694-20130618.patch.yarn-694-branch-2.1-beta > > > AM uses the NMToken to authenticate all the AM-NM communication. > NM will validate NMToken in below manner > * If NMToken is using current or previous master key then the NMToken is > valid. In this case it will update its cache with this key corresponding to > appId. > * If NMToken is using the master key which is present in NM's cache > corresponding to AM's appId then it will be validated based on this. > * If NMToken is invalid then NM will reject AM calls. > Modification for ContainerToken > * At present RPC validates AM-NM communication based on ContainerToken. It > will be replaced with NMToken. Also now onwards AM will use NMToken per NM > (replacing earlier behavior of ContainerToken per container per NM). > * startContainer in case of Secured environment is using ContainerToken from > UGI YARN-617; however after this it will use it from the payload (Container). > * ContainerToken will exist and it will only be used to validate the AM's > container start request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-569) CapacityScheduler: support for preemption (using a capacity monitor)
[
https://issues.apache.org/jira/browse/YARN-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated YARN-569:
---
Attachment: YARN-569.9.patch
bq. One other thing to check would be if the preemption policy will use
refreshed values when the capacity scheduler config is refreshed on the fly.
Looks like cloneQueues() will take the absolute used and guaranteed numbers on
every clone. So we should be good wrt that. Would be good to check other values
the policy looks at.
*nod* Right now, the policy rebuilds its view of the scheduler at every pass,
but it doesn't refresh its own config parameters.
bq. Noticed formatting issues with spaces in the patch. eg. cloneQueues()
Did another pass over the patch, fixed up spacing, formatting, and removed
obvious whitespace changes. Sorry, did a few of these already, but missed a few.
Also moved the check in the {{ApplicationMasterService}} as part of this patch.
> CapacityScheduler: support for preemption (using a capacity monitor)
>
>
> Key: YARN-569
> URL: https://issues.apache.org/jira/browse/YARN-569
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Reporter: Carlo Curino
>Assignee: Carlo Curino
> Attachments: 3queues.pdf, CapScheduler_with_preemption.pdf,
> preemption.2.patch, YARN-569.1.patch, YARN-569.2.patch, YARN-569.3.patch,
> YARN-569.4.patch, YARN-569.5.patch, YARN-569.6.patch, YARN-569.8.patch,
> YARN-569.9.patch, YARN-569.patch, YARN-569.patch
>
>
> There is a tension between the fast-pace reactive role of the
> CapacityScheduler, which needs to respond quickly to
> applications resource requests, and node updates, and the more introspective,
> time-based considerations
> needed to observe and correct for capacity balance. To this purpose we opted
> instead of hacking the delicate
> mechanisms of the CapacityScheduler directly to add support for preemption by
> means of a "Capacity Monitor",
> which can be run optionally as a separate service (much like the
> NMLivelinessMonitor).
> The capacity monitor (similarly to equivalent functionalities in the fairness
> scheduler) operates running on intervals
> (e.g., every 3 seconds), observe the state of the assignment of resources to
> queues from the capacity scheduler,
> performs off-line computation to determine if preemption is needed, and how
> best to "edit" the current schedule to
> improve capacity, and generates events that produce four possible actions:
> # Container de-reservations
> # Resource-based preemptions
> # Container-based preemptions
> # Container killing
> The actions listed above are progressively more costly, and it is up to the
> policy to use them as desired to achieve the rebalancing goals.
> Note that due to the "lag" in the effect of these actions the policy should
> operate at the macroscopic level (e.g., preempt tens of containers
> from a queue) and not trying to tightly and consistently micromanage
> container allocations.
> - Preemption policy (ProportionalCapacityPreemptionPolicy):
> -
> Preemption policies are by design pluggable, in the following we present an
> initial policy (ProportionalCapacityPreemptionPolicy) we have been
> experimenting with. The ProportionalCapacityPreemptionPolicy behaves as
> follows:
> # it gathers from the scheduler the state of the queues, in particular, their
> current capacity, guaranteed capacity and pending requests (*)
> # if there are pending requests from queues that are under capacity it
> computes a new ideal balanced state (**)
> # it computes the set of preemptions needed to repair the current schedule
> and achieve capacity balance (accounting for natural completion rates, and
> respecting bounds on the amount of preemption we allow for each round)
> # it selects which applications to preempt from each over-capacity queue (the
> last one in the FIFO order)
> # it remove reservations from the most recently assigned app until the amount
> of resource to reclaim is obtained, or until no more reservations exits
> # (if not enough) it issues preemptions for containers from the same
> applications (reverse chronological order, last assigned container first)
> again until necessary or until no containers except the AM container are left,
> # (if not enough) it moves onto unreserve and preempt from the next
> application.
> # containers that have been asked to preempt are tracked across executions.
> If a containers is among the one to be preempted for more than a certain
> time, the container is moved in a the list of containers to be forcibly
> killed.
> Notes:
> (*) at the moment, in order to avoid double-counting of the requests, we only
> look at the "ANY" part of pending resou
[jira] [Created] (YARN-854) App submission fails on secure deploy
Ramya Sunil created YARN-854:
Summary: App submission fails on secure deploy
Key: YARN-854
URL: https://issues.apache.org/jira/browse/YARN-854
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.1.0-beta
Reporter: Ramya Sunil
Fix For: 2.1.0-beta
App submission on secure cluster fails with the following exception:
{noformat}
INFO mapreduce.Job: Job jobID failed with state FAILED due to: Application
applicationID failed 2 times due to AM Container for appattemptID exited with
exitCode: -1000 due to: App initialization failed (255) with output: main :
command provided 0
main : user is qa_user
javax.security.sasl.SaslException: DIGEST-MD5: digest response format
violation. Mismatched response. [Caused by
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
DIGEST-MD5: digest response format violation. Mismatched response.]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
at
org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:65)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:348)
Caused by:
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
DIGEST-MD5: digest response format violation. Mismatched response.
at org.apache.hadoop.ipc.Client.call(Client.java:1298)
at org.apache.hadoop.ipc.Client.call(Client.java:1250)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:204)
at $Proxy7.heartbeat(Unknown Source)
at
org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
... 3 more
.Failing this attempt.. Failing the application.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-854) App submission fails on secure deploy
[
https://issues.apache.org/jira/browse/YARN-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated YARN-854:
-
Priority: Blocker (was: Major)
> App submission fails on secure deploy
> -
>
> Key: YARN-854
> URL: https://issues.apache.org/jira/browse/YARN-854
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.1.0-beta
>Reporter: Ramya Sunil
>Assignee: Omkar Vinit Joshi
>Priority: Blocker
> Fix For: 2.1.0-beta
>
>
> App submission on secure cluster fails with the following exception:
> {noformat}
> INFO mapreduce.Job: Job jobID failed with state FAILED due to: Application
> applicationID failed 2 times due to AM Container for appattemptID exited with
> exitCode: -1000 due to: App initialization failed (255) with output: main :
> command provided 0
> main : user is qa_user
> javax.security.sasl.SaslException: DIGEST-MD5: digest response format
> violation. Mismatched response. [Caused by
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> DIGEST-MD5: digest response format violation. Mismatched response.]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:65)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:348)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> DIGEST-MD5: digest response format violation. Mismatched response.
> at org.apache.hadoop.ipc.Client.call(Client.java:1298)
> at org.apache.hadoop.ipc.Client.call(Client.java:1250)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:204)
> at $Proxy7.heartbeat(Unknown Source)
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
> ... 3 more
> .Failing this attempt.. Failing the application.
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-854) App submission fails on secure deploy
[
https://issues.apache.org/jira/browse/YARN-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli reassigned YARN-854:
Assignee: Omkar Vinit Joshi
Omkar, can you please check? Tx!
> App submission fails on secure deploy
> -
>
> Key: YARN-854
> URL: https://issues.apache.org/jira/browse/YARN-854
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.1.0-beta
>Reporter: Ramya Sunil
>Assignee: Omkar Vinit Joshi
> Fix For: 2.1.0-beta
>
>
> App submission on secure cluster fails with the following exception:
> {noformat}
> INFO mapreduce.Job: Job jobID failed with state FAILED due to: Application
> applicationID failed 2 times due to AM Container for appattemptID exited with
> exitCode: -1000 due to: App initialization failed (255) with output: main :
> command provided 0
> main : user is qa_user
> javax.security.sasl.SaslException: DIGEST-MD5: digest response format
> violation. Mismatched response. [Caused by
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> DIGEST-MD5: digest response format violation. Mismatched response.]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:65)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:348)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> DIGEST-MD5: digest response format violation. Mismatched response.
> at org.apache.hadoop.ipc.Client.call(Client.java:1298)
> at org.apache.hadoop.ipc.Client.call(Client.java:1250)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:204)
> at $Proxy7.heartbeat(Unknown Source)
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
> ... 3 more
> .Failing this attempt.. Failing the application.
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-727) ClientRMProtocol.getAllApplications should accept ApplicationType as a parameter
[ https://issues.apache.org/jira/browse/YARN-727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688446#comment-13688446 ] Hitesh Shah commented on YARN-727: -- Comments: ResourceMgrDelegate.java: - In getAllJobs(), why is this not matching against only the MAPREDUCE application type? - Why does ResourceMgrDelegate extent YarnClient? I dont believe it should. Could you file a jira for this. yarn_service_protos.proto: - please look at the coding conventions followed in this file - its lowercase with _ separators ApplicationCLI.java: - code should use Option instance for "appTypes" and not rely on parsing the string. Let the GnuParser do it. - Look to making use of Option.setValueSeparator, and Option.getValues - Current code seems like using "--list --foobar=type1,type2" will work. Does the webservice also handle multiple app types or does that need to be changed too? > ClientRMProtocol.getAllApplications should accept ApplicationType as a > parameter > > > Key: YARN-727 > URL: https://issues.apache.org/jira/browse/YARN-727 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.1.0-beta >Reporter: Siddharth Seth >Assignee: Xuan Gong > Attachments: YARN-727.10.patch, YARN-727.11.patch, YARN-727.12.patch, > YARN-727.13.patch, YARN-727.14.patch, YARN-727.1.patch, YARN-727.2.patch, > YARN-727.3.patch, YARN-727.4.patch, YARN-727.5.patch, YARN-727.6.patch, > YARN-727.7.patch, YARN-727.8.patch, YARN-727.9.patch > > > Now that an ApplicationType is registered on ApplicationSubmission, > getAllApplications should be able to use this string to query for a specific > application type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-569) CapacityScheduler: support for preemption (using a capacity monitor)
[
https://issues.apache.org/jira/browse/YARN-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688467#comment-13688467
]
Hadoop QA commented on YARN-569:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588685/YARN-569.9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1352//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1352//console
This message is automatically generated.
> CapacityScheduler: support for preemption (using a capacity monitor)
>
>
> Key: YARN-569
> URL: https://issues.apache.org/jira/browse/YARN-569
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Reporter: Carlo Curino
>Assignee: Carlo Curino
> Attachments: 3queues.pdf, CapScheduler_with_preemption.pdf,
> preemption.2.patch, YARN-569.1.patch, YARN-569.2.patch, YARN-569.3.patch,
> YARN-569.4.patch, YARN-569.5.patch, YARN-569.6.patch, YARN-569.8.patch,
> YARN-569.9.patch, YARN-569.patch, YARN-569.patch
>
>
> There is a tension between the fast-pace reactive role of the
> CapacityScheduler, which needs to respond quickly to
> applications resource requests, and node updates, and the more introspective,
> time-based considerations
> needed to observe and correct for capacity balance. To this purpose we opted
> instead of hacking the delicate
> mechanisms of the CapacityScheduler directly to add support for preemption by
> means of a "Capacity Monitor",
> which can be run optionally as a separate service (much like the
> NMLivelinessMonitor).
> The capacity monitor (similarly to equivalent functionalities in the fairness
> scheduler) operates running on intervals
> (e.g., every 3 seconds), observe the state of the assignment of resources to
> queues from the capacity scheduler,
> performs off-line computation to determine if preemption is needed, and how
> best to "edit" the current schedule to
> improve capacity, and generates events that produce four possible actions:
> # Container de-reservations
> # Resource-based preemptions
> # Container-based preemptions
> # Container killing
> The actions listed above are progressively more costly, and it is up to the
> policy to use them as desired to achieve the rebalancing goals.
> Note that due to the "lag" in the effect of these actions the policy should
> operate at the macroscopic level (e.g., preempt tens of containers
> from a queue) and not trying to tightly and consistently micromanage
> container allocations.
> - Preemption policy (ProportionalCapacityPreemptionPolicy):
> -
> Preemption policies are by design pluggable, in the following we present an
> initial policy (ProportionalCapacityPreemptionPolicy) we have been
> experimenting with. The ProportionalCapacityPreemptionPolicy behaves as
> follows:
> # it gathers from the scheduler the state of the queues, in particular, their
> current capacity, guaranteed capacity and pending requests (*)
> # if there are pending requests from queues that are under capacity it
> computes a new ideal balanced state (**)
> # it computes the set of preemptions needed to repair the current schedule
> and achieve capacity balance (accounting for natural completion rates, and
> respecting bounds on the amount of preemption we allow for each round)
> # it selects which applications to preempt from each over-capacity queue (the
> last one in the FIFO order)
> # it remove reservations from the most recently assigned app until the amount
> of resource to reclaim is obtained, or until no more rese
[jira] [Created] (YARN-855) YarnClient.init should ensure that yarn parameters are present
Siddharth Seth created YARN-855: --- Summary: YarnClient.init should ensure that yarn parameters are present Key: YARN-855 URL: https://issues.apache.org/jira/browse/YARN-855 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.0.5-alpha Reporter: Siddharth Seth It currently accepts a Configuration object in init and doesn't check whether it contains yarn parameters or is a YarnConfiguration. Should either accept YarnConfiguration, check existence of parameters or create a YarnConfiguration based on the configuration passed to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-791) Ensure that RM RPC APIs that return nodes are consistent with /nodes REST API
[ https://issues.apache.org/jira/browse/YARN-791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688500#comment-13688500 ] Sandy Ryza commented on YARN-791: - [~vinodkv], does Alejandro's suggestion seem acceptable to you? If so, I will make those changes. > Ensure that RM RPC APIs that return nodes are consistent with /nodes REST API > - > > Key: YARN-791 > URL: https://issues.apache.org/jira/browse/YARN-791 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, resourcemanager >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-791-1.patch, YARN-791-2.patch, YARN-791-3.patch, > YARN-791-4.patch, YARN-791.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-854) App submission fails on secure deploy
[
https://issues.apache.org/jira/browse/YARN-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Omkar Vinit Joshi updated YARN-854:
---
Attachment: YARN-854.20130619.patch
> App submission fails on secure deploy
> -
>
> Key: YARN-854
> URL: https://issues.apache.org/jira/browse/YARN-854
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.1.0-beta
>Reporter: Ramya Sunil
>Assignee: Omkar Vinit Joshi
>Priority: Blocker
> Fix For: 2.1.0-beta
>
> Attachments: YARN-854.20130619.patch
>
>
> App submission on secure cluster fails with the following exception:
> {noformat}
> INFO mapreduce.Job: Job jobID failed with state FAILED due to: Application
> applicationID failed 2 times due to AM Container for appattemptID exited with
> exitCode: -1000 due to: App initialization failed (255) with output: main :
> command provided 0
> main : user is qa_user
> javax.security.sasl.SaslException: DIGEST-MD5: digest response format
> violation. Mismatched response. [Caused by
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> DIGEST-MD5: digest response format violation. Mismatched response.]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:65)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:348)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> DIGEST-MD5: digest response format violation. Mismatched response.
> at org.apache.hadoop.ipc.Client.call(Client.java:1298)
> at org.apache.hadoop.ipc.Client.call(Client.java:1250)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:204)
> at $Proxy7.heartbeat(Unknown Source)
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
> ... 3 more
> .Failing this attempt.. Failing the application.
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-856) Clicking on 'Tracking UI History' link leads to server error 500
Kam Kasravi created YARN-856: Summary: Clicking on 'Tracking UI History' link leads to server error 500 Key: YARN-856 URL: https://issues.apache.org/jira/browse/YARN-856 Project: Hadoop YARN Issue Type: Bug Components: site Affects Versions: 2.0.3-alpha Environment: browser - chrome 27.x Reporter: Kam Kasravi 1. Browse to http://localhost:8088/cluster/apps 2. Select History link on the far right 3. See error page below HTTP ERROR 500 Problem accessing /proxy/application_1371597044883_0001/. Reason: java.net.URISyntaxException: Expected authority at index 7: http:// Caused by: java.io.IOException: java.net.URISyntaxException: Expected authority at index 7: http:// at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:347) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1069) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.net.URISyntaxException: Expected authority at index 7: http:// at java.net.URI$Parser.fail(URI.java:2810) at java.net.URI$Parser.failExpecting(URI.java:2816) at java.net.URI$Parser.parseHierarchical(URI.java:3064) at java.net.URI$Parser.parse(URI.java:3015) at java.net.URI.(URI.java:577) at org.apache.hadoop.yarn.server.webproxy.ProxyUriUtils.getUriFromAMUrl(ProxyUriUtils.java:146) at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:292) ... 36 more -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-857) Errors when localizing end up with the localization failure not being seen by the NM
Hitesh Shah created YARN-857: Summary: Errors when localizing end up with the localization failure not being seen by the NM Key: YARN-857 URL: https://issues.apache.org/jira/browse/YARN-857 Project: Hadoop YARN Issue Type: Bug Reporter: Hitesh Shah at org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:106) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:978) Traced this down to both DefaultExecutor and LinuxExecutor's startLocalizer functions both of which do not look at the exit code for the localizer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-854) App submission fails on secure deploy
[
https://issues.apache.org/jira/browse/YARN-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688586#comment-13688586
]
Hadoop QA commented on YARN-854:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588705/YARN-854.20130619.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerReboot
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch
org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerShutdown
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestContainerManager
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1353//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1353//console
This message is automatically generated.
> App submission fails on secure deploy
> -
>
> Key: YARN-854
> URL: https://issues.apache.org/jira/browse/YARN-854
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.1.0-beta
>Reporter: Ramya Sunil
>Assignee: Omkar Vinit Joshi
>Priority: Blocker
> Fix For: 2.1.0-beta
>
> Attachments: YARN-854.20130619.patch
>
>
> App submission on secure cluster fails with the following exception:
> {noformat}
> INFO mapreduce.Job: Job jobID failed with state FAILED due to: Application
> applicationID failed 2 times due to AM Container for appattemptID exited with
> exitCode: -1000 due to: App initialization failed (255) with output: main :
> command provided 0
> main : user is qa_user
> javax.security.sasl.SaslException: DIGEST-MD5: digest response format
> violation. Mismatched response. [Caused by
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> DIGEST-MD5: digest response format violation. Mismatched response.]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:65)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:348)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> DIGEST-MD5: digest response format violation. Mismatched response.
> at org.apache.hadoop.ipc.Client.call(Client.java:1298)
> at org.apache.hadoop.ipc.Client.call(Client.java:1250)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.ja
[jira] [Commented] (YARN-854) App submission fails on secure deploy
[
https://issues.apache.org/jira/browse/YARN-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688588#comment-13688588
]
Omkar Vinit Joshi commented on YARN-854:
Fixing one more thing.. making sure LocalizerTokenSecretManager is used
irrespective of security.
> App submission fails on secure deploy
> -
>
> Key: YARN-854
> URL: https://issues.apache.org/jira/browse/YARN-854
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.1.0-beta
>Reporter: Ramya Sunil
>Assignee: Omkar Vinit Joshi
>Priority: Blocker
> Fix For: 2.1.0-beta
>
> Attachments: YARN-854.20130619.patch
>
>
> App submission on secure cluster fails with the following exception:
> {noformat}
> INFO mapreduce.Job: Job jobID failed with state FAILED due to: Application
> applicationID failed 2 times due to AM Container for appattemptID exited with
> exitCode: -1000 due to: App initialization failed (255) with output: main :
> command provided 0
> main : user is qa_user
> javax.security.sasl.SaslException: DIGEST-MD5: digest response format
> violation. Mismatched response. [Caused by
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> DIGEST-MD5: digest response format violation. Mismatched response.]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:65)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:348)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> DIGEST-MD5: digest response format violation. Mismatched response.
> at org.apache.hadoop.ipc.Client.call(Client.java:1298)
> at org.apache.hadoop.ipc.Client.call(Client.java:1250)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:204)
> at $Proxy7.heartbeat(Unknown Source)
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
> ... 3 more
> .Failing this attempt.. Failing the application.
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-857) Errors when localizing end up with the localization failure not being seen by the NM
[ https://issues.apache.org/jira/browse/YARN-857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated YARN-857: - Description: at org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:106) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:978) Traced this down to DefaultExecutor which does not look at the exit code for the localizer. was: at org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:106) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:978) Traced this down to both DefaultExecutor and LinuxExecutor's startLocalizer functions both of which do not look at the exit code for the localizer. > Errors when localizing end up with the localization failure not being seen by > the NM > > > Key: YARN-857 > URL: https://issues.apache.org/jira/browse/YARN-857 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah > > at > org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:106) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:978) > Traced this down to DefaultExecutor which does not look at the exit code for > the localizer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-857) Errors when localizing end up with the localization failure not being seen by the NM
[ https://issues.apache.org/jira/browse/YARN-857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah reassigned YARN-857: Assignee: Hitesh Shah > Errors when localizing end up with the localization failure not being seen by > the NM > > > Key: YARN-857 > URL: https://issues.apache.org/jira/browse/YARN-857 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah > > at > org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:106) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:978) > Traced this down to both DefaultExecutor and LinuxExecutor's startLocalizer > functions both of which do not look at the exit code for the localizer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-858) Move Resources and the caculators to a new package util.resource in yarn-common
Jian He created YARN-858: Summary: Move Resources and the caculators to a new package util.resource in yarn-common Key: YARN-858 URL: https://issues.apache.org/jira/browse/YARN-858 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-858) Move Resources and the calculators to a new package util.resource in yarn-common
[ https://issues.apache.org/jira/browse/YARN-858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-858: Summary: Move Resources and the calculators to a new package util.resource in yarn-common (was: Move Resources and the caculators to a new package util.resource in yarn-common ) > Move Resources and the calculators to a new package util.resource in > yarn-common > - > > Key: YARN-858 > URL: https://issues.apache.org/jira/browse/YARN-858 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > > For userland use -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-858) Move Resources and the caculators to a new package util.resource in yarn-common
[ https://issues.apache.org/jira/browse/YARN-858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-858: - Description: For userland use > Move Resources and the caculators to a new package util.resource in > yarn-common > > > Key: YARN-858 > URL: https://issues.apache.org/jira/browse/YARN-858 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > > For userland use -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-858) Move Resources and the calculators to a new package util.resource in yarn-common
[ https://issues.apache.org/jira/browse/YARN-858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-858: - Attachment: YARN-858.patch Moved DefaultResourceCalculator, DominantResourceCalculator, ResourceCalculator, Resources to yarn-common > Move Resources and the calculators to a new package util.resource in > yarn-common > - > > Key: YARN-858 > URL: https://issues.apache.org/jira/browse/YARN-858 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-858.patch > > > For userland use -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-857) Errors when localizing end up with the localization failure not being seen by the NM
[ https://issues.apache.org/jira/browse/YARN-857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated YARN-857: - Attachment: YARN-857.1.patch Unfortunately could not come up with a unit test which tests that exact bit where a -1 is returned by causing a throw in localizeFiles. Suggestions welcome. > Errors when localizing end up with the localization failure not being seen by > the NM > > > Key: YARN-857 > URL: https://issues.apache.org/jira/browse/YARN-857 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah > Attachments: YARN-857.1.patch > > > at > org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:106) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:978) > Traced this down to DefaultExecutor which does not look at the exit code for > the localizer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-858) Move Resources and the calculators to a new package util.resource in yarn-common
[ https://issues.apache.org/jira/browse/YARN-858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688664#comment-13688664 ] Sandy Ryza commented on YARN-858: - The JIRA description says "for userland use", but the classes are marked @Private. Is this intentional? As discussed in YARN-827, these APIs are not yet mature and should not yet be made public. > Move Resources and the calculators to a new package util.resource in > yarn-common > - > > Key: YARN-858 > URL: https://issues.apache.org/jira/browse/YARN-858 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-858.patch > > > For userland use -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-858) Move Resources and the calculators to a new package util.resource in yarn-common
[ https://issues.apache.org/jira/browse/YARN-858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688669#comment-13688669 ] Jian He commented on YARN-858: -- Looks like this is duplicative as YARN-827 > Move Resources and the calculators to a new package util.resource in > yarn-common > - > > Key: YARN-858 > URL: https://issues.apache.org/jira/browse/YARN-858 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-858.patch > > > For userland use -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-858) Move Resources and the calculators to a new package util.resource in yarn-common
[ https://issues.apache.org/jira/browse/YARN-858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688674#comment-13688674 ] Jian He commented on YARN-858: -- close this > Move Resources and the calculators to a new package util.resource in > yarn-common > - > > Key: YARN-858 > URL: https://issues.apache.org/jira/browse/YARN-858 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-858.patch > > > For userland use -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-858) Move Resources and the calculators to a new package util.resource in yarn-common
[ https://issues.apache.org/jira/browse/YARN-858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He resolved YARN-858. -- Resolution: Duplicate > Move Resources and the calculators to a new package util.resource in > yarn-common > - > > Key: YARN-858 > URL: https://issues.apache.org/jira/browse/YARN-858 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-858.patch > > > For userland use -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-857) Errors when localizing end up with the localization failure not being seen by the NM
[
https://issues.apache.org/jira/browse/YARN-857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688678#comment-13688678
]
Hadoop QA commented on YARN-857:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588726/YARN-857.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch
org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerReboot
org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerShutdown
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestContainerManager
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1355//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1355//console
This message is automatically generated.
> Errors when localizing end up with the localization failure not being seen by
> the NM
>
>
> Key: YARN-857
> URL: https://issues.apache.org/jira/browse/YARN-857
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Hitesh Shah
>Assignee: Hitesh Shah
> Attachments: YARN-857.1.patch
>
>
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:106)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:978)
> Traced this down to DefaultExecutor which does not look at the exit code for
> the localizer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-858) Move Resources and the calculators to a new package util.resource in yarn-common
[
https://issues.apache.org/jira/browse/YARN-858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688682#comment-13688682
]
Hadoop QA commented on YARN-858:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588723/YARN-858.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 22 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing
org.apache.hadoop.yarn.server.resourcemanager.TestResourceManager
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
org.apache.hadoop.yarn.server.resourcemanager.security.TestAMRMTokens
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher
org.apache.hadoop.yarn.server.resourcemanager.TestRM
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId
org.apache.hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestNodesPage
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebApp
org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerUtils
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1354//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/1354//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1354//console
This message is automatically generated.
> Move Resources and the calculators to a new package util.resource in
> yarn-common
> -
>
> Key: YARN-858
> URL: https://issues.apache.org/jira/browse/YARN-858
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-858.patch
>
>
> For userland use
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-827) Need to make Resource arithmetic methods public
[ https://issues.apache.org/jira/browse/YARN-827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-827: - Attachment: YARN-827.patch upload a patch to attract attentions. Moved DefaultResourceCalculator, DominantResourceCalculator, ResourceCalculator, Resources to yarn-common > Need to make Resource arithmetic methods public > --- > > Key: YARN-827 > URL: https://issues.apache.org/jira/browse/YARN-827 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Zhijie Shen >Priority: Critical > Attachments: YARN-827.patch > > > org.apache.hadoop.yarn.server.resourcemanager.resource has stuff like > Resources and Calculators that help compare/add resources etc. Without these > users will be forced to replicate the logic, potentially incorrectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-727) ClientRMProtocol.getAllApplications should accept ApplicationType as a parameter
[ https://issues.apache.org/jira/browse/YARN-727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688710#comment-13688710 ] Xuan Gong commented on YARN-727: bq.Does the webservice also handle multiple app types or does that need to be changed too? [~hitesh] I do not think we have filter with parameter app_types in current webservice. There are filters we are using to get appInfo right now: 1. filter with parameter app_id 2. filter with parameters: state, finalStatus, user, queue, limit, startedTimeBegin, startedTimeEnd, finishedTimeBegin, finishedTimeEnd We can add a new parameter named app_types in filter 2. Do you want me to fix it here or open another ticket ? > ClientRMProtocol.getAllApplications should accept ApplicationType as a > parameter > > > Key: YARN-727 > URL: https://issues.apache.org/jira/browse/YARN-727 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.1.0-beta >Reporter: Siddharth Seth >Assignee: Xuan Gong > Attachments: YARN-727.10.patch, YARN-727.11.patch, YARN-727.12.patch, > YARN-727.13.patch, YARN-727.14.patch, YARN-727.1.patch, YARN-727.2.patch, > YARN-727.3.patch, YARN-727.4.patch, YARN-727.5.patch, YARN-727.6.patch, > YARN-727.7.patch, YARN-727.8.patch, YARN-727.9.patch > > > Now that an ApplicationType is registered on ApplicationSubmission, > getAllApplications should be able to use this string to query for a specific > application type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-857) Errors when localizing end up with the localization failure not being seen by the NM
[ https://issues.apache.org/jira/browse/YARN-857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688717#comment-13688717 ] Vinod Kumar Vavilapalli commented on YARN-857: -- I just ran these tests locally and are already failing. The failing tests should pass after YARN-854 changes. > Errors when localizing end up with the localization failure not being seen by > the NM > > > Key: YARN-857 > URL: https://issues.apache.org/jira/browse/YARN-857 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah > Attachments: YARN-857.1.patch > > > at > org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:106) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:978) > Traced this down to DefaultExecutor which does not look at the exit code for > the localizer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-857) Errors when localizing end up with the localization failure not being seen by the NM
[ https://issues.apache.org/jira/browse/YARN-857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688723#comment-13688723 ] Vinod Kumar Vavilapalli commented on YARN-857: -- I applied the patch locally too and can validate that tests like TestContainerManager now failing instead of timing out. > Errors when localizing end up with the localization failure not being seen by > the NM > > > Key: YARN-857 > URL: https://issues.apache.org/jira/browse/YARN-857 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah > Attachments: YARN-857.1.patch > > > at > org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:106) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:978) > Traced this down to DefaultExecutor which does not look at the exit code for > the localizer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-857) Errors when localizing end up with the localization failure not being seen by the NM
[ https://issues.apache.org/jira/browse/YARN-857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-857: - Attachment: YARN-857.2.patch Here's a patch with a test-case which fails(by timeout) without the code changes and passes with.. > Errors when localizing end up with the localization failure not being seen by > the NM > > > Key: YARN-857 > URL: https://issues.apache.org/jira/browse/YARN-857 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah > Attachments: YARN-857.1.patch, YARN-857.2.patch > > > at > org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:106) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:978) > Traced this down to DefaultExecutor which does not look at the exit code for > the localizer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-854) App submission fails on secure deploy
[
https://issues.apache.org/jira/browse/YARN-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Omkar Vinit Joshi updated YARN-854:
---
Attachment: YARN-854.20130619.1.patch
> App submission fails on secure deploy
> -
>
> Key: YARN-854
> URL: https://issues.apache.org/jira/browse/YARN-854
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.1.0-beta
>Reporter: Ramya Sunil
>Assignee: Omkar Vinit Joshi
>Priority: Blocker
> Fix For: 2.1.0-beta
>
> Attachments: YARN-854.20130619.1.patch, YARN-854.20130619.patch
>
>
> App submission on secure cluster fails with the following exception:
> {noformat}
> INFO mapreduce.Job: Job jobID failed with state FAILED due to: Application
> applicationID failed 2 times due to AM Container for appattemptID exited with
> exitCode: -1000 due to: App initialization failed (255) with output: main :
> command provided 0
> main : user is qa_user
> javax.security.sasl.SaslException: DIGEST-MD5: digest response format
> violation. Mismatched response. [Caused by
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> DIGEST-MD5: digest response format violation. Mismatched response.]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:65)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:348)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> DIGEST-MD5: digest response format violation. Mismatched response.
> at org.apache.hadoop.ipc.Client.call(Client.java:1298)
> at org.apache.hadoop.ipc.Client.call(Client.java:1250)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:204)
> at $Proxy7.heartbeat(Unknown Source)
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
> ... 3 more
> .Failing this attempt.. Failing the application.
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-857) Errors when localizing end up with the localization failure not being seen by the NM
[
https://issues.apache.org/jira/browse/YARN-857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688785#comment-13688785
]
Hadoop QA commented on YARN-857:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588734/YARN-857.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerReboot
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch
org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerShutdown
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestContainerManager
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1356//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1356//console
This message is automatically generated.
> Errors when localizing end up with the localization failure not being seen by
> the NM
>
>
> Key: YARN-857
> URL: https://issues.apache.org/jira/browse/YARN-857
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Hitesh Shah
>Assignee: Hitesh Shah
> Attachments: YARN-857.1.patch, YARN-857.2.patch
>
>
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:106)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:978)
> Traced this down to DefaultExecutor which does not look at the exit code for
> the localizer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-854) App submission fails on secure deploy
[
https://issues.apache.org/jira/browse/YARN-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Omkar Vinit Joshi updated YARN-854:
---
Attachment: YARN-854.20130619.2.patch
> App submission fails on secure deploy
> -
>
> Key: YARN-854
> URL: https://issues.apache.org/jira/browse/YARN-854
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.1.0-beta
>Reporter: Ramya Sunil
>Assignee: Omkar Vinit Joshi
>Priority: Blocker
> Fix For: 2.1.0-beta
>
> Attachments: YARN-854.20130619.1.patch, YARN-854.20130619.2.patch,
> YARN-854.20130619.patch
>
>
> App submission on secure cluster fails with the following exception:
> {noformat}
> INFO mapreduce.Job: Job jobID failed with state FAILED due to: Application
> applicationID failed 2 times due to AM Container for appattemptID exited with
> exitCode: -1000 due to: App initialization failed (255) with output: main :
> command provided 0
> main : user is qa_user
> javax.security.sasl.SaslException: DIGEST-MD5: digest response format
> violation. Mismatched response. [Caused by
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> DIGEST-MD5: digest response format violation. Mismatched response.]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:65)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:348)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> DIGEST-MD5: digest response format violation. Mismatched response.
> at org.apache.hadoop.ipc.Client.call(Client.java:1298)
> at org.apache.hadoop.ipc.Client.call(Client.java:1250)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:204)
> at $Proxy7.heartbeat(Unknown Source)
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
> ... 3 more
> .Failing this attempt.. Failing the application.
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-851) Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently.
[ https://issues.apache.org/jira/browse/YARN-851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-851: --- Attachment: YARN-851-20130619.patch > Share NMTokens using NMTokenCache (api-based) instead of memory based > approach which is used currently. > --- > > Key: YARN-851 > URL: https://issues.apache.org/jira/browse/YARN-851 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi > Attachments: YARN-851-20130618.patch, YARN-851-20130619.patch > > > It is a follow up ticket for YARN-694. Changing the way NMTokens are shared. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-851) Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently.
[ https://issues.apache.org/jira/browse/YARN-851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688812#comment-13688812 ] Omkar Vinit Joshi commented on YARN-851: Fixed above comments. removed getAllNMTokens(). > Share NMTokens using NMTokenCache (api-based) instead of memory based > approach which is used currently. > --- > > Key: YARN-851 > URL: https://issues.apache.org/jira/browse/YARN-851 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi > Attachments: YARN-851-20130618.patch, YARN-851-20130619.patch > > > It is a follow up ticket for YARN-694. Changing the way NMTokens are shared. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-854) App submission fails on secure deploy
[
https://issues.apache.org/jira/browse/YARN-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688822#comment-13688822
]
Hadoop QA commented on YARN-854:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588747/YARN-854.20130619.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestContainerManager
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1358//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1358//console
This message is automatically generated.
> App submission fails on secure deploy
> -
>
> Key: YARN-854
> URL: https://issues.apache.org/jira/browse/YARN-854
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.1.0-beta
>Reporter: Ramya Sunil
>Assignee: Omkar Vinit Joshi
>Priority: Blocker
> Fix For: 2.1.0-beta
>
> Attachments: YARN-854.20130619.1.patch, YARN-854.20130619.2.patch,
> YARN-854.20130619.patch
>
>
> App submission on secure cluster fails with the following exception:
> {noformat}
> INFO mapreduce.Job: Job jobID failed with state FAILED due to: Application
> applicationID failed 2 times due to AM Container for appattemptID exited with
> exitCode: -1000 due to: App initialization failed (255) with output: main :
> command provided 0
> main : user is qa_user
> javax.security.sasl.SaslException: DIGEST-MD5: digest response format
> violation. Mismatched response. [Caused by
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> DIGEST-MD5: digest response format violation. Mismatched response.]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:65)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:348)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> DIGEST-MD5: digest response format violation. Mismatched response.
> at org.apache.hadoop.ipc.Client.call(Client.java:1298)
> at org.apache.hadoop.ipc.Client.call(Client.java:1250)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:204)
> at $Proxy7.heartbeat(Unknown Source)
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
> ... 3 more
> .Failing this attempt.. Failing the application.
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more informat
[jira] [Commented] (YARN-851) Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently.
[
https://issues.apache.org/jira/browse/YARN-851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688833#comment-13688833
]
Hadoop QA commented on YARN-851:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588749/YARN-851-20130619.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client:
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1360//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1360//console
This message is automatically generated.
> Share NMTokens using NMTokenCache (api-based) instead of memory based
> approach which is used currently.
> ---
>
> Key: YARN-851
> URL: https://issues.apache.org/jira/browse/YARN-851
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Omkar Vinit Joshi
>Assignee: Omkar Vinit Joshi
> Attachments: YARN-851-20130618.patch, YARN-851-20130619.patch
>
>
> It is a follow up ticket for YARN-694. Changing the way NMTokens are shared.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-854) App submission fails on secure deploy
[
https://issues.apache.org/jira/browse/YARN-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688834#comment-13688834
]
Vinod Kumar Vavilapalli commented on YARN-854:
--
Looks good to me. +1. TestContainerManager is happening irrespective of this -
caused by YARN-848.
> App submission fails on secure deploy
> -
>
> Key: YARN-854
> URL: https://issues.apache.org/jira/browse/YARN-854
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.1.0-beta
>Reporter: Ramya Sunil
>Assignee: Omkar Vinit Joshi
>Priority: Blocker
> Fix For: 2.1.0-beta
>
> Attachments: YARN-854.20130619.1.patch, YARN-854.20130619.2.patch,
> YARN-854.20130619.patch
>
>
> App submission on secure cluster fails with the following exception:
> {noformat}
> INFO mapreduce.Job: Job jobID failed with state FAILED due to: Application
> applicationID failed 2 times due to AM Container for appattemptID exited with
> exitCode: -1000 due to: App initialization failed (255) with output: main :
> command provided 0
> main : user is qa_user
> javax.security.sasl.SaslException: DIGEST-MD5: digest response format
> violation. Mismatched response. [Caused by
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> DIGEST-MD5: digest response format violation. Mismatched response.]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:65)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:348)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> DIGEST-MD5: digest response format violation. Mismatched response.
> at org.apache.hadoop.ipc.Client.call(Client.java:1298)
> at org.apache.hadoop.ipc.Client.call(Client.java:1250)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:204)
> at $Proxy7.heartbeat(Unknown Source)
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
> ... 3 more
> .Failing this attempt.. Failing the application.
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-854) App submission fails on secure deploy
[
https://issues.apache.org/jira/browse/YARN-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688835#comment-13688835
]
Hadoop QA commented on YARN-854:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588747/YARN-854.20130619.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestContainerManager
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1359//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1359//console
This message is automatically generated.
> App submission fails on secure deploy
> -
>
> Key: YARN-854
> URL: https://issues.apache.org/jira/browse/YARN-854
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.1.0-beta
>Reporter: Ramya Sunil
>Assignee: Omkar Vinit Joshi
>Priority: Blocker
> Fix For: 2.1.0-beta
>
> Attachments: YARN-854.20130619.1.patch, YARN-854.20130619.2.patch,
> YARN-854.20130619.patch
>
>
> App submission on secure cluster fails with the following exception:
> {noformat}
> INFO mapreduce.Job: Job jobID failed with state FAILED due to: Application
> applicationID failed 2 times due to AM Container for appattemptID exited with
> exitCode: -1000 due to: App initialization failed (255) with output: main :
> command provided 0
> main : user is qa_user
> javax.security.sasl.SaslException: DIGEST-MD5: digest response format
> violation. Mismatched response. [Caused by
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> DIGEST-MD5: digest response format violation. Mismatched response.]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:65)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:348)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> DIGEST-MD5: digest response format violation. Mismatched response.
> at org.apache.hadoop.ipc.Client.call(Client.java:1298)
> at org.apache.hadoop.ipc.Client.call(Client.java:1250)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:204)
> at $Proxy7.heartbeat(Unknown Source)
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
> ... 3 more
> .Failing this attempt.. Failing the application.
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more informat
[jira] [Commented] (YARN-854) App submission fails on secure deploy
[
https://issues.apache.org/jira/browse/YARN-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688844#comment-13688844
]
Hudson commented on YARN-854:
-
Integrated in Hadoop-trunk-Commit #3987 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/3987/])
YARN-854. Fixing YARN bugs that are failing applications in secure
environment. Contributed by Omkar Vinit Joshi. (Revision 1494845)
Result = SUCCESS
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494845
Files :
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/RegisterApplicationMasterResponsePBImpl.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java
> App submission fails on secure deploy
> -
>
> Key: YARN-854
> URL: https://issues.apache.org/jira/browse/YARN-854
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.1.0-beta
>Reporter: Ramya Sunil
>Assignee: Omkar Vinit Joshi
>Priority: Blocker
> Fix For: 2.1.0-beta
>
> Attachments: YARN-854.20130619.1.patch, YARN-854.20130619.2.patch,
> YARN-854.20130619.patch
>
>
> App submission on secure cluster fails with the following exception:
> {noformat}
> INFO mapreduce.Job: Job jobID failed with state FAILED due to: Application
> applicationID failed 2 times due to AM Container for appattemptID exited with
> exitCode: -1000 due to: App initialization failed (255) with output: main :
> command provided 0
> main : user is qa_user
> javax.security.sasl.SaslException: DIGEST-MD5: digest response format
> violation. Mismatched response. [Caused by
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> DIGEST-MD5: digest response format violation. Mismatched response.]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:65)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:235)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:348)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> DIGEST-MD5: digest response format violation. Mismatched response.
> at org.apache.hadoop.ipc.Client.call(Client.java:1298)
> at org.apache.hadoop.ipc.Client.call(Client.java:1250)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:204)
> at $Proxy7.heartbeat(Unknown Source)
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
> ... 3 more
> .Failing this attempt.. Failing the application.
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-859) "Applications Per User" is giving ambiguous values in scheduler UI
Nishan Shetty created YARN-859: -- Summary: "Applications Per User" is giving ambiguous values in scheduler UI Key: YARN-859 URL: https://issues.apache.org/jira/browse/YARN-859 Project: Hadoop YARN Issue Type: Bug Components: capacityscheduler Affects Versions: 2.0.1-alpha Reporter: Nishan Shetty Priority: Minor 1.Configure "yarn.scheduler.capacity.root.default.user-limit-factor" as 2 Observe that Applications Per User values in scheuler UI is giving ambiguous values Max applications per user cannot be more than that of cluster Max Applications: 1000 Max Applications Per User: 2000 Max Active Applications: 5 Max Active Applications Per User: 10 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-827) Need to make Resource arithmetic methods public
[ https://issues.apache.org/jira/browse/YARN-827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688929#comment-13688929 ] Arun C Murthy commented on YARN-827: Let's mark all as @Private please. > Need to make Resource arithmetic methods public > --- > > Key: YARN-827 > URL: https://issues.apache.org/jira/browse/YARN-827 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Zhijie Shen >Priority: Critical > Attachments: YARN-827.patch > > > org.apache.hadoop.yarn.server.resourcemanager.resource has stuff like > Resources and Calculators that help compare/add resources etc. Without these > users will be forced to replicate the logic, potentially incorrectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-853) maximum-am-resource-percent doesn't work consistently with refreshQueues
[ https://issues.apache.org/jira/browse/YARN-853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated YARN-853: --- Attachment: YARN-853.patch > maximum-am-resource-percent doesn't work consistently with refreshQueues > > > Key: YARN-853 > URL: https://issues.apache.org/jira/browse/YARN-853 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 3.0.0, 2.1.0-beta, 2.0.5-alpha >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: YARN-853.patch > > > If we update yarn.scheduler.capacity.maximum-am-resource-percent / > yarn.scheduler.capacity..maximum-am-resource-percent > configuration and then do the refreshNodes, it uses the new config value to > calculate Max Active Applications and Max Active Application Per User. If we > add new node after issuing 'rmadmin -refreshQueues' command, it uses the old > maximum-am-resource-percent config value to calculate Max Active Applications > and Max Active Application Per User. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
