[jira] [Commented] (YARN-1363) Get / Cancel / Renew delegation token api should be non blocking
[
https://issues.apache.org/jira/browse/YARN-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915500#comment-13915500
]
Hadoop QA commented on YARN-1363:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12631670/YARN-1363.11.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3213//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3213//console
This message is automatically generated.
> Get / Cancel / Renew delegation token api should be non blocking
>
>
> Key: YARN-1363
> URL: https://issues.apache.org/jira/browse/YARN-1363
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Omkar Vinit Joshi
>Assignee: Zhijie Shen
> Attachments: YARN-1363.1.patch, YARN-1363.10.patch,
> YARN-1363.10.patch, YARN-1363.11.patch, YARN-1363.12.patch,
> YARN-1363.2.patch, YARN-1363.3.patch, YARN-1363.4.patch, YARN-1363.5.patch,
> YARN-1363.6.patch, YARN-1363.7.patch, YARN-1363.8.patch, YARN-1363.9.patch
>
>
> Today GetDelgationToken, CancelDelegationToken and RenewDelegationToken are
> all blocking apis.
> * As a part of these calls we try to update RMStateStore and that may slow it
> down.
> * Now as we have limited number of client request handlers we may fill up
> client handlers quickly.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1445: Attachment: YARN-1445.6.patch recreate the patch based on the latest trunk > Separate FINISHING and FINISHED state in YarnApplicationState > - > > Key: YARN-1445 > URL: https://issues.apache.org/jira/browse/YARN-1445 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-1445.1.patch, YARN-1445.2.patch, YARN-1445.3.patch, > YARN-1445.4.patch, YARN-1445.5.patch, YARN-1445.5.patch, YARN-1445.6.patch > > > Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to > YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1363) Get / Cancel / Renew delegation token api should be non blocking
[ https://issues.apache.org/jira/browse/YARN-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-1363: -- Attachment: YARN-1363.12.patch Fix a nit > Get / Cancel / Renew delegation token api should be non blocking > > > Key: YARN-1363 > URL: https://issues.apache.org/jira/browse/YARN-1363 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Zhijie Shen > Attachments: YARN-1363.1.patch, YARN-1363.10.patch, > YARN-1363.10.patch, YARN-1363.11.patch, YARN-1363.12.patch, > YARN-1363.2.patch, YARN-1363.3.patch, YARN-1363.4.patch, YARN-1363.5.patch, > YARN-1363.6.patch, YARN-1363.7.patch, YARN-1363.8.patch, YARN-1363.9.patch > > > Today GetDelgationToken, CancelDelegationToken and RenewDelegationToken are > all blocking apis. > * As a part of these calls we try to update RMStateStore and that may slow it > down. > * Now as we have limited number of client request handlers we may fill up > client handlers quickly. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1528) ZKRMStateStore and EmbeddedElector should allow setting ZK auth information
[ https://issues.apache.org/jira/browse/YARN-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915494#comment-13915494 ] Karthik Kambatla commented on YARN-1528: I meant more util methods being added in the future - zk connections, retries etc. for SharedCacheManager etc. > ZKRMStateStore and EmbeddedElector should allow setting ZK auth information > --- > > Key: YARN-1528 > URL: https://issues.apache.org/jira/browse/YARN-1528 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.3.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla >Priority: Blocker > Attachments: yarn-1528-1.patch > > > ZK store and embedded election allow setting ZK-acls but not auth information -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1528) ZKRMStateStore and EmbeddedElector should allow setting ZK auth information
[ https://issues.apache.org/jira/browse/YARN-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915487#comment-13915487 ] Karthik Kambatla commented on YARN-1528: The configs are YARN specific, will have to send the config names across if we move them to ZKUtil. Also, I see more RM-specific ZK-util methods being added to this new class. > ZKRMStateStore and EmbeddedElector should allow setting ZK auth information > --- > > Key: YARN-1528 > URL: https://issues.apache.org/jira/browse/YARN-1528 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.3.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla >Priority: Blocker > Attachments: yarn-1528-1.patch > > > ZK store and embedded election allow setting ZK-acls but not auth information -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1528) ZKRMStateStore and EmbeddedElector should allow setting ZK auth information
[ https://issues.apache.org/jira/browse/YARN-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915467#comment-13915467 ] Sandy Ryza commented on YARN-1528: -- Can those helpers just go in ZKUtil? LGTM otherwise. > ZKRMStateStore and EmbeddedElector should allow setting ZK auth information > --- > > Key: YARN-1528 > URL: https://issues.apache.org/jira/browse/YARN-1528 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.3.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla >Priority: Blocker > Attachments: yarn-1528-1.patch > > > ZK store and embedded election allow setting ZK-acls but not auth information -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1761) RMAdminCLI should check whether HA is enabled before executes transitionToActive/transitionToStandby
[ https://issues.apache.org/jira/browse/YARN-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915465#comment-13915465 ] Xuan Gong commented on YARN-1761: - Simply using conf.get(YarnConfiguration.RM_HA_ENABLED) to check whether RM_HA is enabled is not enough. I think we should use ConfigurationProvider to get the configurations. > RMAdminCLI should check whether HA is enabled before executes > transitionToActive/transitionToStandby > > > Key: YARN-1761 > URL: https://issues.apache.org/jira/browse/YARN-1761 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1758) MiniYARNCluster broken post YARN-1666
[
https://issues.apache.org/jira/browse/YARN-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915464#comment-13915464
]
Hadoop QA commented on YARN-1758:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12631668/YARN-1758.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3212//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3212//console
This message is automatically generated.
> MiniYARNCluster broken post YARN-1666
> -
>
> Key: YARN-1758
> URL: https://issues.apache.org/jira/browse/YARN-1758
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Hitesh Shah
>Assignee: Xuan Gong
>Priority: Blocker
> Attachments: YARN-1758.1.patch
>
>
> NPE seen when trying to use MiniYARNCluster
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1445) Separate FINISHING and FINISHED state in YarnApplicationState
[ https://issues.apache.org/jira/browse/YARN-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-1445: -- Target Version/s: 2.4.0 > Separate FINISHING and FINISHED state in YarnApplicationState > - > > Key: YARN-1445 > URL: https://issues.apache.org/jira/browse/YARN-1445 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-1445.1.patch, YARN-1445.2.patch, YARN-1445.3.patch, > YARN-1445.4.patch, YARN-1445.5.patch, YARN-1445.5.patch > > > Today, we will transmit both RMAppState.FINISHING and RMAppState.FINISHED to > YarnApplicationState.FINISHED. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1363) Get / Cancel / Renew delegation token api should be non blocking
[ https://issues.apache.org/jira/browse/YARN-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915449#comment-13915449 ] Zhijie Shen commented on YARN-1363: --- We need to think about whether get/cancel/renew operation is idempotent to survive RM fail over. If it's a big additional change, we should solve it another ticket. > Get / Cancel / Renew delegation token api should be non blocking > > > Key: YARN-1363 > URL: https://issues.apache.org/jira/browse/YARN-1363 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Zhijie Shen > Attachments: YARN-1363.1.patch, YARN-1363.10.patch, > YARN-1363.10.patch, YARN-1363.11.patch, YARN-1363.2.patch, YARN-1363.3.patch, > YARN-1363.4.patch, YARN-1363.5.patch, YARN-1363.6.patch, YARN-1363.7.patch, > YARN-1363.8.patch, YARN-1363.9.patch > > > Today GetDelgationToken, CancelDelegationToken and RenewDelegationToken are > all blocking apis. > * As a part of these calls we try to update RMStateStore and that may slow it > down. > * Now as we have limited number of client request handlers we may fill up > client handlers quickly. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1363) Get / Cancel / Renew delegation token api should be non blocking
[ https://issues.apache.org/jira/browse/YARN-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-1363: -- Attachment: YARN-1363.11.patch > Get / Cancel / Renew delegation token api should be non blocking > > > Key: YARN-1363 > URL: https://issues.apache.org/jira/browse/YARN-1363 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Zhijie Shen > Attachments: YARN-1363.1.patch, YARN-1363.10.patch, > YARN-1363.10.patch, YARN-1363.11.patch, YARN-1363.2.patch, YARN-1363.3.patch, > YARN-1363.4.patch, YARN-1363.5.patch, YARN-1363.6.patch, YARN-1363.7.patch, > YARN-1363.8.patch, YARN-1363.9.patch > > > Today GetDelgationToken, CancelDelegationToken and RenewDelegationToken are > all blocking apis. > * As a part of these calls we try to update RMStateStore and that may slow it > down. > * Now as we have limited number of client request handlers we may fill up > client handlers quickly. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1363) Get / Cancel / Renew delegation token api should be non blocking
[ https://issues.apache.org/jira/browse/YARN-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915442#comment-13915442 ] Zhijie Shen commented on YARN-1363: --- Jian, thanks for the comments. bq. we don’t need a separate api getIsObtained in GetDelegationTokenResponse, setting the token as null means not obtained. This is because I want to make three kinds of responses semantically consistent, and since we anyway make the change incompatible, it's good to change to what looks better. bq. RMDelegationTokenEventType: CREATE_RMDT-> CREATE, similarly others. Done bq. RMDelegationTokenIdentifier.renew(): I think it’s fine to blockingly renew the token here instead of keeping polling as this renew method is called by DelegationTokenRenewer which is already asynchronously renewing. This is because renew/cancel is called from ApplicationClientProtocol. bq. RMDTOperationState.IN_PROFESS typo Done bq. can RMDTOperationKey contain RMDTIdentifer and the operation type as the key? I understand you concern here, but I intentionally extract owner, user and renewer from RMDTIdentifer to construct the key, as they are the only fields in RMDTIdentifer that are used to by hashCode() and equals(). However, to make the code clearer, I add a constructor for RMDTOperationKey to take RMDTIdentifer as param. bq. createTokenAsync: we can return the RMDTOperationValue which includes the token, and so we don’t need a separate method pollToken to get the token. similarly for renew I did the following instead: in TokenAsync, instead of returning the state, returning the result/throwing the exception if the state is FINISHED, which should make the code clearer. bq. the following code is duplicated in the try and catch, similarly for cancel. I refactored the code in handleRMDelegationTokenEvent. > Get / Cancel / Renew delegation token api should be non blocking > > > Key: YARN-1363 > URL: https://issues.apache.org/jira/browse/YARN-1363 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Zhijie Shen > Attachments: YARN-1363.1.patch, YARN-1363.10.patch, > YARN-1363.10.patch, YARN-1363.2.patch, YARN-1363.3.patch, YARN-1363.4.patch, > YARN-1363.5.patch, YARN-1363.6.patch, YARN-1363.7.patch, YARN-1363.8.patch, > YARN-1363.9.patch > > > Today GetDelgationToken, CancelDelegationToken and RenewDelegationToken are > all blocking apis. > * As a part of these calls we try to update RMStateStore and that may slow it > down. > * Now as we have limited number of client request handlers we may fill up > client handlers quickly. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1758) MiniYARNCluster broken post YARN-1666
[ https://issues.apache.org/jira/browse/YARN-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915440#comment-13915440 ] Xuan Gong commented on YARN-1758: - Upload a patch to fix this issue, which includes: 1. if Configuration file is not exist, instead of throwing exception, it will simply return null 2. RM, Scheduler or other active services will do the null check before calling conf.addResource(). If the inputStream is NULL (meaning the configuration files are not exist), we will not call conf.addResource() to reload the configuration. In this case, we will get NullPointerException 3. create a testcase to verify that RM can start successfully even we do not provide any configuration files. > MiniYARNCluster broken post YARN-1666 > - > > Key: YARN-1758 > URL: https://issues.apache.org/jira/browse/YARN-1758 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Xuan Gong >Priority: Blocker > Attachments: YARN-1758.1.patch > > > NPE seen when trying to use MiniYARNCluster -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1758) MiniYARNCluster broken post YARN-1666
[ https://issues.apache.org/jira/browse/YARN-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1758: Attachment: YARN-1758.1.patch > MiniYARNCluster broken post YARN-1666 > - > > Key: YARN-1758 > URL: https://issues.apache.org/jira/browse/YARN-1758 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Xuan Gong >Priority: Blocker > Attachments: YARN-1758.1.patch > > > NPE seen when trying to use MiniYARNCluster -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1723) AMRMClientAsync missing blacklist addition and removal functionality
[ https://issues.apache.org/jira/browse/YARN-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-1723: - Issue Type: Sub-task (was: Bug) Parent: YARN-386 > AMRMClientAsync missing blacklist addition and removal functionality > > > Key: YARN-1723 > URL: https://issues.apache.org/jira/browse/YARN-1723 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.2.0 >Reporter: Bikas Saha > Fix For: 2.4.0 > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1723) AMRMClientAsync missing blacklist addition and removal functionality
[ https://issues.apache.org/jira/browse/YARN-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915368#comment-13915368 ] Junping Du commented on YARN-1723: -- Like YARN-771, move it to under YARN-386 to get better tracked. > AMRMClientAsync missing blacklist addition and removal functionality > > > Key: YARN-1723 > URL: https://issues.apache.org/jira/browse/YARN-1723 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Bikas Saha > Fix For: 2.4.0 > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1758) MiniYARNCluster broken post YARN-1666
[ https://issues.apache.org/jira/browse/YARN-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1758: -- Priority: Blocker (was: Major) Target Version/s: 2.4.0 > MiniYARNCluster broken post YARN-1666 > - > > Key: YARN-1758 > URL: https://issues.apache.org/jira/browse/YARN-1758 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Xuan Gong >Priority: Blocker > > NPE seen when trying to use MiniYARNCluster -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1528) ZKRMStateStore and EmbeddedElector should allow setting ZK auth information
[
https://issues.apache.org/jira/browse/YARN-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915289#comment-13915289
]
Hadoop QA commented on YARN-1528:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12631614/yarn-1528-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3211//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3211//console
This message is automatically generated.
> ZKRMStateStore and EmbeddedElector should allow setting ZK auth information
> ---
>
> Key: YARN-1528
> URL: https://issues.apache.org/jira/browse/YARN-1528
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.3.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
>Priority: Blocker
> Attachments: yarn-1528-1.patch
>
>
> ZK store and embedded election allow setting ZK-acls but not auth information
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1525) Web UI should redirect to active RM when HA is enabled.
[ https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915279#comment-13915279 ] Cindy Li commented on YARN-1525: @Karthik, I've been trying to reproduce your problem. It takes me a while to setup a secure cluster. > Web UI should redirect to active RM when HA is enabled. > --- > > Key: YARN-1525 > URL: https://issues.apache.org/jira/browse/YARN-1525 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Cindy Li > Attachments: YARN1525.patch, YARN1525.patch, YARN1525.patch, > YARN1525.patch, YARN1525.patch.v1, YARN1525.patch.v2, YARN1525.patch.v3, > YARN1525.v7.patch, YARN1525.v7.patch, YARN1525.v8.patch, YARN1525.v9.patch > > > When failover happens, web UI should redirect to the current active rm. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1525) Web UI should redirect to active RM when HA is enabled.
[ https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915275#comment-13915275 ] Karthik Kambatla commented on YARN-1525: Just checking if there are any updates here. [~cindy2012] - if you are caught up with other things, we can lend a hand. > Web UI should redirect to active RM when HA is enabled. > --- > > Key: YARN-1525 > URL: https://issues.apache.org/jira/browse/YARN-1525 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Cindy Li > Attachments: YARN1525.patch, YARN1525.patch, YARN1525.patch, > YARN1525.patch, YARN1525.patch.v1, YARN1525.patch.v2, YARN1525.patch.v3, > YARN1525.v7.patch, YARN1525.v7.patch, YARN1525.v8.patch, YARN1525.v9.patch > > > When failover happens, web UI should redirect to the current active rm. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1577) Unmanaged AM is broken because of YARN-1493
[ https://issues.apache.org/jira/browse/YARN-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915243#comment-13915243 ] Naren Koneru commented on YARN-1577: I see that a patch has been submitted today for YARN-1389. I will look into this once that issue is resolved since we are gonna make use of the APIs there (getApplicationAttemptReport) to fi the UnManagedAMLauncher, etc > Unmanaged AM is broken because of YARN-1493 > --- > > Key: YARN-1577 > URL: https://issues.apache.org/jira/browse/YARN-1577 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.3.0 >Reporter: Jian He >Assignee: Naren Koneru >Priority: Blocker > > Today unmanaged AM client is waiting for app state to be Accepted to launch > the AM. This is broken since we changed in YARN-1493 to start the attempt > after the application is Accepted. We may need to introduce an attempt state > report that client can rely on to query the attempt state and choose to > launch the unmanaged AM. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1765) Write test cases to verify that killApplication API works in RM HA
[ https://issues.apache.org/jira/browse/YARN-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1765: -- Assignee: Xuan Gong (was: Vinod Kumar Vavilapalli) > Write test cases to verify that killApplication API works in RM HA > -- > > Key: YARN-1765 > URL: https://issues.apache.org/jira/browse/YARN-1765 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-1765.1.patch, YARN-1765.2.patch, YARN-1765.2.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1363) Get / Cancel / Renew delegation token api should be non blocking
[
https://issues.apache.org/jira/browse/YARN-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915176#comment-13915176
]
Jian He commented on YARN-1363:
---
The code looks much simpler now than earlier's approach, thanks for the update,
some comments:
- we don’t need a separate api getIsObtained in GetDelegationTokenResponse,
setting the token as null means not obtained.
- RMDelegationTokenEventType: CREATE_RMDT-> CREATE, similarly others.
- RMDelegationTokenIdentifier.renew(): I think it’s fine to blockingly renew
the token here instead of keeping polling as this renew method is called by
DelegationTokenRenewer which is already asynchronously renewing.
- RMDTOperationState.IN_PROFESS typo
- can RMDTOperationKey contain RMDTIdentifer and the operation type as the key?
- createTokenAsync: we can return the RMDTOperationValue which includes the
token, and so we don’t need a separate method pollToken to get the token.
similarly for renew
- the following code is duplicated in the try and catch, similarly for cancel.
{code}
key = new RMDTOperationKey(
rmDTIdentifier.getOwner(), rmDTIdentifier.getRealUser(),
rmDTIdentifier.getRenewer(), RMDTOperationType.RENEW);
value = operations.get(key);
if (value != null) {
value.state = RMDTOperationState.FINISHED;
value.timestamp = System.currentTimeMillis();
{code}
> Get / Cancel / Renew delegation token api should be non blocking
>
>
> Key: YARN-1363
> URL: https://issues.apache.org/jira/browse/YARN-1363
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Omkar Vinit Joshi
>Assignee: Zhijie Shen
> Attachments: YARN-1363.1.patch, YARN-1363.10.patch,
> YARN-1363.10.patch, YARN-1363.2.patch, YARN-1363.3.patch, YARN-1363.4.patch,
> YARN-1363.5.patch, YARN-1363.6.patch, YARN-1363.7.patch, YARN-1363.8.patch,
> YARN-1363.9.patch
>
>
> Today GetDelgationToken, CancelDelegationToken and RenewDelegationToken are
> all blocking apis.
> * As a part of these calls we try to update RMStateStore and that may slow it
> down.
> * Now as we have limited number of client request handlers we may fill up
> client handlers quickly.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1768) yarn kill non-existent application is too verbose
[ https://issues.apache.org/jira/browse/YARN-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1768: -- Target Version/s: 2.4.0 > yarn kill non-existent application is too verbose > - > > Key: YARN-1768 > URL: https://issues.apache.org/jira/browse/YARN-1768 > Project: Hadoop YARN > Issue Type: Bug > Components: client >Affects Versions: 2.2.0 >Reporter: Hitesh Shah >Priority: Minor > > Instead of catching ApplicationNotFound and logging a simple app not found > message, the whole stack trace is logged. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1704) Review LICENSE and NOTICE to reflect new levelDB releated libraries being used
[ https://issues.apache.org/jira/browse/YARN-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billie Rinaldi updated YARN-1704: - Attachment: YARN-1704.3.patch Updated patch specifying that we're bundling binaries in our binary distribution, and adding links for the projects mentioned in the license file. > Review LICENSE and NOTICE to reflect new levelDB releated libraries being used > -- > > Key: YARN-1704 > URL: https://issues.apache.org/jira/browse/YARN-1704 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi >Priority: Blocker > Attachments: YARN-1704.1.patch, YARN-1704.2.patch, YARN-1704.3.patch > > > Make any changes necessary in LICENSE and NOTICE related to dependencies > introduced by the application timeline store. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1768) yarn kill non-existent application is too verbose
[ https://issues.apache.org/jira/browse/YARN-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated YARN-1768: -- Priority: Minor (was: Major) > yarn kill non-existent application is too verbose > - > > Key: YARN-1768 > URL: https://issues.apache.org/jira/browse/YARN-1768 > Project: Hadoop YARN > Issue Type: Bug > Components: client >Affects Versions: 2.2.0 >Reporter: Hitesh Shah >Priority: Minor > > Instead of catching ApplicationNotFound and logging a simple app not found > message, the whole stack trace is logged. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1768) yarn kill non-existent application is too verbose
[ https://issues.apache.org/jira/browse/YARN-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated YARN-1768: -- Component/s: client > yarn kill non-existent application is too verbose > - > > Key: YARN-1768 > URL: https://issues.apache.org/jira/browse/YARN-1768 > Project: Hadoop YARN > Issue Type: Bug > Components: client >Affects Versions: 2.2.0 >Reporter: Hitesh Shah >Priority: Minor > > Instead of catching ApplicationNotFound and logging a simple app not found > message, the whole stack trace is logged. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (YARN-1765) Write test cases to verify that killApplication API works in RM HA
[ https://issues.apache.org/jira/browse/YARN-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli reassigned YARN-1765: - Assignee: Vinod Kumar Vavilapalli (was: Xuan Gong) > Write test cases to verify that killApplication API works in RM HA > -- > > Key: YARN-1765 > URL: https://issues.apache.org/jira/browse/YARN-1765 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Vinod Kumar Vavilapalli > Attachments: YARN-1765.1.patch, YARN-1765.2.patch, YARN-1765.2.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1528) ZKRMStateStore and EmbeddedElector should allow setting ZK auth information
[ https://issues.apache.org/jira/browse/YARN-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1528: --- Attachment: yarn-1528-1.patch Straight-forward patch. Added RMZKUtils since both ZK-store and EmbeddedElectorService use this. > ZKRMStateStore and EmbeddedElector should allow setting ZK auth information > --- > > Key: YARN-1528 > URL: https://issues.apache.org/jira/browse/YARN-1528 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.3.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla >Priority: Blocker > Attachments: yarn-1528-1.patch > > > ZK store and embedded election allow setting ZK-acls but not auth information -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1768) yarn kill non-existent application is too verbose
[ https://issues.apache.org/jira/browse/YARN-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915130#comment-13915130 ] Hitesh Shah commented on YARN-1768: --- hadoop-yarn-2.2.0/bin/yarn application -kill application_1393537192691_0006 WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files. 14/02/27 14:29:32:822 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/02/27 14:29:33:281 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8020 Exception in thread "main" org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1393537192691_0006' doesn't exist in RM. at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:247) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:120) at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:241) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:394) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042) > yarn kill non-existent application is too verbose > - > > Key: YARN-1768 > URL: https://issues.apache.org/jira/browse/YARN-1768 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Hitesh Shah > > Instead of catching ApplicationNotFound and logging a simple app not found > message, the whole stack trace is logged. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1768) yarn kill non-existent application is too verbose
[ https://issues.apache.org/jira/browse/YARN-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated YARN-1768: -- Affects Version/s: 2.2.0 > yarn kill non-existent application is too verbose > - > > Key: YARN-1768 > URL: https://issues.apache.org/jira/browse/YARN-1768 > Project: Hadoop YARN > Issue Type: Bug > Components: client >Affects Versions: 2.2.0 >Reporter: Hitesh Shah >Priority: Minor > > Instead of catching ApplicationNotFound and logging a simple app not found > message, the whole stack trace is logged. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1768) yarn kill non-existent application is too verbose
Hitesh Shah created YARN-1768: - Summary: yarn kill non-existent application is too verbose Key: YARN-1768 URL: https://issues.apache.org/jira/browse/YARN-1768 Project: Hadoop YARN Issue Type: Bug Reporter: Hitesh Shah Instead of catching ApplicationNotFound and logging a simple app not found message, the whole stack trace is logged. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1704) Review LICENSE and NOTICE to reflect new levelDB releated libraries being used
[ https://issues.apache.org/jira/browse/YARN-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915119#comment-13915119 ] Jason Lowe commented on YARN-1704: -- bq. The license for snappy mentions test resources that have different licenses (https://code.google.com/p/snappy/source/browse/trunk/COPYING), but I don't think those files are included in the leveldbjni native libraries. Do you agree? Agreed. The binary libraries in leveldbjni-all do not contain the test resources that are covered by those other licenses. bq. For the NOTICE file, I changed the wording to match the LICENSE file, "The binary distribution of this product bundles X, which has the following notices." Do you think we should change the wording in license and notice to say this product bundles the binaries of X? IANAL but it makes sense to me that we should state we are bundling the binaries of leveldb and snappy since that's what leveldbjni-all does. > Review LICENSE and NOTICE to reflect new levelDB releated libraries being used > -- > > Key: YARN-1704 > URL: https://issues.apache.org/jira/browse/YARN-1704 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi >Priority: Blocker > Attachments: YARN-1704.1.patch, YARN-1704.2.patch > > > Make any changes necessary in LICENSE and NOTICE related to dependencies > introduced by the application timeline store. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1704) Review LICENSE and NOTICE to reflect new levelDB releated libraries being used
[ https://issues.apache.org/jira/browse/YARN-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915111#comment-13915111 ] Jason Lowe commented on YARN-1704: -- The jar contains binaries for: - linux32 - linux64 - osx - windows32 - windows64 > Review LICENSE and NOTICE to reflect new levelDB releated libraries being used > -- > > Key: YARN-1704 > URL: https://issues.apache.org/jira/browse/YARN-1704 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi >Priority: Blocker > Attachments: YARN-1704.1.patch, YARN-1704.2.patch > > > Make any changes necessary in LICENSE and NOTICE related to dependencies > introduced by the application timeline store. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1410) Handle RM fails over after getApplicationID() and before submitApplication().
[ https://issues.apache.org/jira/browse/YARN-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915112#comment-13915112 ] Xuan Gong commented on YARN-1410: - bq. When can appId be null in the submission-context? It will not happen right now. Take DistributedShell as an example, before we submit the application, we will get an applicationId which is used to set some directories for local resources, shell_script, etc. That is why we need the applicationId as the global unique ID. It may not be necessary for users’ own applications. They can just simply call yarnClient#submitApplication() to submit their applications. That is why we add null check for applicationId in ClientRMService# submitApplication(). If we really think this check is un-necessary, we should at least document this in yarnClient#submitApplication(), saying, “Before you use this api to submit the application, make sure you have an applicationId”. Also we should not expose those apis, such as ApplicationSubmissionContext#newInstance() or BuilderUtils# newApplicationSubmissionContext(), to public for users to create ApplicationSubmissionContext object. We should only get ApplicationSubmissionContext by calling getNewApplication(), which can get applicationId, too. bq. Documentation Sure. I will add those. bq. Does DistributedShell need any changes to reflect the potential change in appId after fail-over? If so, let's fix that too here. Please file a MR ticket to fix MapReduce too if needed. Fixes are needed in either case if anyone caches the appId from GetNewApplicationResponse. I do not think we need make any changes. DistributedShell and MapReduce has applicationId before submits the application. When failover happens, the old applicationId will be re-used. So, the applicationId return from yarnClient#submitApplication() or from SubmitApplicationResponse is the same as the application we used to submit application. bq. Orthogonal to this ticket, we need to make sure clients don't pass in invalid application-IDs as part of the submission-context. It can be validated by simply looking at our counter and may be also caching recently used appIDs (atleast within a single RM). I remember we had a JIRA for this somewhere. We also need throttles so that malicious client don't exhaust appIDs. ApplicationId has two pieces of information: ResourceManager.getClusterTimeStamp() (the time RM become active) and applicationCounter.incrementAndGet(). Since we allow user to re-use old or create their own applicationId, the situation you described may happen. For the HA case, if failover happens several times, clusterTimeStamp for the same RM will be different. Because everytime when RM become active, we will get a new clusterTimeStamp. So, we could check the clusterTimeStamp and app counter at the same time. For the given applicationid, if applicationId#getClusterTimestamp == ResourceManager.getClusterTimeStamp() and applicationId#getId > applicationCounter.get(), then we can consider this applicationId as malicious applicationid. > Handle RM fails over after getApplicationID() and before submitApplication(). > - > > Key: YARN-1410 > URL: https://issues.apache.org/jira/browse/YARN-1410 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Xuan Gong > Attachments: YARN-1410-outline.patch, YARN-1410.1.patch, > YARN-1410.2.patch, YARN-1410.2.patch, YARN-1410.3.patch, YARN-1410.4.patch, > YARN-1410.5.patch, YARN-1410.6.patch, YARN-1410.7.patch, YARN-1410.8.patch, > YARN-1410.9.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > App submission involves > 1) creating appId > 2) using that appId to submit an ApplicationSubmissionContext to the user. > The client may have obtained an appId from an RM, the RM may have failed > over, and the client may submit the app to the new RM. > Since the new RM has a different notion of cluster timestamp (used to create > app id) the new RM may reject the app submission resulting in unexpected > failure on the client side. > The same may happen for other 2 step client API operations. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1704) Review LICENSE and NOTICE to reflect new levelDB releated libraries being used
[ https://issues.apache.org/jira/browse/YARN-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915102#comment-13915102 ] Vinod Kumar Vavilapalli commented on YARN-1704: --- bq. Running nm on the .so embedded in the jar shows the symbols are there, and it works on a machine that doesn't have those RPMs installed. Interesting. Off topic, but does this mean this jar will only work on only one platform where it is is built? What about Mac/Windows/Ubuntu etc? > Review LICENSE and NOTICE to reflect new levelDB releated libraries being used > -- > > Key: YARN-1704 > URL: https://issues.apache.org/jira/browse/YARN-1704 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi >Priority: Blocker > Attachments: YARN-1704.1.patch, YARN-1704.2.patch > > > Make any changes necessary in LICENSE and NOTICE related to dependencies > introduced by the application timeline store. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1704) Review LICENSE and NOTICE to reflect new levelDB releated libraries being used
[ https://issues.apache.org/jira/browse/YARN-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billie Rinaldi updated YARN-1704: - Attachment: YARN-1704.2.patch Thanks for the additional review. I have added licenses for leveldb and snappy into the LICENSE file. The license for snappy mentions test resources that have different licenses (https://code.google.com/p/snappy/source/browse/trunk/COPYING), but I don't think those files are included in the leveldbjni native libraries. Do you agree? For the NOTICE file, I changed the wording to match the LICENSE file, "The binary distribution of this product bundles X, which has the following notices." Do you think we should change the wording in license and notice to say this product bundles the binaries of X? > Review LICENSE and NOTICE to reflect new levelDB releated libraries being used > -- > > Key: YARN-1704 > URL: https://issues.apache.org/jira/browse/YARN-1704 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi >Priority: Blocker > Attachments: YARN-1704.1.patch, YARN-1704.2.patch > > > Make any changes necessary in LICENSE and NOTICE related to dependencies > introduced by the application timeline store. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1704) Review LICENSE and NOTICE to reflect new levelDB releated libraries being used
[ https://issues.apache.org/jira/browse/YARN-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915040#comment-13915040 ] Jason Lowe commented on YARN-1704: -- Snappy and leveldb are included in leveldbjni-all. Running nm on the .so embedded in the jar shows the symbols are there, and it works on a machine that doesn't have those RPMs installed. So we'll need to cover those as well, and both appear to have a BSD 3-clause license. > Review LICENSE and NOTICE to reflect new levelDB releated libraries being used > -- > > Key: YARN-1704 > URL: https://issues.apache.org/jira/browse/YARN-1704 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi >Priority: Blocker > Attachments: YARN-1704.1.patch > > > Make any changes necessary in LICENSE and NOTICE related to dependencies > introduced by the application timeline store. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1729) TimelineWebServices always passes primary and secondary filters as strings
[
https://issues.apache.org/jira/browse/YARN-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915011#comment-13915011
]
Hadoop QA commented on YARN-1729:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12631589/YARN-1729.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3210//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3210//console
This message is automatically generated.
> TimelineWebServices always passes primary and secondary filters as strings
> --
>
> Key: YARN-1729
> URL: https://issues.apache.org/jira/browse/YARN-1729
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
> Attachments: YARN-1729.1.patch, YARN-1729.2.patch, YARN-1729.3.patch,
> YARN-1729.4.patch
>
>
> Primary filters and secondary filter values can be arbitrary json-compatible
> Object. The web services should determine if the filters specified as query
> parameters are objects or strings before passing them to the store.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1704) Review LICENSE and NOTICE to reflect new levelDB releated libraries being used
[ https://issues.apache.org/jira/browse/YARN-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1704: -- Priority: Blocker (was: Major) Target Version/s: 2.4.0 Summary: Review LICENSE and NOTICE to reflect new levelDB releated libraries being used (was: Review LICENSE and NOTICE) > Review LICENSE and NOTICE to reflect new levelDB releated libraries being used > -- > > Key: YARN-1704 > URL: https://issues.apache.org/jira/browse/YARN-1704 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi >Priority: Blocker > Attachments: YARN-1704.1.patch > > > Make any changes necessary in LICENSE and NOTICE related to dependencies > introduced by the application timeline store. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (YARN-1704) Review LICENSE and NOTICE
[ https://issues.apache.org/jira/browse/YARN-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli reassigned YARN-1704: - Assignee: Billie Rinaldi (was: Vinod Kumar Vavilapalli) Tx for doing this Billie! Your outline looks fine to me. Just one thing - may be in the notice file, we should explicitly say what 'include' means - that they are runtime dependencies whose binaries are packaged together with our releases. Thoughts? [~jlowe], care to pitch in and cross-verify? Just to be sure. Additional eyeballs help in matters like this. I saw you using this library for the NM restart stuff, so.. > Review LICENSE and NOTICE > - > > Key: YARN-1704 > URL: https://issues.apache.org/jira/browse/YARN-1704 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Attachments: YARN-1704.1.patch > > > Make any changes necessary in LICENSE and NOTICE related to dependencies > introduced by the application timeline store. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1410) Handle RM fails over after getApplicationID() and before submitApplication().
[ https://issues.apache.org/jira/browse/YARN-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914995#comment-13914995 ] Vinod Kumar Vavilapalli commented on YARN-1410: --- h4. Questions When can appId be null in the submission-context? h4. Documentation Though it isn't an incompatible change, it needs extra coding on client side to handle fail-over. Let's make sure we document that clearly. Can we document the appID related methods in SubmitApplicationResponse.java, GetNewApplicationResponse.java, ApplicationClientProtocol.getNewApplication(..) API and ApplicationClientProtocol.submitApplication(..) API to clearly indicate what clients need to do when we return a new appID. h4. Other changes needed Does DistributedShell need any changes to reflect the potential change in appId after fail-over? If so, let's fix that too here. Please file a MR ticket to fix MapReduce too if needed. Fixes are needed in either case if anyone caches the appId from GetNewApplicationResponse. h4. Beyond this JIRA Orthogonal to this ticket, we need to make sure clients don't pass in invalid application-IDs as part of the submission-context. It can be validated by simply looking at our counter and may be also caching recently used appIDs (atleast within a single RM). I remember we had a JIRA for this somewhere. We also need throttles so that malicious client don't exhaust appIDs. > Handle RM fails over after getApplicationID() and before submitApplication(). > - > > Key: YARN-1410 > URL: https://issues.apache.org/jira/browse/YARN-1410 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Xuan Gong > Attachments: YARN-1410-outline.patch, YARN-1410.1.patch, > YARN-1410.2.patch, YARN-1410.2.patch, YARN-1410.3.patch, YARN-1410.4.patch, > YARN-1410.5.patch, YARN-1410.6.patch, YARN-1410.7.patch, YARN-1410.8.patch, > YARN-1410.9.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > App submission involves > 1) creating appId > 2) using that appId to submit an ApplicationSubmissionContext to the user. > The client may have obtained an appId from an RM, the RM may have failed > over, and the client may submit the app to the new RM. > Since the new RM has a different notion of cluster timestamp (used to create > app id) the new RM may reject the app submission resulting in unexpected > failure on the client side. > The same may happen for other 2 step client API operations. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1729) TimelineWebServices always passes primary and secondary filters as strings
[ https://issues.apache.org/jira/browse/YARN-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billie Rinaldi updated YARN-1729: - Attachment: YARN-1729.4.patch Added testing and fixed bug that [~zshen] noticed in the numeric integer/long edge case. > TimelineWebServices always passes primary and secondary filters as strings > -- > > Key: YARN-1729 > URL: https://issues.apache.org/jira/browse/YARN-1729 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Attachments: YARN-1729.1.patch, YARN-1729.2.patch, YARN-1729.3.patch, > YARN-1729.4.patch > > > Primary filters and secondary filter values can be arbitrary json-compatible > Object. The web services should determine if the filters specified as query > parameters are objects or strings before passing them to the store. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1752) Unexpected Unregistered event at Attempt Launched state
[
https://issues.apache.org/jira/browse/YARN-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914961#comment-13914961
]
Jian He commented on YARN-1752:
---
Took a quick look at the patch, we already have a method in
ApplicationMasterService.hasApplicationMasterRegistered(), we can use that to
do the check?
> Unexpected Unregistered event at Attempt Launched state
> ---
>
> Key: YARN-1752
> URL: https://issues.apache.org/jira/browse/YARN-1752
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jian He
>Assignee: Rohith
> Attachments: YARN-1752.1.patch, YARN-1752.2.patch
>
>
> {code}
> 2014-02-21 14:56:03,453 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> UNREGISTERED at LAUNCHED
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:647)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:103)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:733)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:714)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
> at java.lang.Thread.run(Thread.java:695)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1740) Redirection from AM-URL is broken with HTTPS_ONLY policy
[ https://issues.apache.org/jira/browse/YARN-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1740: -- Issue Type: Bug (was: Sub-task) Parent: (was: YARN-1280) > Redirection from AM-URL is broken with HTTPS_ONLY policy > > > Key: YARN-1740 > URL: https://issues.apache.org/jira/browse/YARN-1740 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Yesha Vora >Assignee: Jian He > Attachments: YARN-1740.1.patch, YARN-1740.2.patch, YARN-1740.3.patch, > YARN-1740.4.patch > > > Steps to reproduce: > 1) Run a sleep job > 2) Run: yarn application -list command to find AM URL. > root@host1:~# yarn application -list > Total number of applications (application-types: [] and states: SUBMITTED, > ACCEPTED, RUNNING):1 > Application-Id Application-Name Application-Type User Queue State Final-State > Progress Tracking-URL > application_1383251398986_0003 Sleep job MAPREDUCE hdfs default RUNNING > UNDEFINED 5% http://host1:40653 > 3) Try to access "http://host1:40653/ws/v1/mapreduce/info"; url. > This URL redirects to > http://RM_host:RM_https_port/proxy/application_1383251398986_0003/ws/v1/mapreduce/info > Here, Http protocol is used with HTTPS port for RM. > The expected Url is > https://RM_host:RM_https_port/proxy/application_1383251398986_0003/ws/v1/mapreduce/info -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (YARN-1528) ZKRMStateStore and EmbeddedElector should allow setting ZK auth information
[ https://issues.apache.org/jira/browse/YARN-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla reassigned YARN-1528: -- Assignee: Karthik Kambatla > ZKRMStateStore and EmbeddedElector should allow setting ZK auth information > --- > > Key: YARN-1528 > URL: https://issues.apache.org/jira/browse/YARN-1528 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.3.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla >Priority: Minor > > ZK store and embedded election allow setting ZK-acls but not auth information -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1528) ZKRMStateStore and EmbeddedElector should allow setting ZK auth information
[ https://issues.apache.org/jira/browse/YARN-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1528: --- Priority: Blocker (was: Minor) Target Version/s: 2.4.0 (was: ) > ZKRMStateStore and EmbeddedElector should allow setting ZK auth information > --- > > Key: YARN-1528 > URL: https://issues.apache.org/jira/browse/YARN-1528 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.3.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla >Priority: Blocker > > ZK store and embedded election allow setting ZK-acls but not auth information -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1740) Redirection from AM-URL is broken with HTTPS_ONLY policy
[ https://issues.apache.org/jira/browse/YARN-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914927#comment-13914927 ] Vinod Kumar Vavilapalli commented on YARN-1740: --- Like the test now :) Exactly validates what needs to be. Looks good. +1. Checking this in. > Redirection from AM-URL is broken with HTTPS_ONLY policy > > > Key: YARN-1740 > URL: https://issues.apache.org/jira/browse/YARN-1740 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Yesha Vora >Assignee: Jian He > Attachments: YARN-1740.1.patch, YARN-1740.2.patch, YARN-1740.3.patch, > YARN-1740.4.patch > > > Steps to reproduce: > 1) Run a sleep job > 2) Run: yarn application -list command to find AM URL. > root@host1:~# yarn application -list > Total number of applications (application-types: [] and states: SUBMITTED, > ACCEPTED, RUNNING):1 > Application-Id Application-Name Application-Type User Queue State Final-State > Progress Tracking-URL > application_1383251398986_0003 Sleep job MAPREDUCE hdfs default RUNNING > UNDEFINED 5% http://host1:40653 > 3) Try to access "http://host1:40653/ws/v1/mapreduce/info"; url. > This URL redirects to > http://RM_host:RM_https_port/proxy/application_1383251398986_0003/ws/v1/mapreduce/info > Here, Http protocol is used with HTTPS port for RM. > The expected Url is > https://RM_host:RM_https_port/proxy/application_1383251398986_0003/ws/v1/mapreduce/info -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1410) Handle RM fails over after getApplicationID() and before submitApplication().
[
https://issues.apache.org/jira/browse/YARN-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914909#comment-13914909
]
Hadoop QA commented on YARN-1410:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12631569/YARN-1410.9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3209//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3209//console
This message is automatically generated.
> Handle RM fails over after getApplicationID() and before submitApplication().
> -
>
> Key: YARN-1410
> URL: https://issues.apache.org/jira/browse/YARN-1410
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bikas Saha
>Assignee: Xuan Gong
> Attachments: YARN-1410-outline.patch, YARN-1410.1.patch,
> YARN-1410.2.patch, YARN-1410.2.patch, YARN-1410.3.patch, YARN-1410.4.patch,
> YARN-1410.5.patch, YARN-1410.6.patch, YARN-1410.7.patch, YARN-1410.8.patch,
> YARN-1410.9.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> App submission involves
> 1) creating appId
> 2) using that appId to submit an ApplicationSubmissionContext to the user.
> The client may have obtained an appId from an RM, the RM may have failed
> over, and the client may submit the app to the new RM.
> Since the new RM has a different notion of cluster timestamp (used to create
> app id) the new RM may reject the app submission resulting in unexpected
> failure on the client side.
> The same may happen for other 2 step client API operations.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1363) Get / Cancel / Renew delegation token api should be non blocking
[
https://issues.apache.org/jira/browse/YARN-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914871#comment-13914871
]
Hadoop QA commented on YARN-1363:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12631556/YARN-1363.10.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.mapreduce.v2.TestUberAM
The following test timeouts occurred in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.client.api.impl.TestNMClient
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3208//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3208//console
This message is automatically generated.
> Get / Cancel / Renew delegation token api should be non blocking
>
>
> Key: YARN-1363
> URL: https://issues.apache.org/jira/browse/YARN-1363
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Omkar Vinit Joshi
>Assignee: Zhijie Shen
> Attachments: YARN-1363.1.patch, YARN-1363.10.patch,
> YARN-1363.10.patch, YARN-1363.2.patch, YARN-1363.3.patch, YARN-1363.4.patch,
> YARN-1363.5.patch, YARN-1363.6.patch, YARN-1363.7.patch, YARN-1363.8.patch,
> YARN-1363.9.patch
>
>
> Today GetDelgationToken, CancelDelegationToken and RenewDelegationToken are
> all blocking apis.
> * As a part of these calls we try to update RMStateStore and that may slow it
> down.
> * Now as we have limited number of client request handlers we may fill up
> client handlers quickly.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1577) Unmanaged AM is broken because of YARN-1493
[ https://issues.apache.org/jira/browse/YARN-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914853#comment-13914853 ] Naren Koneru commented on YARN-1577: Vinod, Jian.. thanks for the inputs.. I will look into this further and submit a patch > Unmanaged AM is broken because of YARN-1493 > --- > > Key: YARN-1577 > URL: https://issues.apache.org/jira/browse/YARN-1577 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.3.0 >Reporter: Jian He >Assignee: Naren Koneru >Priority: Blocker > > Today unmanaged AM client is waiting for app state to be Accepted to launch > the AM. This is broken since we changed in YARN-1493 to start the attempt > after the application is Accepted. We may need to introduce an attempt state > report that client can rely on to query the attempt state and choose to > launch the unmanaged AM. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1729) TimelineWebServices always passes primary and secondary filters as strings
[
https://issues.apache.org/jira/browse/YARN-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914834#comment-13914834
]
Billie Rinaldi commented on YARN-1729:
--
I don't understand what you mean. Are you saying that we should test a long in
the integer range (which is done in the following line), or that you're
wondering if this is a valid test?
{code}
assertEquals(42, GenericObjectMapper.read(GenericObjectMapper.write(42l)))
{code}
> TimelineWebServices always passes primary and secondary filters as strings
> --
>
> Key: YARN-1729
> URL: https://issues.apache.org/jira/browse/YARN-1729
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
> Attachments: YARN-1729.1.patch, YARN-1729.2.patch, YARN-1729.3.patch
>
>
> Primary filters and secondary filter values can be arbitrary json-compatible
> Object. The web services should determine if the filters specified as query
> parameters are objects or strings before passing them to the store.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1410) Handle RM fails over after getApplicationID() and before submitApplication().
[ https://issues.apache.org/jira/browse/YARN-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914820#comment-13914820 ] Xuan Gong commented on YARN-1410: - bq. In testAppSubmissionWithoutApplicationId() can you please do an explicit failover and submit the application again. That submission should also pass and the RM should return a new appId different from the first submission. DONE > Handle RM fails over after getApplicationID() and before submitApplication(). > - > > Key: YARN-1410 > URL: https://issues.apache.org/jira/browse/YARN-1410 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Xuan Gong > Attachments: YARN-1410-outline.patch, YARN-1410.1.patch, > YARN-1410.2.patch, YARN-1410.2.patch, YARN-1410.3.patch, YARN-1410.4.patch, > YARN-1410.5.patch, YARN-1410.6.patch, YARN-1410.7.patch, YARN-1410.8.patch, > YARN-1410.9.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > App submission involves > 1) creating appId > 2) using that appId to submit an ApplicationSubmissionContext to the user. > The client may have obtained an appId from an RM, the RM may have failed > over, and the client may submit the app to the new RM. > Since the new RM has a different notion of cluster timestamp (used to create > app id) the new RM may reject the app submission resulting in unexpected > failure on the client side. > The same may happen for other 2 step client API operations. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1410) Handle RM fails over after getApplicationID() and before submitApplication().
[ https://issues.apache.org/jira/browse/YARN-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1410: Attachment: YARN-1410.9.patch > Handle RM fails over after getApplicationID() and before submitApplication(). > - > > Key: YARN-1410 > URL: https://issues.apache.org/jira/browse/YARN-1410 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Xuan Gong > Attachments: YARN-1410-outline.patch, YARN-1410.1.patch, > YARN-1410.2.patch, YARN-1410.2.patch, YARN-1410.3.patch, YARN-1410.4.patch, > YARN-1410.5.patch, YARN-1410.6.patch, YARN-1410.7.patch, YARN-1410.8.patch, > YARN-1410.9.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > App submission involves > 1) creating appId > 2) using that appId to submit an ApplicationSubmissionContext to the user. > The client may have obtained an appId from an RM, the RM may have failed > over, and the client may submit the app to the new RM. > Since the new RM has a different notion of cluster timestamp (used to create > app id) the new RM may reject the app submission resulting in unexpected > failure on the client side. > The same may happen for other 2 step client API operations. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1729) TimelineWebServices always passes primary and secondary filters as strings
[ https://issues.apache.org/jira/browse/YARN-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-1729: -- Summary: TimelineWebServices always passes primary and secondary filters as strings (was: ATSWebServices always passes primary and secondary filters as strings) > TimelineWebServices always passes primary and secondary filters as strings > -- > > Key: YARN-1729 > URL: https://issues.apache.org/jira/browse/YARN-1729 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Attachments: YARN-1729.1.patch, YARN-1729.2.patch, YARN-1729.3.patch > > > Primary filters and secondary filter values can be arbitrary json-compatible > Object. The web services should determine if the filters specified as query > parameters are objects or strings before passing them to the store. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1729) ATSWebServices always passes primary and secondary filters as strings
[ https://issues.apache.org/jira/browse/YARN-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914804#comment-13914804 ] Zhijie Shen commented on YARN-1729: --- In the test case, is it good to test scenario of giving a long object whose value is in the integer range? > ATSWebServices always passes primary and secondary filters as strings > - > > Key: YARN-1729 > URL: https://issues.apache.org/jira/browse/YARN-1729 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Attachments: YARN-1729.1.patch, YARN-1729.2.patch, YARN-1729.3.patch > > > Primary filters and secondary filter values can be arbitrary json-compatible > Object. The web services should determine if the filters specified as query > parameters are objects or strings before passing them to the store. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1410) Handle RM fails over after getApplicationID() and before submitApplication().
[ https://issues.apache.org/jira/browse/YARN-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914801#comment-13914801 ] Bikas Saha commented on YARN-1410: -- Looks good overall. In testAppSubmissionWithoutApplicationId() can you please do an explicit failover and submit the application again. That submission should also pass and the RM should return a new appId different from the first submission. [~vinodkv] are you fine with the new API changes. They should be backwards compatible. > Handle RM fails over after getApplicationID() and before submitApplication(). > - > > Key: YARN-1410 > URL: https://issues.apache.org/jira/browse/YARN-1410 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Xuan Gong > Attachments: YARN-1410-outline.patch, YARN-1410.1.patch, > YARN-1410.2.patch, YARN-1410.2.patch, YARN-1410.3.patch, YARN-1410.4.patch, > YARN-1410.5.patch, YARN-1410.6.patch, YARN-1410.7.patch, YARN-1410.8.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > App submission involves > 1) creating appId > 2) using that appId to submit an ApplicationSubmissionContext to the user. > The client may have obtained an appId from an RM, the RM may have failed > over, and the client may submit the app to the new RM. > Since the new RM has a different notion of cluster timestamp (used to create > app id) the new RM may reject the app submission resulting in unexpected > failure on the client side. > The same may happen for other 2 step client API operations. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-986) YARN should use cluster-id as token service address
[ https://issues.apache.org/jira/browse/YARN-986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914796#comment-13914796 ] Tsuyoshi OZAWA commented on YARN-986: - The test failure is filed as MAPREDUCE-5768 by [~zjshen]. > YARN should use cluster-id as token service address > --- > > Key: YARN-986 > URL: https://issues.apache.org/jira/browse/YARN-986 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Karthik Kambatla >Priority: Blocker > Attachments: yarn-986-1.patch, yarn-986-2.patch, > yarn-986-prelim-0.patch > > > This needs to be done to support non-ip based fail over of RM. Once the > server sets the token service address to be this generic ClusterId/ServiceId, > clients can translate it to appropriate final IP and then be able to select > tokens via TokenSelectors. > Some workarounds for other related issues were put in place at YARN-945. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-986) YARN should use cluster-id as token service address
[ https://issues.apache.org/jira/browse/YARN-986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914752#comment-13914752 ] Karthik Kambatla commented on YARN-986: --- Again, the test failures are unrelated. [~vinodkv] - can you take a look at the updated version please. > YARN should use cluster-id as token service address > --- > > Key: YARN-986 > URL: https://issues.apache.org/jira/browse/YARN-986 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Karthik Kambatla >Priority: Blocker > Attachments: yarn-986-1.patch, yarn-986-2.patch, > yarn-986-prelim-0.patch > > > This needs to be done to support non-ip based fail over of RM. Once the > server sets the token service address to be this generic ClusterId/ServiceId, > clients can translate it to appropriate final IP and then be able to select > tokens via TokenSelectors. > Some workarounds for other related issues were put in place at YARN-945. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1730) Leveldb timeline store needs simple write locking
[
https://issues.apache.org/jira/browse/YARN-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914729#comment-13914729
]
Hadoop QA commented on YARN-1730:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12631555/YARN-1730.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3207//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3207//console
This message is automatically generated.
> Leveldb timeline store needs simple write locking
> -
>
> Key: YARN-1730
> URL: https://issues.apache.org/jira/browse/YARN-1730
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
> Attachments: YARN-1730.1.patch, YARN-1730.2.patch, YARN-1730.3.patch,
> YARN-1730.4.patch, YARN-1730.5.patch, YARN-1730.6.patch
>
>
> The actual data writes are performed atomically in a batch, but a lock should
> be held while identifying a start time for the entity, which precedes every
> write.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1363) Get / Cancel / Renew delegation token api should be non blocking
[ https://issues.apache.org/jira/browse/YARN-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-1363: -- Attachment: YARN-1363.10.patch Kick the jenkins again > Get / Cancel / Renew delegation token api should be non blocking > > > Key: YARN-1363 > URL: https://issues.apache.org/jira/browse/YARN-1363 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Zhijie Shen > Attachments: YARN-1363.1.patch, YARN-1363.10.patch, > YARN-1363.10.patch, YARN-1363.2.patch, YARN-1363.3.patch, YARN-1363.4.patch, > YARN-1363.5.patch, YARN-1363.6.patch, YARN-1363.7.patch, YARN-1363.8.patch, > YARN-1363.9.patch > > > Today GetDelgationToken, CancelDelegationToken and RenewDelegationToken are > all blocking apis. > * As a part of these calls we try to update RMStateStore and that may slow it > down. > * Now as we have limited number of client request handlers we may fill up > client handlers quickly. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1730) Leveldb timeline store needs simple write locking
[ https://issues.apache.org/jira/browse/YARN-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billie Rinaldi updated YARN-1730: - Attachment: YARN-1730.6.patch Attaching a new patch with variables and methods renamed as suggested, and with the write locking moved to the put method. I think expanding the lock is the right thing to do. I didn't move the default cache sizes to YarnConfiguration yet, so let me know if you still want me to do that. > Leveldb timeline store needs simple write locking > - > > Key: YARN-1730 > URL: https://issues.apache.org/jira/browse/YARN-1730 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Attachments: YARN-1730.1.patch, YARN-1730.2.patch, YARN-1730.3.patch, > YARN-1730.4.patch, YARN-1730.5.patch, YARN-1730.6.patch > > > The actual data writes are performed atomically in a batch, but a lock should > be held while identifying a start time for the entity, which precedes every > write. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1730) Leveldb timeline store needs simple write locking
[ https://issues.apache.org/jira/browse/YARN-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914665#comment-13914665 ] Billie Rinaldi commented on YARN-1730: -- bq. Why do we need separate start-time caches for read and write calls? The write cache is essential for having a good write throughput, so you don't have to hit disk to do a lookup each time you do a write. The size of the write cache should be the maximum number of active entities (entities that are still receiving writes). This may vary, so [~zjshen] suggested making it configurable. The read cache is just there to improve read performance, so you don't have to hit disk twice per entity. This cache would have the most recently queried entities instead of the most recently written entities. I add things to the read cache and the write cache when writing because more recently written things are also generally more likely to be queried. The useful size of this cache can be as big as you want. bq. Shouldn't we do the write-locking at the put(..) API level itself and not just creating the start-time? Or at-least when the actual write happens for a given entity? It's a good question; we could decide to do this. Locking when determining the start time is essential so that two writes for the same entity can't come up with different start times. The writes to leveldb in a put are atomic, so that part isn't an issue. The question is whether we care about the following: writes 1 and 2 come in, write 1 sets the start time for an entity, 2 uses that start time, but 2's put completes before 1's. > Leveldb timeline store needs simple write locking > - > > Key: YARN-1730 > URL: https://issues.apache.org/jira/browse/YARN-1730 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Attachments: YARN-1730.1.patch, YARN-1730.2.patch, YARN-1730.3.patch, > YARN-1730.4.patch, YARN-1730.5.patch > > > The actual data writes are performed atomically in a batch, but a lock should > be held while identifying a start time for the entity, which precedes every > write. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1301) Need to log the blacklist additions/removals when YarnSchedule#allocate
[ https://issues.apache.org/jira/browse/YARN-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914599#comment-13914599 ] Hudson commented on YARN-1301: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1711 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1711/]) YARN-1301. Added the INFO level log of the non-empty blacklist additions and removals inside ApplicationMasterService. Contributed by Tsuyoshi Ozawa. (zjshen: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1572400) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java > Need to log the blacklist additions/removals when YarnSchedule#allocate > --- > > Key: YARN-1301 > URL: https://issues.apache.org/jira/browse/YARN-1301 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Tsuyoshi OZAWA >Priority: Minor > Fix For: 2.4.0 > > Attachments: YARN-1301.1.patch, YARN-1301.2.patch, YARN-1301.3.patch, > YARN-1301.4.patch, YARN-1301.5.patch, YARN-1301.6.patch > > > Now without the log, it's hard to debug whether blacklist is updated on the > scheduler side or not -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1588) Rebind NM tokens for previous attempt's running containers to the new attempt
[ https://issues.apache.org/jira/browse/YARN-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914594#comment-13914594 ] Hudson commented on YARN-1588: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1711 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1711/]) YARN-1588. Enhanced RM and the scheduling protocol to also send NMTokens of transferred containers from previous app-attempts to new AMs after YARN-1490. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1572230) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/RegisterApplicationMasterResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NMToken.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/RegisterApplicationMasterResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/NMTokenPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java > Rebind NM tokens for previous attempt's running containers to the new attempt > - > > Key: YARN-1588 > URL: https://issues.apache.org/jira/browse/YARN-1588 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Fix For: 2.4.0 > > Attachments: YARN-1588.1.patch, YARN-1588.1.patch, YARN-1588.2.patch, > YARN-1588.3.patch, YARN-1588.4.patch, YARN-1588.5.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1429) *nix: Allow a way for users to augment classpath of YARN daemons
[ https://issues.apache.org/jira/browse/YARN-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914596#comment-13914596 ] Hudson commented on YARN-1429: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1711 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1711/]) YARN-1429. *nix: Allow a way for users to augment classpath of YARN daemons. (Jarek Jarcec Cecho via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1572405) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/bin/yarn > *nix: Allow a way for users to augment classpath of YARN daemons > > > Key: YARN-1429 > URL: https://issues.apache.org/jira/browse/YARN-1429 > Project: Hadoop YARN > Issue Type: Bug > Components: client >Reporter: Sandy Ryza >Assignee: Jarek Jarcec Cecho >Priority: Trivial > Labels: newbie > Fix For: 2.5.0 > > Attachments: YARN-1429.linux.patch, YARN-1429.patch, YARN-1429.patch > > > YARN_CLASSPATH is referenced in the comments in > ./hadoop-yarn-project/hadoop-yarn/bin/yarn and > ./hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd, but doesn't do anything. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1490) RM should optionally not kill all containers when an ApplicationMaster exits
[ https://issues.apache.org/jira/browse/YARN-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914595#comment-13914595 ] Hudson commented on YARN-1490: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1711 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1711/]) YARN-1588. Enhanced RM and the scheduling protocol to also send NMTokens of transferred containers from previous app-attempts to new AMs after YARN-1490. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1572230) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/RegisterApplicationMasterResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NMToken.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/RegisterApplicationMasterResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/NMTokenPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java > RM should optionally not kill all containers when an ApplicationMaster exits > > > Key: YARN-1490 > URL: https://issues.apache.org/jira/browse/YARN-1490 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Jian He > Fix For: 2.4.0 > > Attachments: YARN-1490.1.patch, YARN-1490.10.patch, > YARN-1490.11.patch, YARN-1490.11.patch, YARN-1490.12.patch, > YARN-1490.2.patch, YARN-1490.3.patch, YARN-1490.4.patch, YARN-1490.5.patch, > YARN-1490.6.patch, YARN-1490.7.patch, YARN-1490.8.patch, YARN-1490.9.patch, > org.apache.oozie.service.TestRecoveryService_thread-dump.txt > > > This is needed to enable work-preserving AM restart. Some apps can chose to > reconnect with old running containers, some may not want to. This should be > an option. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1599) webUI rm.webapp.AppBlock should redirect to a history App page if and when available
[ https://issues.apache.org/jira/browse/YARN-1599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914550#comment-13914550 ] Jason Lowe commented on YARN-1599: -- I'd rather not change the meaning of the exisiting URLs. If the user wants to see the app-specific UI if there is one, that's the purpose of the proxy URL provided by the job client when the job launches. If the user wants to see the RM's details on the application then that's the purpose of the RM app URL. If the issue is users tend to click on the left link instead of the right link when they want to go to the app-framework-specific UI for an app then I'd rather discuss swapping the links in the application table rather than change what URLs go where by default. bq. However, you are correct that 404 on the link from RM app to AM logs should be fixed as well. 404 is caused by the log aggregation after the local logs are deleted. I'm confused about the 404 scenario. If log aggregation is configured and working properly, the NM should redirect to the aggregated logs server when the logs are not present, and the logs are not deleted until log aggregation completes. The user would briefly see "Redirecting to log server for container_xxx" during the redirect. Is yarn.log.server.url configured properly? > webUI rm.webapp.AppBlock should redirect to a history App page if and when > available > > > Key: YARN-1599 > URL: https://issues.apache.org/jira/browse/YARN-1599 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.0.5-alpha, 2.2.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Attachments: Screen Shot 2014-01-16 at 6.52.17 PM.png, Screen Shot > 2014-01-16 at 7.30.32 PM.png, YARN-1599.v01.patch, YARN-1599.v02.patch, > YARN-1599.v03.patch > > > When the log aggregation is enabled, and the application finishes, our users > think that the AppMaster logs were lost because the link to the AM attempt > logs are not updated and result in HTTP 404. Only tracking URL is updated. In > order to have a smoother user experience, we propose to simply redirect to > the new tracking URL when the page with invalid log links is accessed. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1490) RM should optionally not kill all containers when an ApplicationMaster exits
[ https://issues.apache.org/jira/browse/YARN-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914516#comment-13914516 ] Hudson commented on YARN-1490: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1686 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1686/]) YARN-1588. Enhanced RM and the scheduling protocol to also send NMTokens of transferred containers from previous app-attempts to new AMs after YARN-1490. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1572230) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/RegisterApplicationMasterResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NMToken.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/RegisterApplicationMasterResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/NMTokenPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java > RM should optionally not kill all containers when an ApplicationMaster exits > > > Key: YARN-1490 > URL: https://issues.apache.org/jira/browse/YARN-1490 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Jian He > Fix For: 2.4.0 > > Attachments: YARN-1490.1.patch, YARN-1490.10.patch, > YARN-1490.11.patch, YARN-1490.11.patch, YARN-1490.12.patch, > YARN-1490.2.patch, YARN-1490.3.patch, YARN-1490.4.patch, YARN-1490.5.patch, > YARN-1490.6.patch, YARN-1490.7.patch, YARN-1490.8.patch, YARN-1490.9.patch, > org.apache.oozie.service.TestRecoveryService_thread-dump.txt > > > This is needed to enable work-preserving AM restart. Some apps can chose to > reconnect with old running containers, some may not want to. This should be > an option. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1301) Need to log the blacklist additions/removals when YarnSchedule#allocate
[ https://issues.apache.org/jira/browse/YARN-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914520#comment-13914520 ] Hudson commented on YARN-1301: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1686 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1686/]) YARN-1301. Added the INFO level log of the non-empty blacklist additions and removals inside ApplicationMasterService. Contributed by Tsuyoshi Ozawa. (zjshen: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1572400) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java > Need to log the blacklist additions/removals when YarnSchedule#allocate > --- > > Key: YARN-1301 > URL: https://issues.apache.org/jira/browse/YARN-1301 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Tsuyoshi OZAWA >Priority: Minor > Fix For: 2.4.0 > > Attachments: YARN-1301.1.patch, YARN-1301.2.patch, YARN-1301.3.patch, > YARN-1301.4.patch, YARN-1301.5.patch, YARN-1301.6.patch > > > Now without the log, it's hard to debug whether blacklist is updated on the > scheduler side or not -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1588) Rebind NM tokens for previous attempt's running containers to the new attempt
[ https://issues.apache.org/jira/browse/YARN-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914515#comment-13914515 ] Hudson commented on YARN-1588: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1686 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1686/]) YARN-1588. Enhanced RM and the scheduling protocol to also send NMTokens of transferred containers from previous app-attempts to new AMs after YARN-1490. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1572230) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/RegisterApplicationMasterResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NMToken.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/RegisterApplicationMasterResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/NMTokenPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java > Rebind NM tokens for previous attempt's running containers to the new attempt > - > > Key: YARN-1588 > URL: https://issues.apache.org/jira/browse/YARN-1588 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Fix For: 2.4.0 > > Attachments: YARN-1588.1.patch, YARN-1588.1.patch, YARN-1588.2.patch, > YARN-1588.3.patch, YARN-1588.4.patch, YARN-1588.5.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1429) *nix: Allow a way for users to augment classpath of YARN daemons
[ https://issues.apache.org/jira/browse/YARN-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914517#comment-13914517 ] Hudson commented on YARN-1429: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1686 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1686/]) YARN-1429. *nix: Allow a way for users to augment classpath of YARN daemons. (Jarek Jarcec Cecho via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1572405) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/bin/yarn > *nix: Allow a way for users to augment classpath of YARN daemons > > > Key: YARN-1429 > URL: https://issues.apache.org/jira/browse/YARN-1429 > Project: Hadoop YARN > Issue Type: Bug > Components: client >Reporter: Sandy Ryza >Assignee: Jarek Jarcec Cecho >Priority: Trivial > Labels: newbie > Fix For: 2.5.0 > > Attachments: YARN-1429.linux.patch, YARN-1429.patch, YARN-1429.patch > > > YARN_CLASSPATH is referenced in the comments in > ./hadoop-yarn-project/hadoop-yarn/bin/yarn and > ./hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd, but doesn't do anything. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1713) Implement getnewapplication and submitapp as part of RM web service
[
https://issues.apache.org/jira/browse/YARN-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914467#comment-13914467
]
Hadoop QA commented on YARN-1713:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12631510/apache-yarn-1713.cumulative.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3206//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3206//console
This message is automatically generated.
> Implement getnewapplication and submitapp as part of RM web service
> ---
>
> Key: YARN-1713
> URL: https://issues.apache.org/jira/browse/YARN-1713
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: apache-yarn-1713.3.patch, apache-yarn-1713.4.patch,
> apache-yarn-1713.5.patch, apache-yarn-1713.cumulative.2.patch,
> apache-yarn-1713.cumulative.3.patch, apache-yarn-1713.cumulative.4.patch,
> apache-yarn-1713.cumulative.patch, apache-yarn-1713.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1702) Expose kill app functionality as part of RM web services
[ https://issues.apache.org/jira/browse/YARN-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-1702: Attachment: apache-yarn-1702.8.patch > Expose kill app functionality as part of RM web services > > > Key: YARN-1702 > URL: https://issues.apache.org/jira/browse/YARN-1702 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: apache-yarn-1702.2.patch, apache-yarn-1702.3.patch, > apache-yarn-1702.4.patch, apache-yarn-1702.5.patch, apache-yarn-1702.7.patch, > apache-yarn-1702.8.patch > > > Expose functionality to kill an app via the ResourceManager web services API. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1713) Implement getnewapplication and submitapp as part of RM web service
[ https://issues.apache.org/jira/browse/YARN-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914437#comment-13914437 ] Varun Vasudev commented on YARN-1713: - > The following piece can be incorporated into the above > ApplicationSubmissionContext.newInstance( appContext.setKeepContainersAcrossApplicationAttempts( newApp.isKeepContainers()); Fixed > hasQueueAccess is not used anywhere? I think we can just use hasAccess() in > all the places and parameterize ApplicationAccessType and > ApplicationAccessType? hasAccess returns true only if the user has app and queue access. For killing an app, we only need app access. Maybe we can change hasAcess to use hasAppAccess and hasQueueAccess instead? > Rename AppSubmissionInfo to AppSubmissionContextInfo ?, similarly > name="appsubmission" -> “appsubmissioncontext”. Similarly, > ContainerLaunchInfo -> ContainerLaunchContextInfo Fixed > AppSubmissionInfo: we can initialize cancelTokensWhenComplete to true, > keepContainers to false Fixed > Also , we can replace the "Location" with constant HttpHeaders.LOCATION Fixed > Implement getnewapplication and submitapp as part of RM web service > --- > > Key: YARN-1713 > URL: https://issues.apache.org/jira/browse/YARN-1713 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: apache-yarn-1713.3.patch, apache-yarn-1713.4.patch, > apache-yarn-1713.5.patch, apache-yarn-1713.cumulative.2.patch, > apache-yarn-1713.cumulative.3.patch, apache-yarn-1713.cumulative.4.patch, > apache-yarn-1713.cumulative.patch, apache-yarn-1713.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1713) Implement getnewapplication and submitapp as part of RM web service
[ https://issues.apache.org/jira/browse/YARN-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-1713: Attachment: apache-yarn-1713.cumulative.4.patch > Implement getnewapplication and submitapp as part of RM web service > --- > > Key: YARN-1713 > URL: https://issues.apache.org/jira/browse/YARN-1713 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: apache-yarn-1713.3.patch, apache-yarn-1713.4.patch, > apache-yarn-1713.5.patch, apache-yarn-1713.cumulative.2.patch, > apache-yarn-1713.cumulative.3.patch, apache-yarn-1713.cumulative.4.patch, > apache-yarn-1713.cumulative.patch, apache-yarn-1713.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1713) Implement getnewapplication and submitapp as part of RM web service
[ https://issues.apache.org/jira/browse/YARN-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-1713: Attachment: apache-yarn-1713.5.patch > Implement getnewapplication and submitapp as part of RM web service > --- > > Key: YARN-1713 > URL: https://issues.apache.org/jira/browse/YARN-1713 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: apache-yarn-1713.3.patch, apache-yarn-1713.4.patch, > apache-yarn-1713.5.patch, apache-yarn-1713.cumulative.2.patch, > apache-yarn-1713.cumulative.3.patch, apache-yarn-1713.cumulative.4.patch, > apache-yarn-1713.cumulative.patch, apache-yarn-1713.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1588) Rebind NM tokens for previous attempt's running containers to the new attempt
[ https://issues.apache.org/jira/browse/YARN-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914381#comment-13914381 ] Hudson commented on YARN-1588: -- FAILURE: Integrated in Hadoop-Yarn-trunk #494 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/494/]) YARN-1588. Enhanced RM and the scheduling protocol to also send NMTokens of transferred containers from previous app-attempts to new AMs after YARN-1490. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1572230) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/RegisterApplicationMasterResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NMToken.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/RegisterApplicationMasterResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/NMTokenPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java > Rebind NM tokens for previous attempt's running containers to the new attempt > - > > Key: YARN-1588 > URL: https://issues.apache.org/jira/browse/YARN-1588 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Fix For: 2.4.0 > > Attachments: YARN-1588.1.patch, YARN-1588.1.patch, YARN-1588.2.patch, > YARN-1588.3.patch, YARN-1588.4.patch, YARN-1588.5.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1301) Need to log the blacklist additions/removals when YarnSchedule#allocate
[ https://issues.apache.org/jira/browse/YARN-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914386#comment-13914386 ] Hudson commented on YARN-1301: -- FAILURE: Integrated in Hadoop-Yarn-trunk #494 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/494/]) YARN-1301. Added the INFO level log of the non-empty blacklist additions and removals inside ApplicationMasterService. Contributed by Tsuyoshi Ozawa. (zjshen: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1572400) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java > Need to log the blacklist additions/removals when YarnSchedule#allocate > --- > > Key: YARN-1301 > URL: https://issues.apache.org/jira/browse/YARN-1301 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Tsuyoshi OZAWA >Priority: Minor > Fix For: 2.4.0 > > Attachments: YARN-1301.1.patch, YARN-1301.2.patch, YARN-1301.3.patch, > YARN-1301.4.patch, YARN-1301.5.patch, YARN-1301.6.patch > > > Now without the log, it's hard to debug whether blacklist is updated on the > scheduler side or not -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1429) *nix: Allow a way for users to augment classpath of YARN daemons
[ https://issues.apache.org/jira/browse/YARN-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914383#comment-13914383 ] Hudson commented on YARN-1429: -- FAILURE: Integrated in Hadoop-Yarn-trunk #494 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/494/]) YARN-1429. *nix: Allow a way for users to augment classpath of YARN daemons. (Jarek Jarcec Cecho via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1572405) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/bin/yarn > *nix: Allow a way for users to augment classpath of YARN daemons > > > Key: YARN-1429 > URL: https://issues.apache.org/jira/browse/YARN-1429 > Project: Hadoop YARN > Issue Type: Bug > Components: client >Reporter: Sandy Ryza >Assignee: Jarek Jarcec Cecho >Priority: Trivial > Labels: newbie > Fix For: 2.5.0 > > Attachments: YARN-1429.linux.patch, YARN-1429.patch, YARN-1429.patch > > > YARN_CLASSPATH is referenced in the comments in > ./hadoop-yarn-project/hadoop-yarn/bin/yarn and > ./hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd, but doesn't do anything. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1490) RM should optionally not kill all containers when an ApplicationMaster exits
[ https://issues.apache.org/jira/browse/YARN-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914382#comment-13914382 ] Hudson commented on YARN-1490: -- FAILURE: Integrated in Hadoop-Yarn-trunk #494 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/494/]) YARN-1588. Enhanced RM and the scheduling protocol to also send NMTokens of transferred containers from previous app-attempts to new AMs after YARN-1490. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1572230) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/RegisterApplicationMasterResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NMToken.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/RegisterApplicationMasterResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/NMTokenPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java > RM should optionally not kill all containers when an ApplicationMaster exits > > > Key: YARN-1490 > URL: https://issues.apache.org/jira/browse/YARN-1490 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Jian He > Fix For: 2.4.0 > > Attachments: YARN-1490.1.patch, YARN-1490.10.patch, > YARN-1490.11.patch, YARN-1490.11.patch, YARN-1490.12.patch, > YARN-1490.2.patch, YARN-1490.3.patch, YARN-1490.4.patch, YARN-1490.5.patch, > YARN-1490.6.patch, YARN-1490.7.patch, YARN-1490.8.patch, YARN-1490.9.patch, > org.apache.oozie.service.TestRecoveryService_thread-dump.txt > > > This is needed to enable work-preserving AM restart. Some apps can chose to > reconnect with old running containers, some may not want to. This should be > an option. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1389) ApplicationClientProtocol and ApplicationHistoryProtocol should expose analog APIs
[
https://issues.apache.org/jira/browse/YARN-1389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914365#comment-13914365
]
Hadoop QA commented on YARN-1389:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12631496/YARN-1389-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3205//console
This message is automatically generated.
> ApplicationClientProtocol and ApplicationHistoryProtocol should expose analog
> APIs
> --
>
> Key: YARN-1389
> URL: https://issues.apache.org/jira/browse/YARN-1389
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Mayank Bansal
>Assignee: Mayank Bansal
> Attachments: YARN-1389-1.patch
>
>
> As we plan to have the APIs in ApplicationHistoryProtocol to expose the
> reports of *finished* application attempts and containers, we should do the
> same for ApplicationClientProtocol, which will return the reports of
> *running* attempts and containers.
> Later on, we can improve YarnClient to direct the query of running instance
> to ApplicationClientProtocol, while that of finished instance to
> ApplicationHistoryProtocol, making it transparent to the users.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1389) ApplicationClientProtocol and ApplicationHistoryProtocol should expose analog APIs
[ https://issues.apache.org/jira/browse/YARN-1389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated YARN-1389: Attachment: YARN-1389-1.patch Attaching patch, Working on test cases. Thanks, Mayank > ApplicationClientProtocol and ApplicationHistoryProtocol should expose analog > APIs > -- > > Key: YARN-1389 > URL: https://issues.apache.org/jira/browse/YARN-1389 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Mayank Bansal >Assignee: Mayank Bansal > Attachments: YARN-1389-1.patch > > > As we plan to have the APIs in ApplicationHistoryProtocol to expose the > reports of *finished* application attempts and containers, we should do the > same for ApplicationClientProtocol, which will return the reports of > *running* attempts and containers. > Later on, we can improve YarnClient to direct the query of running instance > to ApplicationClientProtocol, while that of finished instance to > ApplicationHistoryProtocol, making it transparent to the users. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1599) webUI rm.webapp.AppBlock should redirect to a history App page if and when available
[ https://issues.apache.org/jira/browse/YARN-1599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914329#comment-13914329 ] Gera Shegalov commented on YARN-1599: - [~vinodkv], with v03 we are no longer forcing the redirect. I agree with this point, raised by [~jlowe] as well. You can still access the RM app via the corresponding link, as you can see from the screenshot. I simply moved the tracking UI link that our users really want to see to the 1st column. And now users tend to click it first. bq. Why don't we fix this issue if that is the driving force. I don't expect a 404 if users go through the proxy (which they should!). Can you throw more light on when this can happen? However, you are correct that 404 on the link from RM app to AM logs should be fixed as well. 404 is caused by the log aggregation after the local logs are deleted. > webUI rm.webapp.AppBlock should redirect to a history App page if and when > available > > > Key: YARN-1599 > URL: https://issues.apache.org/jira/browse/YARN-1599 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.0.5-alpha, 2.2.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Attachments: Screen Shot 2014-01-16 at 6.52.17 PM.png, Screen Shot > 2014-01-16 at 7.30.32 PM.png, YARN-1599.v01.patch, YARN-1599.v02.patch, > YARN-1599.v03.patch > > > When the log aggregation is enabled, and the application finishes, our users > think that the AppMaster logs were lost because the link to the AM attempt > logs are not updated and result in HTTP 404. Only tracking URL is updated. In > order to have a smoother user experience, we propose to simply redirect to > the new tracking URL when the page with invalid log links is accessed. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1765) Write test cases to verify that killApplication API works in RM HA
[
https://issues.apache.org/jira/browse/YARN-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914332#comment-13914332
]
Hadoop QA commented on YARN-1765:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12631483/YARN-1765.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3204//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3204//console
This message is automatically generated.
> Write test cases to verify that killApplication API works in RM HA
> --
>
> Key: YARN-1765
> URL: https://issues.apache.org/jira/browse/YARN-1765
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-1765.1.patch, YARN-1765.2.patch, YARN-1765.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1765) Write test cases to verify that killApplication API works in RM HA
[
https://issues.apache.org/jira/browse/YARN-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914312#comment-13914312
]
Hadoop QA commented on YARN-1765:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12631481/YARN-1765.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3203//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3203//console
This message is automatically generated.
> Write test cases to verify that killApplication API works in RM HA
> --
>
> Key: YARN-1765
> URL: https://issues.apache.org/jira/browse/YARN-1765
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-1765.1.patch, YARN-1765.2.patch, YARN-1765.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1765) Write test cases to verify that killApplication API works in RM HA
[ https://issues.apache.org/jira/browse/YARN-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1765: Attachment: YARN-1765.2.patch > Write test cases to verify that killApplication API works in RM HA > -- > > Key: YARN-1765 > URL: https://issues.apache.org/jira/browse/YARN-1765 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-1765.1.patch, YARN-1765.2.patch, YARN-1765.2.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1765) Write test cases to verify that killApplication API works in RM HA
[ https://issues.apache.org/jira/browse/YARN-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914223#comment-13914223 ] Xuan Gong commented on YARN-1765: - bq. No need for RMAppManager.getRMContext(). The context is already available in the constructor, just cache it. done bq. MockRM changes are completely unnecessary, the API you want already exists fixed bq. Move the test to a failover package? We should move all tests related to HA/fail-over to a new package. It's getting increasingly crowded in the base package. Can we do this at separate ticket ? It may need some time to figure out the proper directory to have all HA/fail-over tests bq. Document all the tests in the test-file and add timeouts. done bq. Doesn't like all the tests have the right names. Fix them to be more apt. done bq. In failOver* methods, assert the expected app and app-attempt state before killing the app. Also assert app-state immediately after fail-over succeeds. done bq. Validate the response from killApp() method in all tests. Specifically validate KillApplicationResponse.getIsKillCompleted(). done bq. There is one other important test that we need to write as follow added > Write test cases to verify that killApplication API works in RM HA > -- > > Key: YARN-1765 > URL: https://issues.apache.org/jira/browse/YARN-1765 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-1765.1.patch, YARN-1765.2.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1765) Write test cases to verify that killApplication API works in RM HA
[ https://issues.apache.org/jira/browse/YARN-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1765: Attachment: YARN-1765.2.patch > Write test cases to verify that killApplication API works in RM HA > -- > > Key: YARN-1765 > URL: https://issues.apache.org/jira/browse/YARN-1765 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-1765.1.patch, YARN-1765.2.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1713) Implement getnewapplication and submitapp as part of RM web service
[ https://issues.apache.org/jira/browse/YARN-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914219#comment-13914219 ] Jian He commented on YARN-1713: --- - Also , we can replace the "Location" with constant HttpHeaders.LOCATION > Implement getnewapplication and submitapp as part of RM web service > --- > > Key: YARN-1713 > URL: https://issues.apache.org/jira/browse/YARN-1713 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: apache-yarn-1713.3.patch, apache-yarn-1713.4.patch, > apache-yarn-1713.cumulative.2.patch, apache-yarn-1713.cumulative.3.patch, > apache-yarn-1713.cumulative.patch, apache-yarn-1713.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1740) Redirection from AM-URL is broken with HTTPS_ONLY policy
[
https://issues.apache.org/jira/browse/YARN-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914193#comment-13914193
]
Hadoop QA commented on YARN-1740:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12631473/YARN-1740.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3202//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3202//console
This message is automatically generated.
> Redirection from AM-URL is broken with HTTPS_ONLY policy
>
>
> Key: YARN-1740
> URL: https://issues.apache.org/jira/browse/YARN-1740
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Yesha Vora
>Assignee: Jian He
> Attachments: YARN-1740.1.patch, YARN-1740.2.patch, YARN-1740.3.patch,
> YARN-1740.4.patch
>
>
> Steps to reproduce:
> 1) Run a sleep job
> 2) Run: yarn application -list command to find AM URL.
> root@host1:~# yarn application -list
> Total number of applications (application-types: [] and states: SUBMITTED,
> ACCEPTED, RUNNING):1
> Application-Id Application-Name Application-Type User Queue State Final-State
> Progress Tracking-URL
> application_1383251398986_0003 Sleep job MAPREDUCE hdfs default RUNNING
> UNDEFINED 5% http://host1:40653
> 3) Try to access "http://host1:40653/ws/v1/mapreduce/info"; url.
> This URL redirects to
> http://RM_host:RM_https_port/proxy/application_1383251398986_0003/ws/v1/mapreduce/info
> Here, Http protocol is used with HTTPS port for RM.
> The expected Url is
> https://RM_host:RM_https_port/proxy/application_1383251398986_0003/ws/v1/mapreduce/info
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
