[jira] [Assigned] (YARN-1531) Update yarn command document
[
https://issues.apache.org/jira/browse/YARN-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA reassigned YARN-1531:
---
Assignee: Akira AJISAKA
Update yarn command document
Key: YARN-1531
URL: https://issues.apache.org/jira/browse/YARN-1531
Project: Hadoop YARN
Issue Type: Bug
Components: documentation
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
There are some options which are not written to Yarn Command document.
For example, yarn rmadmin command options are as follows:
{code}
Usage: yarn rmadmin
-refreshQueues
-refreshNodes
-refreshSuperUserGroupsConfiguration
-refreshUserToGroupsMappings
-refreshAdminAcls
-refreshServiceAcl
-getGroups [username]
-help [cmd]
-transitionToActive serviceId
-transitionToStandby serviceId
-failover [--forcefence] [--forceactive] serviceId serviceId
-getServiceState serviceId
-checkHealth serviceId
{code}
But some of the new options such as -getGroups, -transitionToActive, and
-transitionToStandby are not documented.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1409) NonAggregatingLogHandler can throw RejectedExecutionException
[
https://issues.apache.org/jira/browse/YARN-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi OZAWA updated YARN-1409:
-
Attachment: YARN-1409.3.patch
Thank you for the review, [~jlowe]. I apologize for the delay.
1. Added a test case to reproduce this issue
2. Fixed to avoid creating LogDeleterRunnable instances twice.
NonAggregatingLogHandler can throw RejectedExecutionException
-
Key: YARN-1409
URL: https://issues.apache.org/jira/browse/YARN-1409
Project: Hadoop YARN
Issue Type: Bug
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Attachments: YARN-1409.1.patch, YARN-1409.2.patch, YARN-1409.3.patch
This problem is caused by handling APPLICATION_FINISHED events after calling
sched.shotdown() in NonAggregatingLongHandler#serviceStop().
org.apache.hadoop.mapred.TestJobCleanup can fail because of
RejectedExecutionException by NonAggregatingLogHandler.
{code}
2013-11-13 10:53:06,970 FATAL [AsyncDispatcher event handler]
event.AsyncDispatcher (AsyncDispatcher.java:dispatch(166)) - Error in
dispatcher thread
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@d51df63
rejected from
java.util.concurrent.ScheduledThreadPoolExecutor@7a20e369[Shutting down, pool
size = 4, active threads = 0, queued tasks = 7, completed tasks = 0]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:325)
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:530)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler.handle(NonAggregatingLogHandler.java:121)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler.handle(NonAggregatingLogHandler.java:49)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:159)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:95)
at java.lang.Thread.run(Thread.java:724)
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1409) NonAggregatingLogHandler can throw RejectedExecutionException
[
https://issues.apache.org/jira/browse/YARN-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13856816#comment-13856816
]
Hadoop QA commented on YARN-1409:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12620481/YARN-1409.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2729//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2729//console
This message is automatically generated.
NonAggregatingLogHandler can throw RejectedExecutionException
-
Key: YARN-1409
URL: https://issues.apache.org/jira/browse/YARN-1409
Project: Hadoop YARN
Issue Type: Bug
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Attachments: YARN-1409.1.patch, YARN-1409.2.patch, YARN-1409.3.patch
This problem is caused by handling APPLICATION_FINISHED events after calling
sched.shotdown() in NonAggregatingLongHandler#serviceStop().
org.apache.hadoop.mapred.TestJobCleanup can fail because of
RejectedExecutionException by NonAggregatingLogHandler.
{code}
2013-11-13 10:53:06,970 FATAL [AsyncDispatcher event handler]
event.AsyncDispatcher (AsyncDispatcher.java:dispatch(166)) - Error in
dispatcher thread
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@d51df63
rejected from
java.util.concurrent.ScheduledThreadPoolExecutor@7a20e369[Shutting down, pool
size = 4, active threads = 0, queued tasks = 7, completed tasks = 0]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:325)
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:530)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler.handle(NonAggregatingLogHandler.java:121)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler.handle(NonAggregatingLogHandler.java:49)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:159)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:95)
at java.lang.Thread.run(Thread.java:724)
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1172) Convert *SecretManagers in the RM to services
[ https://issues.apache.org/jira/browse/YARN-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13856824#comment-13856824 ] Hudson commented on YARN-1172: -- FAILURE: Integrated in Hadoop-Yarn-trunk #433 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/433/]) YARN-1172. Convert SecretManagers in RM to services (Tsuyoshi OZAWA via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1553431) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMSecretManagerService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/QueueACLsTestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestClientToAMTokens.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestRMDelegationTokens.java Convert *SecretManagers in the RM to services - Key: YARN-1172 URL: https://issues.apache.org/jira/browse/YARN-1172 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Karthik Kambatla Assignee: Tsuyoshi OZAWA Fix For: 2.4.0 Attachments: YARN-1172.1.patch, YARN-1172.10.patch, YARN-1172.11.patch, YARN-1172.12.patch, YARN-1172.13.patch, YARN-1172.14.patch, YARN-1172.15.patch, YARN-1172.2.patch, YARN-1172.3.patch, YARN-1172.4.patch, YARN-1172.5.patch, YARN-1172.6.patch, YARN-1172.7.patch, YARN-1172.8.patch, YARN-1172.9.patch, yarn-1172-16.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1172) Convert *SecretManagers in the RM to services
[ https://issues.apache.org/jira/browse/YARN-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13856869#comment-13856869 ] Hudson commented on YARN-1172: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1624 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1624/]) YARN-1172. Convert SecretManagers in RM to services (Tsuyoshi OZAWA via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1553431) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMSecretManagerService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/QueueACLsTestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestClientToAMTokens.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestRMDelegationTokens.java Convert *SecretManagers in the RM to services - Key: YARN-1172 URL: https://issues.apache.org/jira/browse/YARN-1172 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Karthik Kambatla Assignee: Tsuyoshi OZAWA Fix For: 2.4.0 Attachments: YARN-1172.1.patch, YARN-1172.10.patch, YARN-1172.11.patch, YARN-1172.12.patch, YARN-1172.13.patch, YARN-1172.14.patch, YARN-1172.15.patch, YARN-1172.2.patch, YARN-1172.3.patch, YARN-1172.4.patch, YARN-1172.5.patch, YARN-1172.6.patch, YARN-1172.7.patch, YARN-1172.8.patch, YARN-1172.9.patch, yarn-1172-16.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1172) Convert *SecretManagers in the RM to services
[ https://issues.apache.org/jira/browse/YARN-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13856899#comment-13856899 ] Hudson commented on YARN-1172: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1650 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1650/]) YARN-1172. Convert SecretManagers in RM to services (Tsuyoshi OZAWA via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1553431) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMSecretManagerService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/QueueACLsTestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestClientToAMTokens.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestRMDelegationTokens.java Convert *SecretManagers in the RM to services - Key: YARN-1172 URL: https://issues.apache.org/jira/browse/YARN-1172 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Karthik Kambatla Assignee: Tsuyoshi OZAWA Fix For: 2.4.0 Attachments: YARN-1172.1.patch, YARN-1172.10.patch, YARN-1172.11.patch, YARN-1172.12.patch, YARN-1172.13.patch, YARN-1172.14.patch, YARN-1172.15.patch, YARN-1172.2.patch, YARN-1172.3.patch, YARN-1172.4.patch, YARN-1172.5.patch, YARN-1172.6.patch, YARN-1172.7.patch, YARN-1172.8.patch, YARN-1172.9.patch, yarn-1172-16.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (YARN-1532) Unable to run any MR program on Windows 7 for hadoop2.2 GA Exception from container-launch:
[
https://issues.apache.org/jira/browse/YARN-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved YARN-1532.
-
Resolution: Duplicate
This looks like a duplicate of YARN-1138, which reports that the default
configuration of {{yarn.application.classpath}} uses Unix-specific syntax for
environment variables. Until a patch is available for that, you can work
around the problem by overriding your configuration in yarn-site.xml to use
Windows-style environment variables, like this:
{code}
property
nameyarn.application.classpath/name
value%HADOOP_CONF_DIR%,%HADOOP_COMMON_HOME%/share/hadoop/common/*,%HADOOP_COMMON_HOME%/share/hadoop/common/lib/*,%HADOOP_HDFS_HOME%/share/hadoop/hdfs/*,%HADOOP_HDFS_HOME%/share/hadoop/hdfs/lib/*,%HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/*,%HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/lib/*,%HADOOP_YARN_HOME%/share/hadoop/yarn/*,%HADOOP_YARN_HOME%/share/hadoop/yarn/lib/*/value
/property
{code}
Unable to run any MR program on Windows 7 for hadoop2.2 GA Exception from
container-launch:
--
Key: YARN-1532
URL: https://issues.apache.org/jira/browse/YARN-1532
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.2.0
Environment: Windows 7(64 bit)
Reporter: Hadoop Developer
Priority: Critical
Unable to run any MR program on Hadoop 2.2 because of some classpath issue
which is failing container launch.
user logs on namenode shows:
java.lang.NoClassDefFoundError: org/apache/hadoop/service/CompositeService
and
Could not find the main class:
org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (YARN-1538) Support web-services for application-submission and AM protocols
Vinod Kumar Vavilapalli created YARN-1538: - Summary: Support web-services for application-submission and AM protocols Key: YARN-1538 URL: https://issues.apache.org/jira/browse/YARN-1538 Project: Hadoop YARN Issue Type: New Feature Reporter: Vinod Kumar Vavilapalli We already have read-only web-services for YARN - MAPREDUCE-2863. It'll be great to support APIs for application submission and the scheduling protocol - as alternatives to RPCs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1538) Support web-services for application-submission and AM protocols
[ https://issues.apache.org/jira/browse/YARN-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857030#comment-13857030 ] Alejandro Abdelnur commented on YARN-1538: -- Is the idea to fully duplicate all AM facing RPC interfaces over HTTP? Having scheduling over HTTP but not the NM interface over HTTP does not buy much as you'll need to use RPC for starting the containers. Also, how about security and token support over HTTP? Support web-services for application-submission and AM protocols Key: YARN-1538 URL: https://issues.apache.org/jira/browse/YARN-1538 Project: Hadoop YARN Issue Type: New Feature Reporter: Vinod Kumar Vavilapalli We already have read-only web-services for YARN - MAPREDUCE-2863. It'll be great to support APIs for application submission and the scheduling protocol - as alternatives to RPCs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1244) Missing yarn queue-cli
[ https://issues.apache.org/jira/browse/YARN-1244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1244: -- Target Version/s: 2.3.0 Labels: newbie (was: ) Issue Type: New Feature (was: Bug) Missing yarn queue-cli -- Key: YARN-1244 URL: https://issues.apache.org/jira/browse/YARN-1244 Project: Hadoop YARN Issue Type: New Feature Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Labels: newbie We don't have a yarn queue CLI. For now mapred still has one that is working, but we need to move over that functionality to yarn CLI itself. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (YARN-1244) Missing yarn queue-cli
[ https://issues.apache.org/jira/browse/YARN-1244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli reassigned YARN-1244: - Assignee: (was: Vinod Kumar Vavilapalli) Missing yarn queue-cli -- Key: YARN-1244 URL: https://issues.apache.org/jira/browse/YARN-1244 Project: Hadoop YARN Issue Type: New Feature Reporter: Vinod Kumar Vavilapalli Labels: newbie We don't have a yarn queue CLI. For now mapred still has one that is working, but we need to move over that functionality to yarn CLI itself. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1244) Missing yarn queue-cli
[ https://issues.apache.org/jira/browse/YARN-1244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857034#comment-13857034 ] Vinod Kumar Vavilapalli commented on YARN-1244: --- Today we have and so need to support at least some of - mapred queue -list - mapred queue -showacls - mapred queue -info - mapred queue -showJobs Missing yarn queue-cli -- Key: YARN-1244 URL: https://issues.apache.org/jira/browse/YARN-1244 Project: Hadoop YARN Issue Type: New Feature Reporter: Vinod Kumar Vavilapalli Labels: newbie We don't have a yarn queue CLI. For now mapred still has one that is working, but we need to move over that functionality to yarn CLI itself. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1539) Queue admin ACLs should NOT be similar to submit-acls w.r.t hierarchy.
Vinod Kumar Vavilapalli created YARN-1539: - Summary: Queue admin ACLs should NOT be similar to submit-acls w.r.t hierarchy. Key: YARN-1539 URL: https://issues.apache.org/jira/browse/YARN-1539 Project: Hadoop YARN Issue Type: Bug Reporter: Vinod Kumar Vavilapalli Priority: Critical Today, Queue admin ACLs are similar to submit-acls w.r.t hierarchy in that if one has to be able to administer a queue, he/she should be an admin of all the queues in the ancestry - an unnecessary burden. This was added in YARN-899 and I believe is wrong semantics as well as implementation. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1539) Queue admin ACLs should NOT be similar to submit-acls w.r.t hierarchy.
[
https://issues.apache.org/jira/browse/YARN-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857050#comment-13857050
]
Sandy Ryza commented on YARN-1539:
--
My understanding was that the way that both submit and admin work is that: to
have access to a queue, you need to be in its access control list OR in the
access control list of any of its ancestors. Where is the unnecessary burden?
Also, would this not be an incompatible change?
{code}
@Override
public boolean hasAccess(QueueACL acl, UserGroupInformation user) {
synchronized (this) {
if (acls.get(acl).isUserAllowed(user)) {
return true;
}
}
if (parent != null) {
return parent.hasAccess(acl, user);
}
return false;
}
{code}
Queue admin ACLs should NOT be similar to submit-acls w.r.t hierarchy.
--
Key: YARN-1539
URL: https://issues.apache.org/jira/browse/YARN-1539
Project: Hadoop YARN
Issue Type: Bug
Reporter: Vinod Kumar Vavilapalli
Priority: Critical
Today, Queue admin ACLs are similar to submit-acls w.r.t hierarchy in that if
one has to be able to administer a queue, he/she should be an admin of all
the queues in the ancestry - an unnecessary burden.
This was added in YARN-899 and I believe is wrong semantics as well as
implementation.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1539) Queue admin ACLs should NOT be similar to submit-acls w.r.t hierarchy.
[ https://issues.apache.org/jira/browse/YARN-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857064#comment-13857064 ] Vinod Kumar Vavilapalli commented on YARN-1539: --- bq. you need to be in its access control list OR in the access control list of any of its ancestors. It is AND not OR in Capacity-Scheduler. And that makes sense for submit-acls. You need to have permissions to submit to a parent-queue, child-queue, leaf queue etc. But not for admin ACLs. Seems like that is different for FairScheduler. All the more we should fix this to be consistent. FairScheduler should also change w.r.t submit-acls. Yes, this would be an incompatible change, but don't know of a solution otherwise. I found this a while back ([here|https://issues.apache.org/jira/browse/YARN-899?focusedCommentId=13781287page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13781287]) but failed to file the ticket and fix it in time. Queue admin ACLs should NOT be similar to submit-acls w.r.t hierarchy. -- Key: YARN-1539 URL: https://issues.apache.org/jira/browse/YARN-1539 Project: Hadoop YARN Issue Type: Bug Reporter: Vinod Kumar Vavilapalli Priority: Critical Today, Queue admin ACLs are similar to submit-acls w.r.t hierarchy in that if one has to be able to administer a queue, he/she should be an admin of all the queues in the ancestry - an unnecessary burden. This was added in YARN-899 and I believe is wrong semantics as well as implementation. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1539) Queue admin ACLs should NOT be similar to submit-acls w.r.t hierarchy.
[ https://issues.apache.org/jira/browse/YARN-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1539: -- Issue Type: Sub-task (was: Bug) Parent: YARN-47 Queue admin ACLs should NOT be similar to submit-acls w.r.t hierarchy. -- Key: YARN-1539 URL: https://issues.apache.org/jira/browse/YARN-1539 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Priority: Critical Today, Queue admin ACLs are similar to submit-acls w.r.t hierarchy in that if one has to be able to administer a queue, he/she should be an admin of all the queues in the ancestry - an unnecessary burden. This was added in YARN-899 and I believe is wrong semantics as well as implementation. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1538) Support web-services for application-submission and AM protocols
[ https://issues.apache.org/jira/browse/YARN-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1538: -- Description: We already have read-only web-services for YARN - MAPREDUCE-2863. It'll be great to support APIs for application submission and the AM protocols - scheduling and container-management - as alternatives to RPCs. (was: We already have read-only web-services for YARN - MAPREDUCE-2863. It'll be great to support APIs for application submission and the scheduling protocol - as alternatives to RPCs.) Yes I meant all of them by saying AM protocols. Fixing description. Support web-services for application-submission and AM protocols Key: YARN-1538 URL: https://issues.apache.org/jira/browse/YARN-1538 Project: Hadoop YARN Issue Type: New Feature Reporter: Vinod Kumar Vavilapalli We already have read-only web-services for YARN - MAPREDUCE-2863. It'll be great to support APIs for application submission and the AM protocols - scheduling and container-management - as alternatives to RPCs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1523) Replace RMNotYetReadyException with the standard hadoop io StandbyException
[ https://issues.apache.org/jira/browse/YARN-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857071#comment-13857071 ] Karthik Kambatla commented on YARN-1523: bq. Should we make the throwStandbyException() a method of HAUtil so that we could reuse it in other places if need be? Like the idea. Let us do it when we have another place that throws StandbyException. bq. When we want the standby RM to re-direct the client to the active RM then how do you think we would do that? The two options appear to be (1) notifying the client (through a derivate of StandbyException) that we are not Active and that the Active is so-and-so (2) relaying the client request to the Active RM and responding to the Client. The first approach might be simpler and have fewer round-trips overall, particularly if there are only few kinds of clients. Will commit this later today. Replace RMNotYetReadyException with the standard hadoop io StandbyException --- Key: YARN-1523 URL: https://issues.apache.org/jira/browse/YARN-1523 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.4.0 Reporter: Bikas Saha Assignee: Karthik Kambatla Attachments: yarn-1523-1.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1539) Queue admin ACLs should NOT be similar to submit-acls w.r.t hierarchy.
[ https://issues.apache.org/jira/browse/YARN-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1539: --- Hadoop Flags: Incompatible change Queue admin ACLs should NOT be similar to submit-acls w.r.t hierarchy. -- Key: YARN-1539 URL: https://issues.apache.org/jira/browse/YARN-1539 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Priority: Critical Today, Queue admin ACLs are similar to submit-acls w.r.t hierarchy in that if one has to be able to administer a queue, he/she should be an admin of all the queues in the ancestry - an unnecessary burden. This was added in YARN-899 and I believe is wrong semantics as well as implementation. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1539) Queue admin ACLs should NOT be similar to submit-acls w.r.t hierarchy.
[ https://issues.apache.org/jira/browse/YARN-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857074#comment-13857074 ] Sandy Ryza commented on YARN-1539: -- The code I posted above is from the Capacity Scheduler. It looks like OR to me. The Fair Scheduler has slightly different code, but we took the semantics from this. Am I missing something? Queue admin ACLs should NOT be similar to submit-acls w.r.t hierarchy. -- Key: YARN-1539 URL: https://issues.apache.org/jira/browse/YARN-1539 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Priority: Critical Today, Queue admin ACLs are similar to submit-acls w.r.t hierarchy in that if one has to be able to administer a queue, he/she should be an admin of all the queues in the ancestry - an unnecessary burden. This was added in YARN-899 and I believe is wrong semantics as well as implementation. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1540) Add an easy way to turn on HA
Karthik Kambatla created YARN-1540: -- Summary: Add an easy way to turn on HA Key: YARN-1540 URL: https://issues.apache.org/jira/browse/YARN-1540 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.4.0 Reporter: Karthik Kambatla Users will have to modify the configuration significantly to turn on HA. It would be nice to have a simpler way of doing this. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1540) Add an easy way to turn on HA
[ https://issues.apache.org/jira/browse/YARN-1540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857089#comment-13857089 ] Karthik Kambatla commented on YARN-1540: The bare minimum configs required to be specified should be: # Enable HA with default options # ZK address for the embedded elector and store # Hostnames for the two RMs A utility method, to be invoked before anything else, could set all other configs based on these. Add an easy way to turn on HA - Key: YARN-1540 URL: https://issues.apache.org/jira/browse/YARN-1540 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.4.0 Reporter: Karthik Kambatla Users will have to modify the configuration significantly to turn on HA. It would be nice to have a simpler way of doing this. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1527) yarn rmadmin command prints wrong usage info:
[
https://issues.apache.org/jira/browse/YARN-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857096#comment-13857096
]
Jian He commented on YARN-1527:
---
patch looks good, +1.
yarn rmadmin command prints wrong usage info:
-
Key: YARN-1527
URL: https://issues.apache.org/jira/browse/YARN-1527
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Jian He
Assignee: Akira AJISAKA
Labels: newbie
Attachments: YARN-1527.patch
The usage should be: yarn rmadmin, instead of java RMAdmin, and the
-refreshQueues should be in the second line.
{code} Usage: java RMAdmin -refreshQueues
-refreshNodes
-refreshSuperUserGroupsConfiguration
-refreshUserToGroupsMappings
-refreshAdminAcls
-refreshServiceAcl
-getGroups [username]
-help [cmd]
-transitionToActive serviceId
-transitionToStandby serviceId
-failover [--forcefence] [--forceactive] serviceId serviceId
-getServiceState serviceId
-checkHealth serviceId
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1527) yarn rmadmin command prints wrong usage info:
[
https://issues.apache.org/jira/browse/YARN-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857099#comment-13857099
]
Jian He commented on YARN-1527:
---
Compiled and run the command myself, now the usage looks as the following:
{code}
Usage: yarn rmadmin
-refreshQueues
-refreshNodes
-refreshSuperUserGroupsConfiguration
-refreshUserToGroupsMappings
-refreshAdminAcls
-refreshServiceAcl
-getGroups [username]
-help [cmd]
-transitionToActive serviceId
-transitionToStandby serviceId
-failover [--forcefence] [--forceactive] serviceId serviceId
-getServiceState serviceId
-checkHealth serviceId
{code}
yarn rmadmin command prints wrong usage info:
-
Key: YARN-1527
URL: https://issues.apache.org/jira/browse/YARN-1527
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Jian He
Assignee: Akira AJISAKA
Labels: newbie
Attachments: YARN-1527.patch
The usage should be: yarn rmadmin, instead of java RMAdmin, and the
-refreshQueues should be in the second line.
{code} Usage: java RMAdmin -refreshQueues
-refreshNodes
-refreshSuperUserGroupsConfiguration
-refreshUserToGroupsMappings
-refreshAdminAcls
-refreshServiceAcl
-getGroups [username]
-help [cmd]
-transitionToActive serviceId
-transitionToStandby serviceId
-failover [--forcefence] [--forceactive] serviceId serviceId
-getServiceState serviceId
-checkHealth serviceId
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1527) yarn rmadmin command prints wrong usage info:
[
https://issues.apache.org/jira/browse/YARN-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857126#comment-13857126
]
Jian He commented on YARN-1527:
---
Committed to trunk and branch-2, thanks [~ajisakaa] !
yarn rmadmin command prints wrong usage info:
-
Key: YARN-1527
URL: https://issues.apache.org/jira/browse/YARN-1527
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Jian He
Assignee: Akira AJISAKA
Labels: newbie
Fix For: 2.4.0
Attachments: YARN-1527.patch
The usage should be: yarn rmadmin, instead of java RMAdmin, and the
-refreshQueues should be in the second line.
{code} Usage: java RMAdmin -refreshQueues
-refreshNodes
-refreshSuperUserGroupsConfiguration
-refreshUserToGroupsMappings
-refreshAdminAcls
-refreshServiceAcl
-getGroups [username]
-help [cmd]
-transitionToActive serviceId
-transitionToStandby serviceId
-failover [--forcefence] [--forceactive] serviceId serviceId
-getServiceState serviceId
-checkHealth serviceId
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1527) yarn rmadmin command prints wrong usage info:
[
https://issues.apache.org/jira/browse/YARN-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857127#comment-13857127
]
Hudson commented on YARN-1527:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #4929 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/4929/])
YARN-1527. Fixed yarn rmadmin command to print the correct usage info.
Contributed by Akira AJISAKA. (jianhe:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1553596)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestRMAdminCLI.java
yarn rmadmin command prints wrong usage info:
-
Key: YARN-1527
URL: https://issues.apache.org/jira/browse/YARN-1527
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Jian He
Assignee: Akira AJISAKA
Labels: newbie
Fix For: 2.4.0
Attachments: YARN-1527.patch
The usage should be: yarn rmadmin, instead of java RMAdmin, and the
-refreshQueues should be in the second line.
{code} Usage: java RMAdmin -refreshQueues
-refreshNodes
-refreshSuperUserGroupsConfiguration
-refreshUserToGroupsMappings
-refreshAdminAcls
-refreshServiceAcl
-getGroups [username]
-help [cmd]
-transitionToActive serviceId
-transitionToStandby serviceId
-failover [--forcefence] [--forceactive] serviceId serviceId
-getServiceState serviceId
-checkHealth serviceId
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (YARN-1541) Invalidate AM Host/Port when app attempt is done so that in the mean-while client doesn’t get wrong information.
Jian He created YARN-1541: - Summary: Invalidate AM Host/Port when app attempt is done so that in the mean-while client doesn’t get wrong information. Key: YARN-1541 URL: https://issues.apache.org/jira/browse/YARN-1541 Project: Hadoop YARN Issue Type: Bug Reporter: Jian He Assignee: Jian He -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1541) Invalidate AM Host/Port when app attempt is done so that in the mean-while client doesn’t get wrong information.
[ https://issues.apache.org/jira/browse/YARN-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-1541: -- Attachment: YARN-1541.1.patch New patch invalidates the AM's host and roc port when the AM attempt is done Invalidate AM Host/Port when app attempt is done so that in the mean-while client doesn’t get wrong information. Key: YARN-1541 URL: https://issues.apache.org/jira/browse/YARN-1541 Project: Hadoop YARN Issue Type: Bug Reporter: Jian He Assignee: Jian He Attachments: YARN-1541.1.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1541) Invalidate AM Host/Port when app attempt is done so that in the mean-while client doesn’t get wrong information.
[
https://issues.apache.org/jira/browse/YARN-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857171#comment-13857171
]
Hadoop QA commented on YARN-1541:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12620560/YARN-1541.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2730//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2730//console
This message is automatically generated.
Invalidate AM Host/Port when app attempt is done so that in the mean-while
client doesn’t get wrong information.
Key: YARN-1541
URL: https://issues.apache.org/jira/browse/YARN-1541
Project: Hadoop YARN
Issue Type: Bug
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-1541.1.patch
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1461) RM API and RM changes to handle tags for running jobs
[ https://issues.apache.org/jira/browse/YARN-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1461: --- Attachment: yarn-1461-1.patch Patch that adds tags. RM API and RM changes to handle tags for running jobs - Key: YARN-1461 URL: https://issues.apache.org/jira/browse/YARN-1461 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.2.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: yarn-1461-1.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1413) [YARN-321] AHS WebUI should server aggregated logs as well
[ https://issues.apache.org/jira/browse/YARN-1413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857194#comment-13857194 ] Vinod Kumar Vavilapalli commented on YARN-1413: --- There's a minor merge conflict. Can you please update? Tx. Also, there seem to be issues with container-logs URL on AHS redirect which in turn will redirect to the node once the log-aggregation finishes - causing an infinite redirect cycle. Are you not seeing this? Is this tested with log-aggregation on? [YARN-321] AHS WebUI should server aggregated logs as well -- Key: YARN-1413 URL: https://issues.apache.org/jira/browse/YARN-1413 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Attachments: YARN-1413-1.patch, YARN-1413-2.patch, YARN-1413-3.patch, YARN-1413-4.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1542) Add unit test for public resource on viewfs
Gera Shegalov created YARN-1542: --- Summary: Add unit test for public resource on viewfs Key: YARN-1542 URL: https://issues.apache.org/jira/browse/YARN-1542 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Reporter: Gera Shegalov -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1461) RM API and RM changes to handle tags for running jobs
[
https://issues.apache.org/jira/browse/YARN-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857219#comment-13857219
]
Hadoop QA commented on YARN-1461:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12620567/yarn-1461-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2731//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/2731//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2731//console
This message is automatically generated.
RM API and RM changes to handle tags for running jobs
-
Key: YARN-1461
URL: https://issues.apache.org/jira/browse/YARN-1461
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Affects Versions: 2.2.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: yarn-1461-1.patch
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (YARN-1461) RM API and RM changes to handle tags for running jobs
[ https://issues.apache.org/jira/browse/YARN-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1461: --- Attachment: yarn-1461-2.patch Patch to fix the findbugs warnings. TestClientRMService passes locally. Will look deeper if it fails again. RM API and RM changes to handle tags for running jobs - Key: YARN-1461 URL: https://issues.apache.org/jira/browse/YARN-1461 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.2.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: yarn-1461-1.patch, yarn-1461-2.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1542) Add unit test for public resource on viewfs
[ https://issues.apache.org/jira/browse/YARN-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated YARN-1542: Attachment: YARN-1542.v01.patch Add unit test for public resource on viewfs --- Key: YARN-1542 URL: https://issues.apache.org/jira/browse/YARN-1542 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Reporter: Gera Shegalov Attachments: YARN-1542.v01.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1461) RM API and RM changes to handle tags for running jobs
[
https://issues.apache.org/jira/browse/YARN-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857241#comment-13857241
]
Hadoop QA commented on YARN-1461:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12620574/yarn-1461-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2732//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2732//console
This message is automatically generated.
RM API and RM changes to handle tags for running jobs
-
Key: YARN-1461
URL: https://issues.apache.org/jira/browse/YARN-1461
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Affects Versions: 2.2.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: yarn-1461-1.patch, yarn-1461-2.patch
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1542) Add unit test for public resource on viewfs
[
https://issues.apache.org/jira/browse/YARN-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857243#comment-13857243
]
Hadoop QA commented on YARN-1542:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12620577/YARN-1542.v01.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:
org.apache.hadoop.yarn.util.TestFSDownload
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2733//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2733//console
This message is automatically generated.
Add unit test for public resource on viewfs
---
Key: YARN-1542
URL: https://issues.apache.org/jira/browse/YARN-1542
Project: Hadoop YARN
Issue Type: Improvement
Components: nodemanager
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: YARN-1542.v01.patch
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1542) Add unit test for public resource on viewfs
[ https://issues.apache.org/jira/browse/YARN-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857255#comment-13857255 ] Gera Shegalov commented on YARN-1542: - The failed unit test is being added by this JIRA to demonstrate the problem. it would be fixed by HADOOP-10191 Add unit test for public resource on viewfs --- Key: YARN-1542 URL: https://issues.apache.org/jira/browse/YARN-1542 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Reporter: Gera Shegalov Assignee: Gera Shegalov Attachments: YARN-1542.v01.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1523) Use StandbyException instead of RMNotYetReadyException
[ https://issues.apache.org/jira/browse/YARN-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1523: --- Summary: Use StandbyException instead of RMNotYetReadyException (was: Replace RMNotYetReadyException with the standard hadoop io StandbyException) Use StandbyException instead of RMNotYetReadyException -- Key: YARN-1523 URL: https://issues.apache.org/jira/browse/YARN-1523 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.4.0 Reporter: Bikas Saha Assignee: Karthik Kambatla Attachments: yarn-1523-1.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1541) Invalidate AM Host/Port when app attempt is done so that in the mean-while client doesn’t get wrong information.
[ https://issues.apache.org/jira/browse/YARN-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857283#comment-13857283 ] Vinod Kumar Vavilapalli commented on YARN-1541: --- Can you add a test in TestRM? Validating transitions directly is not so useful. Invalidate AM Host/Port when app attempt is done so that in the mean-while client doesn’t get wrong information. Key: YARN-1541 URL: https://issues.apache.org/jira/browse/YARN-1541 Project: Hadoop YARN Issue Type: Bug Reporter: Jian He Assignee: Jian He Attachments: YARN-1541.1.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1523) Use StandbyException instead of RMNotYetReadyException
[ https://issues.apache.org/jira/browse/YARN-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857287#comment-13857287 ] Hudson commented on YARN-1523: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4930 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4930/]) YARN-1523. Use StandbyException instead of RMNotYetReadyException (kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1553616) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/exceptions/RMNotYetActiveException.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ResourceManagerAdministrationProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java Use StandbyException instead of RMNotYetReadyException -- Key: YARN-1523 URL: https://issues.apache.org/jira/browse/YARN-1523 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.4.0 Reporter: Bikas Saha Assignee: Karthik Kambatla Fix For: 2.4.0 Attachments: yarn-1523-1.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation
[
https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857315#comment-13857315
]
Xuan Gong commented on YARN-1521:
-
an idempotent operation is one that has no additional effect if it is called
more than once with the same input parameters. That means no matter how many
times you call the operation with the same parameter, the result will be the
same. In our HA case, i think that it means no matter which RM is active, by
calling the operations, we will get the same result.
Based on this, here is my proposal:
{code}
In ApplicationClientProtocol :
{code}
a. getNewApplication(), submitApplication(), forceKillApplication(),
getApplicationReport(),
getApplications(),getQueueUserAclInfo(),getDelegationToken(),renewDelegationToken(),
cancelDelegationToken() need to add idempotent annotation.
When RM becomes Active, it will read back all the Applications, and all related
DelegationTokens back from the zk, in that case, no matter which RM is active,
it will get all the information for all the applications, either running or
finished. That is the reason why we can always get the same results if we call
getApplicationReport(),
getApplications(),getDelegationToken(),renewDelegationToken(),
cancelDelegationToken(). So, the idempotent annotation need to be added for
those operations.
For forceKillApplication(), getNewApplication(), submitApplication(), those
operation need to add idempotent annotation too. Because, when the user calls
them, no matter which RM is active, the application will be submitted or killed.
For getQueueUserAclInfo(), YARN-1459 will guarantee that other RMs will get the
latest configuration if one of RMs are changed its configuration. In this case,
when getQueueUserAclInfo is called, we will always get the same results no
matter which RM is active.
b. getClusterMetrics(), getClusterNodes(),getQueueInfo() do not need to add.
I do not want to add idempotent annotation. Because those are based on the
Nodes information or queue information. The current active RM has the resource
information about all the NMs, and queue info for all the applications. But
when failover happens, the next active RMs will not have all the NMs
information immediately, and it does not have any queue info (such as which
application in which queue).
{code}
In ResourceTracker:
{code}
a.registerNodeManager needs to add idempotent annotation.
Same reason why I want to add idempotent annotation for submitApplication()
b. nodeHeartbeat does not need to add
The new active RM does not know all the NMs immediately.
{code}
In ApplicationMasterProtocol
{code}
Does not need to add for all the methods in ApplicationMasterProtocol.
There are three operations: registerApplicationMaster, finishApplicationMaster,
allocate(). When the failover happens, a new RM will start a new Attempt for
the running AM (just like the RM restart).
{code}
In ResourceManagerAdministrationProtocol
{code}
a.refreshQueues(), refreshNodes(),
refreshSuperUserGroupsConfiguration(),RefreshUserToGroupsMappingsRequest(),
refreshAdminAcls(), refreshServiceAcls() need to add idempotent annotation.
YARN-1459 will guarantee that other RMs will get the latest configuration if
one of RMs are changed its configuration.
b.updateNodeResource() do not need to add
The new active RM does not know all the NMs immediately.
Mark appropriate protocol methods with the idempotent annotation
Key: YARN-1521
URL: https://issues.apache.org/jira/browse/YARN-1521
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Xuan Gong
After YARN-1028, we add the automatically failover into RMProxy. This JIRA is
to identify whether we need to add idempotent annotation and which methods
can be marked as idempotent.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857318#comment-13857318 ] Xuan Gong commented on YARN-1521: - [~bikassaha], [~vinodkv], [~kkambatl] Comments, please Mark appropriate protocol methods with the idempotent annotation Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1543) Stateless multiple resource manager
Tsuyoshi OZAWA created YARN-1543: Summary: Stateless multiple resource manager Key: YARN-1543 URL: https://issues.apache.org/jira/browse/YARN-1543 Project: Hadoop YARN Issue Type: Improvement Reporter: Tsuyoshi OZAWA Currently, YARN is implemented based on statefull single resource manager architecture. This approach can raise operation costs compare to stateless multi resource manager architecture, because we need to care which resource manager is active to ensure availability. We can avoid this problem by saving all states into updatable and transactional RMStateStore such as ZooKeeper or RDBMS. If we take this approach, multiple resource managers can be launched at the same time, we can solve this problem. Operation cost can be reduced, because we don't need to maintain specific(current master) resource manager. This idea is based on a paper: “Omega: flexible, scalable schedulers for large compute clusters”[1]. [1] http://eurosys2013.tudos.org/wp-content/uploads/2013/paper/Schwarzkopf.pdf -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857328#comment-13857328 ] Karthik Kambatla commented on YARN-1521: bq. an idempotent operation is one that has no additional effect if it is called more than once with the same input parameters. That means no matter how many times you call the operation with the same parameter, the result will be the same. In our HA case, i think that it means no matter which RM is active, by calling the operations, we will get the same result. I am not sure if this is completely accurate. Rephrasing the definition from wiki - an idempotent operation can be applied multiple times without changing the result beyond the initial application. The relevance to that in the context of HA and client failover, I believe, has to do with whether an operation can be retried. For instance, submitting an application is not idempotent. Suppose a client is in the process of submitting an application to the Active (say, RM1) and RM1 goes down. RM2 takes over. Now, we don't know the status of the app submission and would be wrong to retry submitting to RM2. In this case, the client shouldn't failover to RM2. Unfortunately, there might be retries in MR and YARN without checking for idempotence or status of the previous attempt. I haven't run into any issues yet, but I think it would be nice to handle all of them cleanly. Mark appropriate protocol methods with the idempotent annotation Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1543) Stateless multiple resource manager
[ https://issues.apache.org/jira/browse/YARN-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated YARN-1543: - Description: Currently, YARN is implemented based on statefull single resource manager architecture. This approach can raise operation costs compare to stateless multi resource manager architecture, because we need to care which resource manager is active to ensure availability. We can avoid this problem by saving all states into updatable and transactional RMStateStore such as ZooKeeper or RDBMS. If we take this approach, multiple resource managers can be launched at the same time. As a result, operation cost can be reduced, because we don't need to maintain specific(current master) resource manager. This idea is based on a paper: “Omega: flexible, scalable schedulers for large compute clusters”[1]. [1] http://eurosys2013.tudos.org/wp-content/uploads/2013/paper/Schwarzkopf.pdf was: Currently, YARN is implemented based on statefull single resource manager architecture. This approach can raise operation costs compare to stateless multi resource manager architecture, because we need to care which resource manager is active to ensure availability. We can avoid this problem by saving all states into updatable and transactional RMStateStore such as ZooKeeper or RDBMS. If we take this approach, multiple resource managers can be launched at the same time, we can solve this problem. Operation cost can be reduced, because we don't need to maintain specific(current master) resource manager. This idea is based on a paper: “Omega: flexible, scalable schedulers for large compute clusters”[1]. [1] http://eurosys2013.tudos.org/wp-content/uploads/2013/paper/Schwarzkopf.pdf Stateless multiple resource manager --- Key: YARN-1543 URL: https://issues.apache.org/jira/browse/YARN-1543 Project: Hadoop YARN Issue Type: Improvement Reporter: Tsuyoshi OZAWA Currently, YARN is implemented based on statefull single resource manager architecture. This approach can raise operation costs compare to stateless multi resource manager architecture, because we need to care which resource manager is active to ensure availability. We can avoid this problem by saving all states into updatable and transactional RMStateStore such as ZooKeeper or RDBMS. If we take this approach, multiple resource managers can be launched at the same time. As a result, operation cost can be reduced, because we don't need to maintain specific(current master) resource manager. This idea is based on a paper: “Omega: flexible, scalable schedulers for large compute clusters”[1]. [1] http://eurosys2013.tudos.org/wp-content/uploads/2013/paper/Schwarzkopf.pdf -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1463) TestContainerManagerSecurity#testContainerManager fails
[
https://issues.apache.org/jira/browse/YARN-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated YARN-1463:
--
Attachment: YARN-1463-20131226.txt
Taking this over as it continues to break builds.
Here's a patch, that fixes the fundamental issue, and avoids starting
http-server where possible or creates spnego keytab/princs where needed.
TestContainerManagerSecurity#testContainerManager fails
---
Key: YARN-1463
URL: https://issues.apache.org/jira/browse/YARN-1463
Project: Hadoop YARN
Issue Type: Test
Affects Versions: 2.4.0
Reporter: Ted Yu
Assignee: Binglin Chang
Attachments: YARN-1463-20131226.txt, YARN-1463.v1.patch,
YARN-1463.v2.patch
Here is stack trace:
{code}
testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
Time elapsed: 1.756 sec ERROR!
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
ResourceManager failed to start. Final state is STOPPED
at
org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:253)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:110)
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1482) WebApplicationProxy should be always-on w.r.t HA even if it is embedded in the RM
[ https://issues.apache.org/jira/browse/YARN-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857351#comment-13857351 ] Vinod Kumar Vavilapalli commented on YARN-1482: --- bq. try to try I meant try to test. WebApplicationProxy should be always-on w.r.t HA even if it is embedded in the RM - Key: YARN-1482 URL: https://issues.apache.org/jira/browse/YARN-1482 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Xuan Gong Attachments: YARN-1482.1.patch, YARN-1482.2.patch This way, even if an RM goes to standby mode, we can affect a redirect to the active. And more importantly, users will not suddenly see all their links stop working. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1544) Add Tools menu to web interface of ApplicationHistoryServer
Shinichi Yamashita created YARN-1544: Summary: Add Tools menu to web interface of ApplicationHistoryServer Key: YARN-1544 URL: https://issues.apache.org/jira/browse/YARN-1544 Project: Hadoop YARN Issue Type: Improvement Affects Versions: YARN-321 Reporter: Shinichi Yamashita Priority: Minor It should add the link of the Tools menu like ResourceManager and NodeManager in web interface of ApplicationHistoryServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1544) Add Tools menu to web interface of ApplicationHistoryServer
[
https://issues.apache.org/jira/browse/YARN-1544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857354#comment-13857354
]
Hadoop QA commented on YARN-1544:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12620593/sample-webui.png
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2735//console
This message is automatically generated.
Add Tools menu to web interface of ApplicationHistoryServer
---
Key: YARN-1544
URL: https://issues.apache.org/jira/browse/YARN-1544
Project: Hadoop YARN
Issue Type: Improvement
Affects Versions: YARN-321
Reporter: Shinichi Yamashita
Priority: Minor
Attachments: YARN-1544.patch, sample-webui.png
It should add the link of the Tools menu like ResourceManager and
NodeManager in web interface of ApplicationHistoryServer.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857358#comment-13857358 ] Bikas Saha commented on YARN-1521: -- After the first application of an idempotent operation, the subsequent application of the operation should not change the state of the system in any substantial manner. So its not clear to me how submitApplication() is idempotent. If the first operation succeeds then the second must fail. getNewApplication() will create a new application id on the RM for every invocation. It may be harmless to retry but its probably not idempotent. renewDelegationToken() - again harmless to retry but probably not idempotent. killApplication() is idempotent. getApplicationReport() is idempotent. getDelegationToken() - probably not since it results in creation of a new delegation token. Then there is the point of which operations are transferable via retries across RM failover. Mark appropriate protocol methods with the idempotent annotation Key: YARN-1521 URL: https://issues.apache.org/jira/browse/YARN-1521 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify whether we need to add idempotent annotation and which methods can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1029) Allow embedding leader election into the RM
[ https://issues.apache.org/jira/browse/YARN-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857368#comment-13857368 ] Bikas Saha commented on YARN-1029: -- bq. The method should have been synchronized in YARN-1481, Vinod Kumar Vavilapalli and I thought we could handle it here. Could do an addendum patch there instead if that is preferred. Please take care of it wherever appropriate. bq. Not sure how advantageous that will be - we ll end up calling the common method instead of ExitUtil.terminate only for the common method to call it? Also, getCause() doesn't exist in AbstractEvent requiring us to add a new kind of event (CausedEvent?) that both these events extend. Seems too complicated for the gain. The advantage is that future changes will need to edit one place in the code and forget about other places. Not a major point though. Making life easier for future devs. bq. Agree - 100%, but would like to do it lazily when another such case pops up. Again, if we organize the newly added code such that its a common event for any module to inform the RM about a fatal error then we are good for the future. Embedded elector can use that event instead of a custom named event. I am sorry I could not understand your comment explaining how the test passes with these timeouts. Been sick for a while. Probably my brain is running slow :P Allow embedding leader election into the RM --- Key: YARN-1029 URL: https://issues.apache.org/jira/browse/YARN-1029 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Karthik Kambatla Attachments: embedded-zkfc-approach.patch, yarn-1029-0.patch, yarn-1029-0.patch, yarn-1029-1.patch, yarn-1029-2.patch, yarn-1029-3.patch, yarn-1029-4.patch, yarn-1029-approach.patch It should be possible to embed common ActiveStandyElector into the RM such that ZooKeeper based leader election and notification is in-built. In conjunction with a ZK state store, this configuration will be a simple deployment option. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
