[jira] [Assigned] (YARN-50) Implement renewal / cancellation of Delegation Tokens
[ https://issues.apache.org/jira/browse/YARN-50?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth reassigned YARN-50: -- Assignee: Siddharth Seth Implement renewal / cancellation of Delegation Tokens - Key: YARN-50 URL: https://issues.apache.org/jira/browse/YARN-50 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Siddharth Seth Currently, delegation tokens issues by the RM and History server cannot be renewed or cancelled. This needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-50) Implement renewal / cancellation of Delegation Tokens
[ https://issues.apache.org/jira/browse/YARN-50?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated YARN-50: --- Attachment: YARN-50_wip.txt Work in progress. Some of the classes can be moved to different packages. Needs to be tested. One flaw with renewing tokens in their current form in the RM, is that the RM ends up depending on the token Renewers to be in the classpath. HDFS token renewal already needs the HDFS renewer (DFSClient.Renewer) to be in the RM classpath. With this patch, MR JobHistoryServer token renewal requires MR jars in the RM classpath. (The patch does not add them - but the default startup scripts already do). It also ends up requiring HistoryServer configuration parameters to be part of the RM configuration. Avoiding these dependencies doesn't seem like it'll be a small change. Implement renewal / cancellation of Delegation Tokens - Key: YARN-50 URL: https://issues.apache.org/jira/browse/YARN-50 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: YARN-50_wip.txt Currently, delegation tokens issues by the RM and History server cannot be renewed or cancelled. This needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-103) Add a yarn AM - RM client module
[
https://issues.apache.org/jira/browse/YARN-103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536908#comment-13536908
]
Alejandro Abdelnur commented on YARN-103:
-
If developers (of AMs) suppose to use the interface only, then they should not
see the implementing class. They should get an interface instance via a
factory. The implementing class should not be in the same package or it should
be package private. Using annotations in public API on methods to state they
are for testing only is calling for trouble as developers will use those
methods.
As I assume this pattern will hold true for more than one interface/impl set,
we could have a generic factory for this:
{code}
public T T create(ClassT klass, Configuration conf);
{code}
Internally we can use our good old {{ReflectionUtils.newInstance()}} and drive
the implementation from a configuration property.
Add a yarn AM - RM client module
Key: YARN-103
URL: https://issues.apache.org/jira/browse/YARN-103
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Bikas Saha
Assignee: Bikas Saha
Attachments: YARN-103.1.patch, YARN-103.2.patch, YARN-103.3.patch,
YARN-103.4.patch, YARN-103.4.wrapper.patch, YARN-103.5.patch,
YARN-103.6.patch, YARN-103.7.patch
Add a basic client wrapper library to the AM RM protocol in order to prevent
proliferation of code being duplicated everywhere. Provide helper functions
to perform reverse mapping of container requests to RM allocation resource
request table format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-103) Add a yarn AM - RM client module
[ https://issues.apache.org/jira/browse/YARN-103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536929#comment-13536929 ] Tom White commented on YARN-103: The interface shouldn't really be implemented by anyone outside of YARN This is the heart of the problem. We don't have a way to say (via the audience annotations) that an interface is for read-only use only - and not for users to implement. An interface may be @Public @Stable from the point of view of a user who wants to call it, but that doesn't mean that folks should implement it themselves, since for interfaces like the one we are discussing we might want to add a new method, say (note that such a change is compatible with @Stable). Adding a new method is fine for the first type of user, but not for the second, since their implementation breaks. In this case, I think it's likely we'll add more methods. For example, it would be useful to add a waitForState method to YarnClient (which is also an interface), which waits for a given application to reach a particular YarnApplicationState. If YarnClient were a class then this would be a compatible change, but if it's an interface then it is not. I think we should do one of the following: 1. Change YarnClient and AMRMClient to be concrete implementations. 2. Leave the interface/implementation distinction and make the interfaces @Public @Unstable. I prefer 1. since these classes are helper classes - they are not a tightly-defined interface. Add a yarn AM - RM client module Key: YARN-103 URL: https://issues.apache.org/jira/browse/YARN-103 Project: Hadoop YARN Issue Type: Improvement Reporter: Bikas Saha Assignee: Bikas Saha Attachments: YARN-103.1.patch, YARN-103.2.patch, YARN-103.3.patch, YARN-103.4.patch, YARN-103.4.wrapper.patch, YARN-103.5.patch, YARN-103.6.patch, YARN-103.7.patch Add a basic client wrapper library to the AM RM protocol in order to prevent proliferation of code being duplicated everywhere. Provide helper functions to perform reverse mapping of container requests to RM allocation resource request table format. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-103) Add a yarn AM - RM client module
[ https://issues.apache.org/jira/browse/YARN-103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536940#comment-13536940 ] Alejandro Abdelnur commented on YARN-103: - If we go that route I'd also prefer #1 and any method required for testing should be package private. Add a yarn AM - RM client module Key: YARN-103 URL: https://issues.apache.org/jira/browse/YARN-103 Project: Hadoop YARN Issue Type: Improvement Reporter: Bikas Saha Assignee: Bikas Saha Attachments: YARN-103.1.patch, YARN-103.2.patch, YARN-103.3.patch, YARN-103.4.patch, YARN-103.4.wrapper.patch, YARN-103.5.patch, YARN-103.6.patch, YARN-103.7.patch Add a basic client wrapper library to the AM RM protocol in order to prevent proliferation of code being duplicated everywhere. Provide helper functions to perform reverse mapping of container requests to RM allocation resource request table format. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-223) Change processTree interface to work better with native code
[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536945#comment-13536945 ] Dennis Y commented on YARN-223: --- This patch breaks compilation of 'Apache Hadoop Gridmix' on branch-2. There is an import of org.apache.hadoop.yarn.util.ResourceCalculatorPlugin.ProcResourceValues which were removed by this patch. 2012-12-20 13:04:28.354 [INFO] 2012-12-20 13:04:28.356 [INFO] Building Apache Hadoop Gridmix 2.0.3-SNAPSHOT 2012-12-20 13:04:28.357 [INFO] 2012-12-20 13:04:28.423 [INFO] 2012-12-20 13:04:28.425 [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ hadoop-gridmix --- 2012-12-20 13:04:28.426 [INFO] 2012-12-20 13:04:28.428 [INFO] --- maven-antrun-plugin:1.6:run (create-testdirs) @ hadoop-gridmix --- 2012-12-20 13:04:28.443 [INFO] Executing tasks 2012-12-20 13:04:28.444 2012-12-20 13:04:28.445 main: 2012-12-20 13:04:28.446 [mkdir] Created dir: /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/test-dir 2012-12-20 13:04:28.448 [INFO] Executed tasks 2012-12-20 13:04:28.450 [INFO] 2012-12-20 13:04:28.452 [INFO] --- maven-resources-plugin:2.2:resources (default-resources) @ hadoop-gridmix --- 2012-12-20 13:04:28.454 [INFO] Using default encoding to copy filtered resources. 2012-12-20 13:04:28.455 [INFO] 2012-12-20 13:04:28.457 [INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ hadoop-gridmix --- 2012-12-20 13:04:28.459 [INFO] Compiling 44 source files to /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/classes 2012-12-20 13:04:28.638 [INFO] 2012-12-20 13:04:28.640 [INFO] --- maven-resources-plugin:2.2:testResources (default-testResources) @ hadoop-gridmix --- 2012-12-20 13:04:28.642 [INFO] Using default encoding to copy filtered resources. 2012-12-20 13:04:28.643 [INFO] 2012-12-20 13:04:28.645 [INFO] --- maven-antrun-plugin:1.6:run (create-log-dir) @ hadoop-gridmix --- 2012-12-20 13:04:28.660 [INFO] Executing tasks 2012-12-20 13:04:28.661 2012-12-20 13:04:28.662 main: 2012-12-20 13:04:28.663[delete] Deleting directory /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/test-dir 2012-12-20 13:04:28.665 [mkdir] Created dir: /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/test-dir 2012-12-20 13:04:28.667 [mkdir] Created dir: /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/log 2012-12-20 13:04:28.669 [INFO] Executed tasks 2012-12-20 13:04:28.670 [INFO] 2012-12-20 13:04:28.672 [INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ hadoop-gridmix --- 2012-12-20 13:04:28.674 [INFO] Compiling 16 source files to /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/test-classes 2012-12-20 13:04:28.774 [INFO] - 2012-12-20 13:04:28.776 [ERROR] COMPILATION ERROR : 2012-12-20 13:04:28.778 [INFO] - 2012-12-20 13:04:28.780 [ERROR] /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestResourceUsageEmulators.java:[35,59] cannot find symbol 2012-12-20 13:04:28.781 symbol : class ProcResourceValues 2012-12-20 13:04:28.782 location: class org.apache.hadoop.yarn.util.ResourceCalculatorPlugin 2012-12-20 13:04:28.784 [INFO] 1 error 2012-12-20 13:04:28.786 [INFO] - Change processTree interface to work better with native code Key: YARN-223 URL: https://issues.apache.org/jira/browse/YARN-223 Project: Hadoop YARN Issue Type: Bug Reporter: Radim Kolar Assignee: Radim Kolar Priority: Critical Fix For: 3.0.0, 2.0.3-alpha Attachments: pstree-update4.txt, pstree-update6.txt, pstree-update6.txt Problem is that on every update of processTree new object is required. This is undesired when working with processTree implementation in native code. replace ProcessTree.getProcessTree() with updateProcessTree(). No new object allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact
[jira] [Commented] (YARN-223) Change processTree interface to work better with native code
[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536956#comment-13536956 ] Hudson commented on YARN-223: - Integrated in Hadoop-Yarn-trunk #71 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/71/]) YARN-223. Update process tree instead of getting new process trees. (Radim Kolar via llu) (Revision 1424244) Result = SUCCESS llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1424244 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java Change processTree interface to work better with native code Key: YARN-223 URL: https://issues.apache.org/jira/browse/YARN-223 Project: Hadoop YARN Issue Type: Bug Reporter: Radim Kolar Assignee: Radim Kolar Priority: Critical Fix For: 3.0.0, 2.0.3-alpha Attachments: pstree-update4.txt, pstree-update6.txt, pstree-update6.txt Problem is that on every update of processTree new object is required. This is undesired when working with processTree implementation in native code. replace ProcessTree.getProcessTree() with updateProcessTree(). No new object allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-223) Change processTree interface to work better with native code
[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536969#comment-13536969 ] Dennis Y commented on YARN-223: --- I've created https://issues.apache.org/jira/browse/MAPREDUCE-4895 and submitted patch Change processTree interface to work better with native code Key: YARN-223 URL: https://issues.apache.org/jira/browse/YARN-223 Project: Hadoop YARN Issue Type: Bug Reporter: Radim Kolar Assignee: Radim Kolar Priority: Critical Fix For: 3.0.0, 2.0.3-alpha Attachments: pstree-update4.txt, pstree-update6.txt, pstree-update6.txt Problem is that on every update of processTree new object is required. This is undesired when working with processTree implementation in native code. replace ProcessTree.getProcessTree() with updateProcessTree(). No new object allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-103) Add a yarn AM - RM client module
[
https://issues.apache.org/jira/browse/YARN-103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536990#comment-13536990
]
Tom White commented on YARN-103:
Some feedback on the current patch:
* Why not use the existing ResourceRequest rather than creating a new type
ContainerRequest? Having two equivalent types seems confusing.
* Is the ANY constant meant to be used by users? It looks like you specify a
ContainerRequest with null hosts and racks in this case. If so, then it would
be useful to add a constructor that doesn't take hosts or racks for that case,
although given my previous point it would be easier to use ResourceRequest.
* {{assertTrue(containersRequestedRack == 2);}} appears on two successive
lines. I think the assertion should be about containersRequestedAny.
* // do a few iterations to ensure RM is not going send new containers - wait
in the loop to allow NMs to heartbeat?
Add a yarn AM - RM client module
Key: YARN-103
URL: https://issues.apache.org/jira/browse/YARN-103
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Bikas Saha
Assignee: Bikas Saha
Attachments: YARN-103.1.patch, YARN-103.2.patch, YARN-103.3.patch,
YARN-103.4.patch, YARN-103.4.wrapper.patch, YARN-103.5.patch,
YARN-103.6.patch, YARN-103.7.patch
Add a basic client wrapper library to the AM RM protocol in order to prevent
proliferation of code being duplicated everywhere. Provide helper functions
to perform reverse mapping of container requests to RM allocation resource
request table format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-223) Change processTree interface to work better with native code
[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537010#comment-13537010 ] Hudson commented on YARN-223: - Integrated in Hadoop-Hdfs-trunk #1260 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1260/]) YARN-223. Update process tree instead of getting new process trees. (Radim Kolar via llu) (Revision 1424244) Result = FAILURE llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1424244 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java Change processTree interface to work better with native code Key: YARN-223 URL: https://issues.apache.org/jira/browse/YARN-223 Project: Hadoop YARN Issue Type: Bug Reporter: Radim Kolar Assignee: Radim Kolar Priority: Critical Fix For: 3.0.0, 2.0.3-alpha Attachments: pstree-update4.txt, pstree-update6.txt, pstree-update6.txt Problem is that on every update of processTree new object is required. This is undesired when working with processTree implementation in native code. replace ProcessTree.getProcessTree() with updateProcessTree(). No new object allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-50) Implement renewal / cancellation of Delegation Tokens
[
https://issues.apache.org/jira/browse/YARN-50?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537075#comment-13537075
]
Daryn Sharp commented on YARN-50:
-
A few minor questions/comments:
# Should the ctors for {{RenewDelegationTokenRequestPBImpl(proto}} and
{{RenewDelegationTokenResponsePBImpl(proto)}} discard the given proto? I'm not
fluent in the protobuf implementation.
# Very minor, a comment in {{YarnClientImpl}} has misspelled ClietnRMProtocol
# Why do HS parameters need to be in RM config? It should be sufficient to
have MR in the class path to pickup the token renewers.
Issues:
# In {{RMDelegationTokenRenewer}}:
## The yarn client should connect to the service address within the token,
rather than assume all RM tokens are for the local RM. It's possible that a
job might have a RM token for a different cluster.
## Are null checks on yarnClient within the try/finally needed? If
{{getYarnClient}} can return null, NPEs may be generated on the renew/cancel
calls which would imply the conditional should replace the try block, or the
conditional should be removed.
# {{ClientRMService#renewDelegationToken/cancelDelegationToken}} throw an
exception when the connection is token based, which is correct, but the error
message says tokens cannot be issued instead of cancelled/renewed.
# Based on the SASL work I'm doing, {{isAllowedDelegationTokenOp}} should be
more like this: {{return
UserGroupInformation.getCurrentUser().getRealAuthenticationMethod() !=
AuthenticationMethod.TOKEN}}. (Aside: I've actually been meaning to push this
check down into the ADTSM but I'll do that on another jira)
Implement renewal / cancellation of Delegation Tokens
-
Key: YARN-50
URL: https://issues.apache.org/jira/browse/YARN-50
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Attachments: YARN-50_wip.txt
Currently, delegation tokens issues by the RM and History server cannot be
renewed or cancelled. This needs to be implemented.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-279) Generalize RM token management
Daryn Sharp created YARN-279: Summary: Generalize RM token management Key: YARN-279 URL: https://issues.apache.org/jira/browse/YARN-279 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Affects Versions: 0.23.5, 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Token renewal/cancelation in the RM presents challenges to support arbitrary tokens. The RM's CLASSPATH is currently required to have token renewer classes and all associated classes for the project's client. The logistics of having installs on the RM of all hadoop projects that submit jobs - just to support client connections to renew/cancel tokens - are untenable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-279) Generalize RM token management
[ https://issues.apache.org/jira/browse/YARN-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537148#comment-13537148 ] Daryn Sharp commented on YARN-279: -- The current token management (renew/cancel) has contaminated the RM with specific knowledge of other systems. Unfortunately the token management cannot be pushed down to the AM because a token may expire before the RM launches the job. Also, the AM would suffer from the same problem of needing other project's jars just to manage the tokens. For the sake of discussion, the implementation of a token's management should be cleanly abstracted and removed from the RM. One approach would be to require resources with tokens to provide a web service that implements renew and cancel. The RM's renew timers then only call a REST api to manage the tokens. A token would only need to contain a base uri for the REST call. Thoughts? Generalize RM token management -- Key: YARN-279 URL: https://issues.apache.org/jira/browse/YARN-279 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Affects Versions: 3.0.0, 2.0.0-alpha, 0.23.5 Reporter: Daryn Sharp Token renewal/cancelation in the RM presents challenges to support arbitrary tokens. The RM's CLASSPATH is currently required to have token renewer classes and all associated classes for the project's client. The logistics of having installs on the RM of all hadoop projects that submit jobs - just to support client connections to renew/cancel tokens - are untenable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-223) Change processTree interface to work better with native code
[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated YARN-223: --- Fix Version/s: 0.23.6 I pulled this into branch-0.23 Change processTree interface to work better with native code Key: YARN-223 URL: https://issues.apache.org/jira/browse/YARN-223 Project: Hadoop YARN Issue Type: Bug Reporter: Radim Kolar Assignee: Radim Kolar Priority: Critical Fix For: 3.0.0, 2.0.3-alpha, 0.23.6 Attachments: pstree-update4.txt, pstree-update6.txt, pstree-update6.txt Problem is that on every update of processTree new object is required. This is undesired when working with processTree implementation in native code. replace ProcessTree.getProcessTree() with updateProcessTree(). No new object allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-223) Change processTree interface to work better with native code
[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537165#comment-13537165 ] Luke Lu commented on YARN-223: -- Sorry folks! I got too used to seeing mapreduce code in YARN :) Change processTree interface to work better with native code Key: YARN-223 URL: https://issues.apache.org/jira/browse/YARN-223 Project: Hadoop YARN Issue Type: Bug Reporter: Radim Kolar Assignee: Radim Kolar Priority: Critical Fix For: 3.0.0, 2.0.3-alpha, 0.23.6 Attachments: pstree-update4.txt, pstree-update6.txt, pstree-update6.txt Problem is that on every update of processTree new object is required. This is undesired when working with processTree implementation in native code. replace ProcessTree.getProcessTree() with updateProcessTree(). No new object allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-223) Change processTree interface to work better with native code
[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537166#comment-13537166 ] Luke Lu commented on YARN-223: -- I think the PreCommit test patch system should either handle cross project patches or outright reject such patches. Change processTree interface to work better with native code Key: YARN-223 URL: https://issues.apache.org/jira/browse/YARN-223 Project: Hadoop YARN Issue Type: Bug Reporter: Radim Kolar Assignee: Radim Kolar Priority: Critical Fix For: 3.0.0, 2.0.3-alpha, 0.23.6 Attachments: pstree-update4.txt, pstree-update6.txt, pstree-update6.txt Problem is that on every update of processTree new object is required. This is undesired when working with processTree implementation in native code. replace ProcessTree.getProcessTree() with updateProcessTree(). No new object allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-276) Capacity Scheduler can hang when submit many jobs concurrently
[ https://issues.apache.org/jira/browse/YARN-276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537168#comment-13537168 ] Robert Joseph Evans commented on YARN-276: -- I am not an expert on the scheduler code, so I have not done an in depth review of the patch. My biggest concern with this is that there is no visibility in the UI/web services about why an app may not have been scheduled. It would be great if you could update CapacitySchedulerLeafQueueInfo.java and the web page that uses it CapacitySchedulerPage.java. Capacity Scheduler can hang when submit many jobs concurrently -- Key: YARN-276 URL: https://issues.apache.org/jira/browse/YARN-276 Project: Hadoop YARN Issue Type: Bug Components: capacityscheduler Affects Versions: 3.0.0, 2.0.1-alpha Reporter: nemon lou Attachments: YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch Original Estimate: 24h Remaining Estimate: 24h In hadoop2.0.1,When i submit many jobs concurrently at the same time,Capacity scheduler can hang with most resources taken up by AM and don't have enough resources for tasks.And then all applications hang there. The cause is that yarn.scheduler.capacity.maximum-am-resource-percent not check directly.Instead ,this property only used for maxActiveApplications. And maxActiveApplications is computed by minimumAllocation (not by Am actually used). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-280) RM does not reject app submission with invalid tokens
Daryn Sharp created YARN-280: Summary: RM does not reject app submission with invalid tokens Key: YARN-280 URL: https://issues.apache.org/jira/browse/YARN-280 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 0.23.5, 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp The RM will launch an app with invalid tokens. The tasks will languish with failed connection retries, followed by task reattempts, followed by app reattempts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-280) RM does not reject app submission with invalid tokens
[ https://issues.apache.org/jira/browse/YARN-280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated YARN-280: - Attachment: YARN-280.patch Instead of scheduling tokens for immediate renewal in the background, the first renewal happens in the current thread so an invalid token will fail the app submission. Once all tokens are known to be valid, they are scheduled for renewal. RM does not reject app submission with invalid tokens - Key: YARN-280 URL: https://issues.apache.org/jira/browse/YARN-280 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 3.0.0, 2.0.0-alpha, 0.23.5 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: YARN-280.patch The RM will launch an app with invalid tokens. The tasks will languish with failed connection retries, followed by task reattempts, followed by app reattempts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Moved] (YARN-281) Add a test for YARN Schedulers' MAXIMUM_ALLOCATION limits
[ https://issues.apache.org/jira/browse/YARN-281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J moved MAPREDUCE-4171 to YARN-281: - Component/s: (was: mrv2) (was: test) scheduler Affects Version/s: (was: 2.0.0-alpha) 2.0.0-alpha Key: YARN-281 (was: MAPREDUCE-4171) Project: Hadoop YARN (was: Hadoop Map/Reduce) Add a test for YARN Schedulers' MAXIMUM_ALLOCATION limits - Key: YARN-281 URL: https://issues.apache.org/jira/browse/YARN-281 Project: Hadoop YARN Issue Type: Test Components: scheduler Affects Versions: 2.0.0-alpha Reporter: Harsh J Assignee: Harsh J Labels: test We currently have tests that test MINIMUM_ALLOCATION limits for FifoScheduler and the likes, but no test for MAXIMUM_ALLOCATION yet. We should add a test to prevent regressions of any kind on such limits. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-278) Fair scheduler maxRunningApps config causes no apps to make progress
[ https://issues.apache.org/jira/browse/YARN-278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-278: Attachment: YARN-278.patch Fair scheduler maxRunningApps config causes no apps to make progress Key: YARN-278 URL: https://issues.apache.org/jira/browse/YARN-278 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager, scheduler Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-278.patch, YARN-278.patch This occurs because the scheduler erroneously chooses apps to offer resources to that are not runnable, then later decides they are not runnable, and doesn't try to give the resources to anyone else. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-282) Fair scheduler web UI double counts Apps Submitted
Sandy Ryza created YARN-282: --- Summary: Fair scheduler web UI double counts Apps Submitted Key: YARN-282 URL: https://issues.apache.org/jira/browse/YARN-282 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Each app submitted is reported twice under Apps Submitted -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-282) Fair scheduler web UI double counts Apps Submitted
[ https://issues.apache.org/jira/browse/YARN-282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-282: Attachment: YARN-282.patch Fair scheduler web UI double counts Apps Submitted -- Key: YARN-282 URL: https://issues.apache.org/jira/browse/YARN-282 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-282.patch Each app submitted is reported twice under Apps Submitted -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-278) Fair scheduler maxRunningApps config causes no apps to make progress
[
https://issues.apache.org/jira/browse/YARN-278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537291#comment-13537291
]
Hadoop QA commented on YARN-278:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12561959/YARN-278.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/241//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/241//console
This message is automatically generated.
Fair scheduler maxRunningApps config causes no apps to make progress
Key: YARN-278
URL: https://issues.apache.org/jira/browse/YARN-278
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager, scheduler
Affects Versions: 2.0.2-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-278.patch, YARN-278.patch
This occurs because the scheduler erroneously chooses apps to offer resources
to that are not runnable, then later decides they are not runnable, and
doesn't try to give the resources to anyone else.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-50) Implement renewal / cancellation of Delegation Tokens
[ https://issues.apache.org/jira/browse/YARN-50?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated YARN-50: --- Attachment: YARN-50.txt Thanks for the quick reviews Daryn and Bobby. Updated patch to address the issues. Since the service address is being used - MR config isn't really required (in the MR part of the patch). bq. Based on the SASL work I'm doing, isAllowedDelegationTokenOp should be more like this: return UserGroupInformation.getCurrentUser().getRealAuthenticationMethod() != AuthenticationMethod.TOKEN. (Aside: I've actually been meaning to push this check down into the ADTSM but I'll do that on another jira) For security related checks, I think an explicit whitelist is a better option. Have included KERBEROS, KERBEROS_SSL and CERTIFICATE. Feel free to change this. bq. The yarn client should connect to the service address within the token, rather than assume all RM tokens are for the local RM. It's possible that a job might have a RM token for a different cluster. I'm not sure this is a valid use case. Get a token from one RM, submit a job to RM2 with RM1s token, which then may submit back to RM1... both RMs end up renewing the token. IAC, using the service address to remove the dependency on the configs. Implement renewal / cancellation of Delegation Tokens - Key: YARN-50 URL: https://issues.apache.org/jira/browse/YARN-50 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Blocker Attachments: YARN-50.txt, YARN-50_wip.txt Currently, delegation tokens issues by the RM and History server cannot be renewed or cancelled. This needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-282) Fair scheduler web UI double counts Apps Submitted
[
https://issues.apache.org/jira/browse/YARN-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537308#comment-13537308
]
Hadoop QA commented on YARN-282:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12561966/YARN-282.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/242//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/242//console
This message is automatically generated.
Fair scheduler web UI double counts Apps Submitted
--
Key: YARN-282
URL: https://issues.apache.org/jira/browse/YARN-282
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.0.2-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-282.patch
Each app submitted is reported twice under Apps Submitted
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-50) Implement renewal / cancellation of Delegation Tokens
[
https://issues.apache.org/jira/browse/YARN-50?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537329#comment-13537329
]
Daryn Sharp commented on YARN-50:
-
I think it looks like my concerns are addressed. Only remark is that
{{ResourceManager}} appears to have some spurious imports added to it. It
looks like changes were made and then reverted except the imports?
Implement renewal / cancellation of Delegation Tokens
-
Key: YARN-50
URL: https://issues.apache.org/jira/browse/YARN-50
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Priority: Blocker
Attachments: YARN-50.txt, YARN-50_wip.txt
Currently, delegation tokens issues by the RM and History server cannot be
renewed or cancelled. This needs to be implemented.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-50) Implement renewal / cancellation of Delegation Tokens
[
https://issues.apache.org/jira/browse/YARN-50?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537335#comment-13537335
]
Daryn Sharp commented on YARN-50:
-
Actually, it looks like {{CancelDelegationTokenResponsePBImpl(port)}} should be
setting viaProto = true?
Implement renewal / cancellation of Delegation Tokens
-
Key: YARN-50
URL: https://issues.apache.org/jira/browse/YARN-50
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Priority: Blocker
Attachments: YARN-50.txt, YARN-50_wip.txt
Currently, delegation tokens issues by the RM and History server cannot be
renewed or cancelled. This needs to be implemented.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-283) Fair scheduler fails to get queue info without root prefix
Sandy Ryza created YARN-283: --- Summary: Fair scheduler fails to get queue info without root prefix Key: YARN-283 URL: https://issues.apache.org/jira/browse/YARN-283 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza If queue1 exists, and a client calls mapred queue -info queue1, an exception is thrown. If they use root.queue1, it works correctly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-283) Fair scheduler fails to get queue info without root prefix
[ https://issues.apache.org/jira/browse/YARN-283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-283: Attachment: YARN-283.patch Fair scheduler fails to get queue info without root prefix -- Key: YARN-283 URL: https://issues.apache.org/jira/browse/YARN-283 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-283.patch If queue1 exists, and a client calls mapred queue -info queue1, an exception is thrown. If they use root.queue1, it works correctly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-283) Fair scheduler fails to get queue info without root prefix
[ https://issues.apache.org/jira/browse/YARN-283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537446#comment-13537446 ] Sandy Ryza commented on YARN-283: - Tested manually that it works. Fair scheduler fails to get queue info without root prefix -- Key: YARN-283 URL: https://issues.apache.org/jira/browse/YARN-283 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-283.patch If queue1 exists, and a client calls mapred queue -info queue1, an exception is thrown. If they use root.queue1, it works correctly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-283) Fair scheduler fails to get queue info without root prefix
[
https://issues.apache.org/jira/browse/YARN-283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537457#comment-13537457
]
Hadoop QA commented on YARN-283:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12561998/YARN-283.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/243//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/243//console
This message is automatically generated.
Fair scheduler fails to get queue info without root prefix
--
Key: YARN-283
URL: https://issues.apache.org/jira/browse/YARN-283
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.2-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-283.patch
If queue1 exists, and a client calls mapred queue -info queue1, an
exception is thrown. If they use root.queue1, it works correctly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-142) Change YARN APIs to throw IOException
[ https://issues.apache.org/jira/browse/YARN-142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-142: --- Attachment: YARN-142.1.patch Use IOException to replace the YarnRemoteException Change YARN APIs to throw IOException - Key: YARN-142 URL: https://issues.apache.org/jira/browse/YARN-142 Project: Hadoop YARN Issue Type: Bug Affects Versions: 0.23.3, 2.0.0-alpha Reporter: Siddharth Seth Assignee: Xuan Gong Priority: Critical Attachments: YARN-142.1.patch Ref: MAPREDUCE-4067 All YARN APIs currently throw YarnRemoteException. 1) This cannot be extended in it's current form. 2) The RPC layer can throw IOExceptions. These end up showing up as UndeclaredThrowableExceptions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-283) Fair scheduler fails to get queue info without root prefix
[ https://issues.apache.org/jira/browse/YARN-283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537537#comment-13537537 ] Karthik Kambatla commented on YARN-283: --- +1 Fair scheduler fails to get queue info without root prefix -- Key: YARN-283 URL: https://issues.apache.org/jira/browse/YARN-283 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-283.patch If queue1 exists, and a client calls mapred queue -info queue1, an exception is thrown. If they use root.queue1, it works correctly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-276) Capacity Scheduler can hang when submit many jobs concurrently
[ https://issues.apache.org/jira/browse/YARN-276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537643#comment-13537643 ] nemon lou commented on YARN-276: Good idea ,Robert.Thank you for your comment. I think it's good to display AM used resources and AM percent limit(or max resources that AMs can use) for each leaf queue on capacity scheduler page. Capacity Scheduler can hang when submit many jobs concurrently -- Key: YARN-276 URL: https://issues.apache.org/jira/browse/YARN-276 Project: Hadoop YARN Issue Type: Bug Components: capacityscheduler Affects Versions: 3.0.0, 2.0.1-alpha Reporter: nemon lou Attachments: YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch Original Estimate: 24h Remaining Estimate: 24h In hadoop2.0.1,When i submit many jobs concurrently at the same time,Capacity scheduler can hang with most resources taken up by AM and don't have enough resources for tasks.And then all applications hang there. The cause is that yarn.scheduler.capacity.maximum-am-resource-percent not check directly.Instead ,this property only used for maxActiveApplications. And maxActiveApplications is computed by minimumAllocation (not by Am actually used). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Moved] (YARN-284) YARN capacity scheduler doesn't spread MR tasks evenly on an underutilized cluster
[ https://issues.apache.org/jira/browse/YARN-284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J moved MAPREDUCE-3268 to YARN-284: - Component/s: (was: scheduler) scheduler Affects Version/s: (was: 0.23.0) 2.0.0-alpha Key: YARN-284 (was: MAPREDUCE-3268) Project: Hadoop YARN (was: Hadoop Map/Reduce) YARN capacity scheduler doesn't spread MR tasks evenly on an underutilized cluster -- Key: YARN-284 URL: https://issues.apache.org/jira/browse/YARN-284 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.0.0-alpha Reporter: Todd Lipcon The fair scheduler in MR1 has the behavior that, if a job is submitted to an under-utilized cluster and the cluster has more open slots than tasks in the job, the tasks are spread evenly throughout the cluster. This improves job latency since more spindles and NICs are utilized to complete the job. In MR2 I see this issue causing significantly longer job runtimes when there is excess capacity in the cluster -- especially on reducers which sometimmes end up clumping together on a smaller set of nodes which then become disk/network constrained. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-285) RM should be able to provide a tracking link for apps that have already been purged
Derek Dagit created YARN-285: Summary: RM should be able to provide a tracking link for apps that have already been purged Key: YARN-285 URL: https://issues.apache.org/jira/browse/YARN-285 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 0.23.5, 3.0.0 Reporter: Derek Dagit Assignee: Derek Dagit As applications complete, the RM tracks their IDs in a completed list. This list is routinely truncated to limit the total number of application remembered by the RM. When a user clicks the History for a job, either the browser is redirected to the application's tracking link obtained from the stored application instance. But when the application has been purged from the RM, an error is displayed. In very busy clusters the rate at which applications complete can cause applications to be purged from the RM's internal list within hours, which breaks the proxy URLs users have saved for their jobs. We would like the RM to provide valid tracking links persist so that users are not frustrated by broken links. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-285) RM should be able to provide a tracking link for apps that have already been purged
[ https://issues.apache.org/jira/browse/YARN-285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Dagit updated YARN-285: - Attachment: YARN-285-branch-0.23.patch RM should be able to provide a tracking link for apps that have already been purged --- Key: YARN-285 URL: https://issues.apache.org/jira/browse/YARN-285 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 3.0.0, 0.23.5 Reporter: Derek Dagit Assignee: Derek Dagit Attachments: YARN-285-branch-0.23.patch As applications complete, the RM tracks their IDs in a completed list. This list is routinely truncated to limit the total number of application remembered by the RM. When a user clicks the History for a job, either the browser is redirected to the application's tracking link obtained from the stored application instance. But when the application has been purged from the RM, an error is displayed. In very busy clusters the rate at which applications complete can cause applications to be purged from the RM's internal list within hours, which breaks the proxy URLs users have saved for their jobs. We would like the RM to provide valid tracking links persist so that users are not frustrated by broken links. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-285) RM should be able to provide a tracking link for apps that have already been purged
[ https://issues.apache.org/jira/browse/YARN-285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Dagit updated YARN-285: - Attachment: YARN-285.patch This is a patch that introduces handling for an optional tracking URI plugin, specified via a new config, that should accept a given ApplicationId and return a valid URI. The plugin is used in cases where a proxy link is requested, but the RM has already purged the given application. The plugin will use the ApplicationId to rewrite the URL, and the browser will be redirected just as when the RM uses the tracking URL of a application the it knows about. The intention of the plugin instead of a simple URL rewrite is to keep architectural separation between Yarn and MapReduce. The mechanism is similar to that involving the ResourceCalculatorPlugin. RM should be able to provide a tracking link for apps that have already been purged --- Key: YARN-285 URL: https://issues.apache.org/jira/browse/YARN-285 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 3.0.0, 0.23.5 Reporter: Derek Dagit Assignee: Derek Dagit Attachments: YARN-285-branch-0.23.patch, YARN-285.patch As applications complete, the RM tracks their IDs in a completed list. This list is routinely truncated to limit the total number of application remembered by the RM. When a user clicks the History for a job, either the browser is redirected to the application's tracking link obtained from the stored application instance. But when the application has been purged from the RM, an error is displayed. In very busy clusters the rate at which applications complete can cause applications to be purged from the RM's internal list within hours, which breaks the proxy URLs users have saved for their jobs. We would like the RM to provide valid tracking links persist so that users are not frustrated by broken links. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
