[jira] [Commented] (YARN-1567) In Fair Scheduler, allow empty queues to change between leaf and parent on allocation file reload
[ https://issues.apache.org/jira/browse/YARN-1567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869602#comment-13869602 ] Karthik Kambatla commented on YARN-1567: +1 In Fair Scheduler, allow empty queues to change between leaf and parent on allocation file reload - Key: YARN-1567 URL: https://issues.apache.org/jira/browse/YARN-1567 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-1567-1.patch, YARN-1567-2.patch, YARN-1567-3.patch, YARN-1567.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1592) backport YARN-957 to branch-0.23
Thomas Graves created YARN-1592: --- Summary: backport YARN-957 to branch-0.23 Key: YARN-1592 URL: https://issues.apache.org/jira/browse/YARN-1592 Project: Hadoop YARN Issue Type: Bug Components: capacityscheduler Affects Versions: 0.23.10 Reporter: Thomas Graves Assignee: Thomas Graves see YARN-957 for description. branch 0.23 has the same bug. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1592) backport YARN-957 to branch-0.23
[ https://issues.apache.org/jira/browse/YARN-1592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated YARN-1592: Attachment: YARN-1592.patch pretty straight forward backport. backport YARN-957 to branch-0.23 Key: YARN-1592 URL: https://issues.apache.org/jira/browse/YARN-1592 Project: Hadoop YARN Issue Type: Bug Components: capacityscheduler Affects Versions: 0.23.10 Reporter: Thomas Graves Assignee: Thomas Graves Attachments: YARN-1592.patch see YARN-957 for description. branch 0.23 has the same bug. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1592) backport YARN-957 to branch-0.23
[
https://issues.apache.org/jira/browse/YARN-1592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869627#comment-13869627
]
Hadoop QA commented on YARN-1592:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12622642/YARN-1592.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2879//console
This message is automatically generated.
backport YARN-957 to branch-0.23
Key: YARN-1592
URL: https://issues.apache.org/jira/browse/YARN-1592
Project: Hadoop YARN
Issue Type: Bug
Components: capacityscheduler
Affects Versions: 0.23.10
Reporter: Thomas Graves
Assignee: Thomas Graves
Attachments: YARN-1592.patch
see YARN-957 for description. branch 0.23 has the same bug.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1592) backport YARN-957 to branch-0.23
[ https://issues.apache.org/jira/browse/YARN-1592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869630#comment-13869630 ] Thomas Graves commented on YARN-1592: - jenkins fails since this patch is for branch-0.23 only. backport YARN-957 to branch-0.23 Key: YARN-1592 URL: https://issues.apache.org/jira/browse/YARN-1592 Project: Hadoop YARN Issue Type: Bug Components: capacityscheduler Affects Versions: 0.23.10 Reporter: Thomas Graves Assignee: Thomas Graves Attachments: YARN-1592.patch see YARN-957 for description. branch 0.23 has the same bug. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-888) clean up POM dependencies
[ https://issues.apache.org/jira/browse/YARN-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869658#comment-13869658 ] Alejandro Abdelnur commented on YARN-888: - test failure (timeout) seems unrelated. [~vinodkv], [~ste...@apache.org], [~kkambatl], [~rvs], over the weekend I've updated the patch with comments in the YARN non-leaf POM stating no dependencies should be added there. No other changes. Unless I hear further comments I'm planning to commit this later today to trunk and branch-2. Thanks for your reviews. clean up POM dependencies - Key: YARN-888 URL: https://issues.apache.org/jira/browse/YARN-888 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: YARN-888.patch, YARN-888.patch, YARN-888.patch, YARN-888.patch, yarn-888-2.patch Intermediate 'pom' modules define dependencies inherited by leaf modules. This is causing issues in intellij IDE. We should normalize the leaf modules like in common, hdfs and tools where all dependencies are defined in each leaf module and the intermediate 'pom' module do not define any dependency. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Comment Edited] (YARN-888) clean up POM dependencies
[ https://issues.apache.org/jira/browse/YARN-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869658#comment-13869658 ] Alejandro Abdelnur edited comment on YARN-888 at 1/13/14 5:22 PM: -- test failure (timeout) seems unrelated. [~vinodkv], [~ste...@apache.org], [~kkambatl], [~rvs], over the weekend I've updated the patch with comments in the YARN non-leaf POMs stating no dependencies should be added there. No other changes. Unless I hear further comments I'm planning to commit this later today to trunk and branch-2. Thanks for your reviews. was (Author: tucu00): test failure (timeout) seems unrelated. [~vinodkv], [~ste...@apache.org], [~kkambatl], [~rvs], over the weekend I've updated the patch with comments in the YARN non-leaf POM stating no dependencies should be added there. No other changes. Unless I hear further comments I'm planning to commit this later today to trunk and branch-2. Thanks for your reviews. clean up POM dependencies - Key: YARN-888 URL: https://issues.apache.org/jira/browse/YARN-888 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: YARN-888.patch, YARN-888.patch, YARN-888.patch, YARN-888.patch, yarn-888-2.patch Intermediate 'pom' modules define dependencies inherited by leaf modules. This is causing issues in intellij IDE. We should normalize the leaf modules like in common, hdfs and tools where all dependencies are defined in each leaf module and the intermediate 'pom' module do not define any dependency. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-888) clean up POM dependencies
[ https://issues.apache.org/jira/browse/YARN-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869714#comment-13869714 ] Steve Loughran commented on YARN-888: - --I've no objections, so +1 from me clean up POM dependencies - Key: YARN-888 URL: https://issues.apache.org/jira/browse/YARN-888 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: YARN-888.patch, YARN-888.patch, YARN-888.patch, YARN-888.patch, yarn-888-2.patch Intermediate 'pom' modules define dependencies inherited by leaf modules. This is causing issues in intellij IDE. We should normalize the leaf modules like in common, hdfs and tools where all dependencies are defined in each leaf module and the intermediate 'pom' module do not define any dependency. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-888) clean up POM dependencies
[ https://issues.apache.org/jira/browse/YARN-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869725#comment-13869725 ] Karthik Kambatla commented on YARN-888: --- Looks good to me. The test passes for me locally, and IntelliJ seems to like the patch. clean up POM dependencies - Key: YARN-888 URL: https://issues.apache.org/jira/browse/YARN-888 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: YARN-888.patch, YARN-888.patch, YARN-888.patch, YARN-888.patch, yarn-888-2.patch Intermediate 'pom' modules define dependencies inherited by leaf modules. This is causing issues in intellij IDE. We should normalize the leaf modules like in common, hdfs and tools where all dependencies are defined in each leaf module and the intermediate 'pom' module do not define any dependency. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-888) clean up POM dependencies
[ https://issues.apache.org/jira/browse/YARN-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869799#comment-13869799 ] Vinod Kumar Vavilapalli commented on YARN-888: -- Ran the tests yesterday but forgot to look at their result. YARN tests are passing fine. Including TestNMClient on my box. Checking this in now. clean up POM dependencies - Key: YARN-888 URL: https://issues.apache.org/jira/browse/YARN-888 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: YARN-888.patch, YARN-888.patch, YARN-888.patch, YARN-888.patch, yarn-888-2.patch Intermediate 'pom' modules define dependencies inherited by leaf modules. This is causing issues in intellij IDE. We should normalize the leaf modules like in common, hdfs and tools where all dependencies are defined in each leaf module and the intermediate 'pom' module do not define any dependency. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Comment Edited] (YARN-888) clean up POM dependencies
[ https://issues.apache.org/jira/browse/YARN-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869806#comment-13869806 ] Vinod Kumar Vavilapalli edited comment on YARN-888 at 1/13/14 6:35 PM: --- Committed this to trunk and branch-2. Thanks Alejandro! Let's pay close attention to any potential issues coming out of this. was (Author: vinodkv): Committed this to trunk and branch-2. Thanks Alenjandro! Let's pay close attention to any potential issues coming out of this. clean up POM dependencies - Key: YARN-888 URL: https://issues.apache.org/jira/browse/YARN-888 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.4.0 Attachments: YARN-888.patch, YARN-888.patch, YARN-888.patch, YARN-888.patch, yarn-888-2.patch Intermediate 'pom' modules define dependencies inherited by leaf modules. This is causing issues in intellij IDE. We should normalize the leaf modules like in common, hdfs and tools where all dependencies are defined in each leaf module and the intermediate 'pom' module do not define any dependency. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-888) clean up POM dependencies
[ https://issues.apache.org/jira/browse/YARN-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869825#comment-13869825 ] Alejandro Abdelnur commented on YARN-888: - Thanks [~vinodkv]. clean up POM dependencies - Key: YARN-888 URL: https://issues.apache.org/jira/browse/YARN-888 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.4.0 Attachments: YARN-888.patch, YARN-888.patch, YARN-888.patch, YARN-888.patch, yarn-888-2.patch Intermediate 'pom' modules define dependencies inherited by leaf modules. This is causing issues in intellij IDE. We should normalize the leaf modules like in common, hdfs and tools where all dependencies are defined in each leaf module and the intermediate 'pom' module do not define any dependency. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1410) Handle client failover during 2 step client API's like app submission
[ https://issues.apache.org/jira/browse/YARN-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869823#comment-13869823 ] Karthik Kambatla commented on YARN-1410: A slightly orthogonal question - the AtMostOnce/Idempotent annotations are honored only in the failover case and not when RM is restart. May be, we should fix up the RetryPolicy in RMProxy to use these annotations in shouldRetry. We should probably do this in a separate JIRA though. Handle client failover during 2 step client API's like app submission - Key: YARN-1410 URL: https://issues.apache.org/jira/browse/YARN-1410 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Xuan Gong Attachments: YARN-1410-outline.patch, YARN-1410.1.patch, YARN-1410.2.patch, YARN-1410.2.patch, YARN-1410.3.patch Original Estimate: 48h Remaining Estimate: 48h App submission involves 1) creating appId 2) using that appId to submit an ApplicationSubmissionContext to the user. The client may have obtained an appId from an RM, the RM may have failed over, and the client may submit the app to the new RM. Since the new RM has a different notion of cluster timestamp (used to create app id) the new RM may reject the app submission resulting in unexpected failure on the client side. The same may happen for other 2 step client API operations. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-888) clean up POM dependencies
[ https://issues.apache.org/jira/browse/YARN-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869821#comment-13869821 ] Hudson commented on YARN-888: - SUCCESS: Integrated in Hadoop-trunk-Commit #4992 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4992/]) YARN-888. Cleaned up POM files so that non-leaf modules don't include any dependencies and thus compact the dependency list for leaf modules. Contributed by Alejandro Abdelnur. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1557801) * /hadoop/common/trunk/hadoop-project/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/pom.xml clean up POM dependencies - Key: YARN-888 URL: https://issues.apache.org/jira/browse/YARN-888 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.4.0 Attachments: YARN-888.patch, YARN-888.patch, YARN-888.patch, YARN-888.patch, yarn-888-2.patch Intermediate 'pom' modules define dependencies inherited by leaf modules. This is causing issues in intellij IDE. We should normalize the leaf modules like in common, hdfs and tools where all dependencies are defined in each leaf module and the intermediate 'pom' module do not define any dependency. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-888) clean up POM dependencies
[ https://issues.apache.org/jira/browse/YARN-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869843#comment-13869843 ] Alejandro Abdelnur commented on YARN-888: - OK, it seems we have our first victim, MAPREDUCE-5722. I don't know why this didn't come up before. Taking care of it right now. clean up POM dependencies - Key: YARN-888 URL: https://issues.apache.org/jira/browse/YARN-888 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.4.0 Attachments: YARN-888.patch, YARN-888.patch, YARN-888.patch, YARN-888.patch, yarn-888-2.patch Intermediate 'pom' modules define dependencies inherited by leaf modules. This is causing issues in intellij IDE. We should normalize the leaf modules like in common, hdfs and tools where all dependencies are defined in each leaf module and the intermediate 'pom' module do not define any dependency. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-888) clean up POM dependencies
[ https://issues.apache.org/jira/browse/YARN-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869856#comment-13869856 ] Alejandro Abdelnur commented on YARN-888: - False alarm, it seems i had stale POMs in my local Maven cache, still we need to take care of MAPREDUCE-5362 (the equiv of this JIRA for MR). clean up POM dependencies - Key: YARN-888 URL: https://issues.apache.org/jira/browse/YARN-888 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.4.0 Attachments: YARN-888.patch, YARN-888.patch, YARN-888.patch, YARN-888.patch, yarn-888-2.patch Intermediate 'pom' modules define dependencies inherited by leaf modules. This is causing issues in intellij IDE. We should normalize the leaf modules like in common, hdfs and tools where all dependencies are defined in each leaf module and the intermediate 'pom' module do not define any dependency. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1515) Ability to dump the container threads and stop the containers in a single RPC
[ https://issues.apache.org/jira/browse/YARN-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869853#comment-13869853 ] Gera Shegalov commented on YARN-1515: - [~vinodkv], I am interested in your feedback in the context of your comment on MAPREDUCE-5044 . Ability to dump the container threads and stop the containers in a single RPC - Key: YARN-1515 URL: https://issues.apache.org/jira/browse/YARN-1515 Project: Hadoop YARN Issue Type: New Feature Components: api, nodemanager Reporter: Gera Shegalov Assignee: Gera Shegalov Attachments: YARN-1515.v01.patch, YARN-1515.v02.patch This is needed to implement MAPREDUCE-5044 to enable thread diagnostics for timed-out task attempts. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1506) Replace set resource change on RMNode/SchedulerNode directly with event notification.
[
https://issues.apache.org/jira/browse/YARN-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869882#comment-13869882
]
Bikas Saha commented on YARN-1506:
--
Thanks for the change. The code looks a lot cleaner now with a bunch on changes
removed.
ADMIN_RESOURCE_UPDATE instead of RESOURCE_UPDATE for the enum would help
clarify that its a forced admin update.
Why not update the total capability here also (like we do for non-running
node). When the node reports back as healthy then we would probably need the
new resource value, right?
{code}
+public void transition(RMNodeImpl rmNode, RMNodeEvent event) {
+ // The node is not usable, only log a warn message
+ LOG.warn(Try to update resource on a + rmNode.getState().toString() +
+ node: +rmNode.toString());
{code}
Why are we doing this indirect subtraction via delta instead of simply
clusterResource-=old; clusterResource+=new. Its the same number of operations
and less confusing to read.
{code}+Resource deltaResource = Resources.subtract(newResource,
oldResource);
+
+// update resource to node
+node.setTotalResource(newResource);
+
+// update resource to clusterResource
+Resources.addTo(clusterResource, deltaResource);{code}
Newly added synchronization? Sometimes getters are deliberately made lockless.
I hope this was not the case here.
{code}
- public Resource getTotalResource() {
+ public synchronized Resource getTotalResource() {
{code}
I think its crucial to have a more complete test (maybe using mockRM) that
verifies the flow from admin service to the scheduler. Most interesting would
be the case when the node is full allocated and then an update reduces the
capacity. Thus resulting in -ve value of available resource on the node. I am
wary that this case may have bugs in handling the -ve value in existing
scheduler code because its unexpected. Its fine for the test to use the default
scheduler.
Replace set resource change on RMNode/SchedulerNode directly with event
notification.
-
Key: YARN-1506
URL: https://issues.apache.org/jira/browse/YARN-1506
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager, scheduler
Reporter: Junping Du
Assignee: Junping Du
Priority: Blocker
Attachments: YARN-1506-v1.patch, YARN-1506-v2.patch,
YARN-1506-v3.patch, YARN-1506-v4.patch, YARN-1506-v5.patch, YARN-1506-v6.patch
According to Vinod's comments on YARN-312
(https://issues.apache.org/jira/browse/YARN-312?focusedCommentId=13846087page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13846087),
we should replace RMNode.setResourceOption() with some resource change event.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1551) Allow user-specified reason for killApplication
[ https://issues.apache.org/jira/browse/YARN-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869858#comment-13869858 ] Gera Shegalov commented on YARN-1551: - [~kkambatl], [~vinodkv], do you have any additional comments? Allow user-specified reason for killApplication --- Key: YARN-1551 URL: https://issues.apache.org/jira/browse/YARN-1551 Project: Hadoop YARN Issue Type: Improvement Reporter: Gera Shegalov Assignee: Gera Shegalov Attachments: YARN-1551.v01.patch, YARN-1551.v02.patch, YARN-1551.v03.patch This completes MAPREDUCE-5648 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1410) Handle client failover during 2 step client API's like app submission
[
https://issues.apache.org/jira/browse/YARN-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869932#comment-13869932
]
Bikas Saha commented on YARN-1410:
--
Xuan, can you please verify Karthik's comment above and fix/file jira?
For all follow up jiras, please mention relevant jiras in comments in the code
for other peoples benefit.
Why is this at most once? This could be retried any number of times until the
RM gives us a new application id, right?
{code}+ @AtMostOnce
public GetNewApplicationResponse getNewApplication({code}
In the common case, this extra operation is going to be pure overhead. Can we
get this information via an exception in the submitApplication() method?
{code}-rmClient.submitApplication(request);
+// check whether the applicationId is present or not
+// before we submit the application
+try {
+ getApplicationReport(applicationId);
+ String message = Application with id + applicationId +
+ is already present! Cannot add a duplicate!;
+ LOG.error(message);
+ throw new YarnException(message);
+} catch (ApplicationNotFoundException ex) {
+ // The applicationId is not present.
+ // submit the application with this applicationId
+ rmClient.submitApplication(request);
+}{code}
Can we create a common yarn client instead of doing it in every
createApplication()/submitApplication() helper method call?
{code}+ private ApplicationId createApplication() {
+int numRetries = 3;
+ApplicationId appId = null;
+while (numRetries-- 0) {
+ Configuration conf = new YarnConfiguration(this.conf);
+ YarnClient client = YarnClient.createYarnClient();{code}
Better name for test or some comments that it is testing the case when appId is
created by previous RM but the app is not saved before failover. So the RM
accepts the old Id.
Can it happen that submitApplication() will fail on the client but the RM has
actually saved the application (with or without failover)? What should we do in
that case?
Why are we removing this code. We should return a new exception for the client.
{code}-// but it is good to fail the invalid submission as early as
possible.
+// The duplication has been checked before application submission
if (rmContext.getRMApps().get(applicationId) != null) {
- String message = Application with id + applicationId +
- is already present! Cannot add a duplicate!;
- LOG.warn(message);
- RMAuditLogger.logFailure(user, AuditConstants.SUBMIT_APP_REQUEST,
- message, ClientRMService, Exception in submitting application,
- applicationId);
- throw RPCUtil.getRemoteException(message);{code}
What are the changes in TestYarnClient for? I dont see any new test added.
There is no testcase for the new functionality of YarnClient where it creates
an appId if its not supplied by the user? If you want, we could do it a
separate jira related to this jira.
Handle client failover during 2 step client API's like app submission
-
Key: YARN-1410
URL: https://issues.apache.org/jira/browse/YARN-1410
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Xuan Gong
Attachments: YARN-1410-outline.patch, YARN-1410.1.patch,
YARN-1410.2.patch, YARN-1410.2.patch, YARN-1410.3.patch
Original Estimate: 48h
Remaining Estimate: 48h
App submission involves
1) creating appId
2) using that appId to submit an ApplicationSubmissionContext to the user.
The client may have obtained an appId from an RM, the RM may have failed
over, and the client may submit the app to the new RM.
Since the new RM has a different notion of cluster timestamp (used to create
app id) the new RM may reject the app submission resulting in unexpected
failure on the client side.
The same may happen for other 2 step client API operations.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1525) Web UI should redirect to active RM when HA is enabled.
[ https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869939#comment-13869939 ] Karthik Kambatla commented on YARN-1525: I have very limited experience with the RM web services. I like the redirect, can we redirect for everything but (1) the REST API for ClusterInfo (/ws/v1/cluster/info), and may be (2) About (/cluster/cluster)? Web UI should redirect to active RM when HA is enabled. --- Key: YARN-1525 URL: https://issues.apache.org/jira/browse/YARN-1525 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Cindy Li When failover happens, web UI should redirect to the current active rm. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1399) Allow users to annotate an application with multiple tags
[ https://issues.apache.org/jira/browse/YARN-1399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869940#comment-13869940 ] Alejandro Abdelnur commented on YARN-1399: -- IMO, we should stick to the current API as Sandy suggest, it is a bit unfortunate the default being ALL instead of OWN but, well, what to do? Allow users to annotate an application with multiple tags - Key: YARN-1399 URL: https://issues.apache.org/jira/browse/YARN-1399 Project: Hadoop YARN Issue Type: Improvement Reporter: Zhijie Shen Nowadays, when submitting an application, users can fill the applicationType field to facilitate searching it later. IMHO, it's good to accept multiple tags to allow users to describe their applications in multiple aspects, including the application type. Then, searching by tags may be more efficient for users to reach their desired application collection. It's pretty much like the tag system of online photo/video/music and etc. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1567) In Fair Scheduler, allow empty queues to change between leaf and parent on allocation file reload
[ https://issues.apache.org/jira/browse/YARN-1567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869960#comment-13869960 ] Sandy Ryza commented on YARN-1567: -- Will commit this later today if there are no objections In Fair Scheduler, allow empty queues to change between leaf and parent on allocation file reload - Key: YARN-1567 URL: https://issues.apache.org/jira/browse/YARN-1567 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-1567-1.patch, YARN-1567-2.patch, YARN-1567-3.patch, YARN-1567.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1399) Allow users to annotate an application with multiple tags
[ https://issues.apache.org/jira/browse/YARN-1399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869959#comment-13869959 ] Alejandro Abdelnur commented on YARN-1399: -- In MR-land when submitting a job we can specify VIEW/MODIFY ACLs. It seems that in Yarn-land this is not possible for AMs. If I'm right with this, it seems like a missing functionality would naturally scope down what is returned by getApplications. And we could do that with in a backwards compatible way. Allow users to annotate an application with multiple tags - Key: YARN-1399 URL: https://issues.apache.org/jira/browse/YARN-1399 Project: Hadoop YARN Issue Type: Improvement Reporter: Zhijie Shen Nowadays, when submitting an application, users can fill the applicationType field to facilitate searching it later. IMHO, it's good to accept multiple tags to allow users to describe their applications in multiple aspects, including the application type. Then, searching by tags may be more efficient for users to reach their desired application collection. It's pretty much like the tag system of online photo/video/music and etc. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-914) Support graceful decommission of nodemanager
[ https://issues.apache.org/jira/browse/YARN-914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870080#comment-13870080 ] Ming Ma commented on YARN-914: -- Junping/Luke, have you looked into the checkpointing framework being done to support preemption? One possible design to support this scenario could be something like: 1. Drain NM with a timeout. When NM is being drained, no more tasks will be assigned to this node. 2. After the timeout, RM - AM - tasks checkpointing will kick in. Task state and application-level state such as map outputs will be preserved; tasks will be rescheduled to other nodes. Support graceful decommission of nodemanager Key: YARN-914 URL: https://issues.apache.org/jira/browse/YARN-914 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Luke Lu Assignee: Junping Du When NMs are decommissioned for non-fault reasons (capacity change etc.), it's desirable to minimize the impact to running applications. Currently if a NM is decommissioned, all running containers on the NM need to be rescheduled on other NMs. Further more, for finished map tasks, if their map output are not fetched by the reducers of the job, these map tasks will need to be rerun as well. We propose to introduce a mechanism to optionally gracefully decommission a node manager. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1410) Handle client failover during 2 step client API's like app submission
[ https://issues.apache.org/jira/browse/YARN-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870094#comment-13870094 ] Xuan Gong commented on YARN-1410: - bq. A slightly orthogonal question - the AtMostOnce/Idempotent annotations are honored only in the failover case and not when RM is restart. May be, we should fix up the RetryPolicy in RMProxy to use these annotations in shouldRetry. We should probably do this in a separate JIRA though. Yes. RM restart did not use the AtMostOnce/Idempotent annotation. But, I am not sure why we need to use these annotation in RM restart. Currently, at RM restart, we use RetryPolicies.retryByException to get RetryPolicy (We only handle ConnectionException and IOException), and use RetryPolicies.retryUpToMaximumTimeWithFixedSleep to specify the RetryPolicy which will give the RetryDecision.RETRY. Also, we use DefaultFailoverProxyProvider as the Proxy provider. Those are not enough for covering the RM restart case ? Also, the AtMostOnce and Idempotent annotation are only used when RetryDecision is FAILOVER_AND_RETRY. So, this is another reason why we do not have them in RM restart case (For the RM restart, the valid RetryDecision is RETRY). Handle client failover during 2 step client API's like app submission - Key: YARN-1410 URL: https://issues.apache.org/jira/browse/YARN-1410 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Xuan Gong Attachments: YARN-1410-outline.patch, YARN-1410.1.patch, YARN-1410.2.patch, YARN-1410.2.patch, YARN-1410.3.patch Original Estimate: 48h Remaining Estimate: 48h App submission involves 1) creating appId 2) using that appId to submit an ApplicationSubmissionContext to the user. The client may have obtained an appId from an RM, the RM may have failed over, and the client may submit the app to the new RM. Since the new RM has a different notion of cluster timestamp (used to create app id) the new RM may reject the app submission resulting in unexpected failure on the client side. The same may happen for other 2 step client API operations. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1593) support out-of-proc AuxiliaryServices
Ming Ma created YARN-1593: - Summary: support out-of-proc AuxiliaryServices Key: YARN-1593 URL: https://issues.apache.org/jira/browse/YARN-1593 Project: Hadoop YARN Issue Type: Improvement Reporter: Ming Ma AuxiliaryServices such as ShuffleHandler currently run in the same process as NM. There are some benefits to host them in dedicated processes. 1. NM rolling restart. If we want to upgrade YARN , NM restart will force the ShuffleHandler restart. If ShuffleHandler runs as a separate process, ShuffleHandler can continue to run during NM restart. NM can reconnect the the running ShuffleHandler after restart. 2. Resource management. It is possible another type of AuxiliaryServices will be implemented. AuxiliaryServices are considered YARN application specific and could consume lots of resources. Running AuxiliaryServices in separate processes allow easier resource management. NM could potentially stop a specific AuxiliaryServices process from running if it consumes resource way above its allocation. Here are some high level ideas: 1. NM provides a hosting process for each AuxiliaryService. Existing AuxiliaryService API doesn't change. 2. The hosting process provides RPC server for AuxiliaryService proxy object inside NM to connect to. 3. When we rolling restart NM, the existing AuxiliaryService processes will continue to run. NM could reconnect to the running AuxiliaryService processes upon restart. 4. Policy and resource management of AuxiliaryServices. So far we don't have immediate need for this. AuxiliaryService could run inside a container and its resource utilization could be taken into account by RM and RM could consider a specific type of applications overutilize cluster resource. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1594) YARN-321 branch needs to be updated after YARN-888 pom changes
[ https://issues.apache.org/jira/browse/YARN-1594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1594: -- Attachment: YARN-1594.txt POM changes to the AHS module. Mostly copied from resourcemanager and getting rid of non-required modules. YARN-321 branch needs to be updated after YARN-888 pom changes -- Key: YARN-1594 URL: https://issues.apache.org/jira/browse/YARN-1594 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Attachments: YARN-1594.txt YARN-888 changed the pom structure. And so latest merge to trunk breaks YARN-321 branch. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1596) Javadoc failures on YARN-321 branch
Vinod Kumar Vavilapalli created YARN-1596: - Summary: Javadoc failures on YARN-321 branch Key: YARN-1596 URL: https://issues.apache.org/jira/browse/YARN-1596 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli There are some javadoc issues on YARN-321 branch. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1594) YARN-321 branch needs to be updated after YARN-888 pom changes
Vinod Kumar Vavilapalli created YARN-1594: - Summary: YARN-321 branch needs to be updated after YARN-888 pom changes Key: YARN-1594 URL: https://issues.apache.org/jira/browse/YARN-1594 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli YARN-888 changed the pom structure. And so latest merge to trunk breaks YARN-321 branch. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1596) Javadoc failures on YARN-321 branch
[ https://issues.apache.org/jira/browse/YARN-1596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1596: -- Attachment: YARN-1596.txt This should fix it.. Javadoc failures on YARN-321 branch --- Key: YARN-1596 URL: https://issues.apache.org/jira/browse/YARN-1596 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Attachments: YARN-1596.txt There are some javadoc issues on YARN-321 branch. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1595) Test failures on YARN-321 branch
Vinod Kumar Vavilapalli created YARN-1595: - Summary: Test failures on YARN-321 branch Key: YARN-1595 URL: https://issues.apache.org/jira/browse/YARN-1595 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli mvn test doesn't pass on YARN-321 branch anymore. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1597) FindBugs warnings on YARN-321 branch
Vinod Kumar Vavilapalli created YARN-1597: - Summary: FindBugs warnings on YARN-321 branch Key: YARN-1597 URL: https://issues.apache.org/jira/browse/YARN-1597 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Attachments: YARN-1597.txt There are a bunch of findBugs warnings on YARN-321 branch. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1597) FindBugs warnings on YARN-321 branch
[ https://issues.apache.org/jira/browse/YARN-1597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1597: -- Attachment: YARN-1597.txt This should fix it.. FindBugs warnings on YARN-321 branch Key: YARN-1597 URL: https://issues.apache.org/jira/browse/YARN-1597 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Attachments: YARN-1597.txt There are a bunch of findBugs warnings on YARN-321 branch. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1594) YARN-321 branch needs to be updated after YARN-888 pom changes
[ https://issues.apache.org/jira/browse/YARN-1594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870180#comment-13870180 ] Zhijie Shen commented on YARN-1594: --- mvn test fails to run in AHS sub-project. Some dependency may be missing. YARN-321 branch needs to be updated after YARN-888 pom changes -- Key: YARN-1594 URL: https://issues.apache.org/jira/browse/YARN-1594 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Attachments: YARN-1594.txt YARN-888 changed the pom structure. And so latest merge to trunk breaks YARN-321 branch. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1598) HA-related rmadmin commands don't work on a secure cluster
Karthik Kambatla created YARN-1598: -- Summary: HA-related rmadmin commands don't work on a secure cluster Key: YARN-1598 URL: https://issues.apache.org/jira/browse/YARN-1598 Project: Hadoop YARN Issue Type: Sub-task Components: client, resourcemanager Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Critical The HA-related commands like -getServiceState -checkHealth etc. don't work in a secure cluster. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1594) YARN-321 branch needs to be updated after YARN-888 pom changes
[ https://issues.apache.org/jira/browse/YARN-1594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1594: -- Attachment: YARN-1594-20140113.txt Tx for trying the patch Zhijie. Can you try this? mvn test passes for me now in the AHS module with this patch. YARN-321 branch needs to be updated after YARN-888 pom changes -- Key: YARN-1594 URL: https://issues.apache.org/jira/browse/YARN-1594 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Attachments: YARN-1594-20140113.txt, YARN-1594.txt YARN-888 changed the pom structure. And so latest merge to trunk breaks YARN-321 branch. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1496) Protocol additions to allow moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870207#comment-13870207 ] Sandy Ryza commented on YARN-1496: -- [~vinodkv], any further concerns with the patch? Protocol additions to allow moving apps between queues -- Key: YARN-1496 URL: https://issues.apache.org/jira/browse/YARN-1496 Project: Hadoop YARN Issue Type: Sub-task Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-1496-1.patch, YARN-1496-2.patch, YARN-1496-3.patch, YARN-1496-4.patch, YARN-1496-5.patch, YARN-1496-6.patch, YARN-1496.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1598) HA-related rmadmin commands don't work on a secure cluster
[ https://issues.apache.org/jira/browse/YARN-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870215#comment-13870215 ] Alejandro Abdelnur commented on YARN-1598: -- LTGM, +1 after jenkins. HA-related rmadmin commands don't work on a secure cluster -- Key: YARN-1598 URL: https://issues.apache.org/jira/browse/YARN-1598 Project: Hadoop YARN Issue Type: Sub-task Components: client, resourcemanager Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Critical Attachments: yarn-1598-1.patch The HA-related commands like -getServiceState -checkHealth etc. don't work in a secure cluster. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1399) Allow users to annotate an application with multiple tags
[ https://issues.apache.org/jira/browse/YARN-1399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870218#comment-13870218 ] Karthik Kambatla commented on YARN-1399: Looking at the all the code changes required - proto changes/ additions, new Java API, new CLI commands, new REST API calls - does seem excessive to make the default restrictive. Further, the users need to switch too and given the REST API is probably going to be the most used, I don't see them moving away from the old API unless they want to filter by tags. [~vinodkv] - given all this and comments from Zhijie, Alejandro and Sandy, should we get back to our original approach of setting the default Scope to ALL, and have Oozie set it to OWN? Allow users to annotate an application with multiple tags - Key: YARN-1399 URL: https://issues.apache.org/jira/browse/YARN-1399 Project: Hadoop YARN Issue Type: Improvement Reporter: Zhijie Shen Nowadays, when submitting an application, users can fill the applicationType field to facilitate searching it later. IMHO, it's good to accept multiple tags to allow users to describe their applications in multiple aspects, including the application type. Then, searching by tags may be more efficient for users to reach their desired application collection. It's pretty much like the tag system of online photo/video/music and etc. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1594) YARN-321 branch needs to be updated after YARN-888 pom changes
[ https://issues.apache.org/jira/browse/YARN-1594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870248#comment-13870248 ] Zhijie Shen commented on YARN-1594: --- Thanks Vinod for the new patch. It passed here as well. YARN-321 branch needs to be updated after YARN-888 pom changes -- Key: YARN-1594 URL: https://issues.apache.org/jira/browse/YARN-1594 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Attachments: YARN-1594-20140113.txt, YARN-1594.txt YARN-888 changed the pom structure. And so latest merge to trunk breaks YARN-321 branch. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1598) HA-related rmadmin commands don't work on a secure cluster
[
https://issues.apache.org/jira/browse/YARN-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870264#comment-13870264
]
Hadoop QA commented on YARN-1598:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12622747/yarn-1598-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.client.api.impl.TestNMClient
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2881//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2881//console
This message is automatically generated.
HA-related rmadmin commands don't work on a secure cluster
--
Key: YARN-1598
URL: https://issues.apache.org/jira/browse/YARN-1598
Project: Hadoop YARN
Issue Type: Sub-task
Components: client, resourcemanager
Affects Versions: 2.4.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
Attachments: yarn-1598-1.patch
The HA-related commands like -getServiceState -checkHealth etc. don't work in
a secure cluster.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (YARN-1599) webUI rm.webapp.AppBlock should redirect to a history App page if and when available
Gera Shegalov created YARN-1599: --- Summary: webUI rm.webapp.AppBlock should redirect to a history App page if and when available Key: YARN-1599 URL: https://issues.apache.org/jira/browse/YARN-1599 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.2.0, 2.0.5-alpha Reporter: Gera Shegalov Assignee: Gera Shegalov When the log aggregation is enabled, and the application finishes, our users think that the AppMaster logs were lost because the link to the AM attempt logs are not updated and result in HTTP 404. Only tracking URL is updated. In order to have a smoother user experience, we propose to simply redirect to the new tracking URL when the page with invalid log links is accessed. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1599) webUI rm.webapp.AppBlock should redirect to a history App page if and when available
[
https://issues.apache.org/jira/browse/YARN-1599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870351#comment-13870351
]
Hadoop QA commented on YARN-1599:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12622780/YARN-1599.v01.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2882//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2882//console
This message is automatically generated.
webUI rm.webapp.AppBlock should redirect to a history App page if and when
available
Key: YARN-1599
URL: https://issues.apache.org/jira/browse/YARN-1599
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: YARN-1599.v01.patch
When the log aggregation is enabled, and the application finishes, our users
think that the AppMaster logs were lost because the link to the AM attempt
logs are not updated and result in HTTP 404. Only tracking URL is updated. In
order to have a smoother user experience, we propose to simply redirect to
the new tracking URL when the page with invalid log links is accessed.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1596) Javadoc failures on YARN-321 branch
[ https://issues.apache.org/jira/browse/YARN-1596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870354#comment-13870354 ] Zhijie Shen commented on YARN-1596: --- Having built with the patch locally, I found he Javadoc warnings in AHS was gone. +1 Javadoc failures on YARN-321 branch --- Key: YARN-1596 URL: https://issues.apache.org/jira/browse/YARN-1596 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Attachments: YARN-1596.txt There are some javadoc issues on YARN-321 branch. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1594) YARN-321 branch needs to be updated after YARN-888 pom changes
[ https://issues.apache.org/jira/browse/YARN-1594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870469#comment-13870469 ] Zhijie Shen commented on YARN-1594: --- Committed to branch YARN-321. Thanks, [~vinodkv]! YARN-321 branch needs to be updated after YARN-888 pom changes -- Key: YARN-1594 URL: https://issues.apache.org/jira/browse/YARN-1594 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Attachments: YARN-1594-20140113.txt, YARN-1594.txt YARN-888 changed the pom structure. And so latest merge to trunk breaks YARN-321 branch. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (YARN-1594) YARN-321 branch needs to be updated after YARN-888 pom changes
[ https://issues.apache.org/jira/browse/YARN-1594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen resolved YARN-1594. --- Resolution: Fixed Hadoop Flags: Reviewed YARN-321 branch needs to be updated after YARN-888 pom changes -- Key: YARN-1594 URL: https://issues.apache.org/jira/browse/YARN-1594 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Attachments: YARN-1594-20140113.txt, YARN-1594.txt YARN-888 changed the pom structure. And so latest merge to trunk breaks YARN-321 branch. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1596) Javadoc failures on YARN-321 branch
[ https://issues.apache.org/jira/browse/YARN-1596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870488#comment-13870488 ] Zhijie Shen commented on YARN-1596: --- Committed to branch YARN-321. Thanks, [~vinodkv]! Javadoc failures on YARN-321 branch --- Key: YARN-1596 URL: https://issues.apache.org/jira/browse/YARN-1596 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Attachments: YARN-1596.txt There are some javadoc issues on YARN-321 branch. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (YARN-1596) Javadoc failures on YARN-321 branch
[ https://issues.apache.org/jira/browse/YARN-1596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen resolved YARN-1596. --- Resolution: Fixed Hadoop Flags: Reviewed Javadoc failures on YARN-321 branch --- Key: YARN-1596 URL: https://issues.apache.org/jira/browse/YARN-1596 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Attachments: YARN-1596.txt There are some javadoc issues on YARN-321 branch. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1597) FindBugs warnings on YARN-321 branch
[ https://issues.apache.org/jira/browse/YARN-1597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870499#comment-13870499 ] Zhijie Shen commented on YARN-1597: --- Committed to branch YARN-321. Thanks, [~vinodkv]! FindBugs warnings on YARN-321 branch Key: YARN-1597 URL: https://issues.apache.org/jira/browse/YARN-1597 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Attachments: YARN-1597.txt There are a bunch of findBugs warnings on YARN-321 branch. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (YARN-1597) FindBugs warnings on YARN-321 branch
[ https://issues.apache.org/jira/browse/YARN-1597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen resolved YARN-1597. --- Resolution: Fixed Hadoop Flags: Reviewed FindBugs warnings on YARN-321 branch Key: YARN-1597 URL: https://issues.apache.org/jira/browse/YARN-1597 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Attachments: YARN-1597.txt There are a bunch of findBugs warnings on YARN-321 branch. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
