[jira] [Updated] (YARN-1551) Allow user-specified reason for killApplication
[ https://issues.apache.org/jira/browse/YARN-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated YARN-1551: Attachment: YARN-1551.v02.patch v02 with message length capped, and corresponding test change. Allow user-specified reason for killApplication --- Key: YARN-1551 URL: https://issues.apache.org/jira/browse/YARN-1551 Project: Hadoop YARN Issue Type: Improvement Reporter: Gera Shegalov Assignee: Gera Shegalov Attachments: YARN-1551.v01.patch, YARN-1551.v02.patch This completes MAPREDUCE-5648 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1522) TestApplicationCleanup.testAppCleanup occasionally fails
[
https://issues.apache.org/jira/browse/YARN-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859435#comment-13859435
]
Hudson commented on YARN-1522:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #438 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/438/])
YARN-1522. Fixed a race condition in the test TestApplicationCleanup that was
causing it to randomly fail. Contributed by Liyin Liang. (vinodkv:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1554328)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
TestApplicationCleanup.testAppCleanup occasionally fails
Key: YARN-1522
URL: https://issues.apache.org/jira/browse/YARN-1522
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Liyin Liang
Assignee: Liyin Liang
Fix For: 2.4.0
Attachments: YARN-1522-1.diff, YARN-1522-2.txt, YARN-1522-2.txt
TestApplicationCleanup is occasionally failing with the error:
{code}
---
Test set: org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup
---
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.215 sec
FAILURE! - in
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup
testAppCleanup(org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup)
Time elapsed: 5.555 sec FAILURE!
junit.framework.AssertionFailedError: expected:1 but was:0
at
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup.testAppCleanup(TestApplicationCleanup.java:119)
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1522) TestApplicationCleanup.testAppCleanup occasionally fails
[
https://issues.apache.org/jira/browse/YARN-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859484#comment-13859484
]
Hudson commented on YARN-1522:
--
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1630 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1630/])
YARN-1522. Fixed a race condition in the test TestApplicationCleanup that was
causing it to randomly fail. Contributed by Liyin Liang. (vinodkv:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1554328)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
TestApplicationCleanup.testAppCleanup occasionally fails
Key: YARN-1522
URL: https://issues.apache.org/jira/browse/YARN-1522
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Liyin Liang
Assignee: Liyin Liang
Fix For: 2.4.0
Attachments: YARN-1522-1.diff, YARN-1522-2.txt, YARN-1522-2.txt
TestApplicationCleanup is occasionally failing with the error:
{code}
---
Test set: org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup
---
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.215 sec
FAILURE! - in
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup
testAppCleanup(org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup)
Time elapsed: 5.555 sec FAILURE!
junit.framework.AssertionFailedError: expected:1 but was:0
at
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup.testAppCleanup(TestApplicationCleanup.java:119)
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859486#comment-13859486 ] Hudson commented on YARN-1121: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1630 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1630/]) YARN-1121. Addendum patch. Fixed AsyncDispatcher hang issue during stop due to a race condition caused by the previous patch. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1554344) * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java RMStateStore should flush all pending store events before closing - Key: YARN-1121 URL: https://issues.apache.org/jira/browse/YARN-1121 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Bikas Saha Assignee: Jian He Fix For: 2.4.0 Attachments: YARN-1121.1.patch, YARN-1121.10.patch, YARN-1121.11.patch, YARN-1121.2.patch, YARN-1121.2.patch, YARN-1121.3.patch, YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch, YARN-1121.6.patch, YARN-1121.7.patch, YARN-1121.8.patch, YARN-1121.9.patch on serviceStop it should wait for all internal pending events to drain before stopping. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1522) TestApplicationCleanup.testAppCleanup occasionally fails
[
https://issues.apache.org/jira/browse/YARN-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859505#comment-13859505
]
Hudson commented on YARN-1522:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1655 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1655/])
YARN-1522. Fixed a race condition in the test TestApplicationCleanup that was
causing it to randomly fail. Contributed by Liyin Liang. (vinodkv:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1554328)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
TestApplicationCleanup.testAppCleanup occasionally fails
Key: YARN-1522
URL: https://issues.apache.org/jira/browse/YARN-1522
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Liyin Liang
Assignee: Liyin Liang
Fix For: 2.4.0
Attachments: YARN-1522-1.diff, YARN-1522-2.txt, YARN-1522-2.txt
TestApplicationCleanup is occasionally failing with the error:
{code}
---
Test set: org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup
---
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.215 sec
FAILURE! - in
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup
testAppCleanup(org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup)
Time elapsed: 5.555 sec FAILURE!
junit.framework.AssertionFailedError: expected:1 but was:0
at
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup.testAppCleanup(TestApplicationCleanup.java:119)
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1121) RMStateStore should flush all pending store events before closing
[ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859507#comment-13859507 ] Hudson commented on YARN-1121: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1655 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1655/]) YARN-1121. Addendum patch. Fixed AsyncDispatcher hang issue during stop due to a race condition caused by the previous patch. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1554344) * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java RMStateStore should flush all pending store events before closing - Key: YARN-1121 URL: https://issues.apache.org/jira/browse/YARN-1121 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Bikas Saha Assignee: Jian He Fix For: 2.4.0 Attachments: YARN-1121.1.patch, YARN-1121.10.patch, YARN-1121.11.patch, YARN-1121.2.patch, YARN-1121.2.patch, YARN-1121.3.patch, YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch, YARN-1121.6.patch, YARN-1121.7.patch, YARN-1121.8.patch, YARN-1121.9.patch on serviceStop it should wait for all internal pending events to drain before stopping. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1297) Miscellaneous Fair Scheduler speedups
[ https://issues.apache.org/jira/browse/YARN-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-1297: - Attachment: YARN-1297.patch Miscellaneous Fair Scheduler speedups - Key: YARN-1297 URL: https://issues.apache.org/jira/browse/YARN-1297 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-1297.patch I ran the Fair Scheduler's core scheduling loop through a profiler to and identified a bunch of minimally invasive changes that can shave off a few milliseconds. The main one is demoting a couple INFO log messages to DEBUG, which brought my benchmark down from 16000 ms to 6000. A few others (which had way less of an impact) were * Most of the time in comparisons was being spent in Math.signum. I switched this to direct ifs and elses and it halved the percent of time spent in comparisons. * I removed some unnecessary instantiations of Resource objects * I made it so that queues' usage wasn't calculated from the applications up each time getResourceUsage was called. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1297) Miscellaneous Fair Scheduler speedups
[ https://issues.apache.org/jira/browse/YARN-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-1297: - Attachment: YARN-1297.patch Miscellaneous Fair Scheduler speedups - Key: YARN-1297 URL: https://issues.apache.org/jira/browse/YARN-1297 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-1297.patch, YARN-1297.patch I ran the Fair Scheduler's core scheduling loop through a profiler to and identified a bunch of minimally invasive changes that can shave off a few milliseconds. The main one is demoting a couple INFO log messages to DEBUG, which brought my benchmark down from 16000 ms to 6000. A few others (which had way less of an impact) were * Most of the time in comparisons was being spent in Math.signum. I switched this to direct ifs and elses and it halved the percent of time spent in comparisons. * I removed some unnecessary instantiations of Resource objects * I made it so that queues' usage wasn't calculated from the applications up each time getResourceUsage was called. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1297) Miscellaneous Fair Scheduler speedups
[
https://issues.apache.org/jira/browse/YARN-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859819#comment-13859819
]
Hadoop QA commented on YARN-1297:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12621001/YARN-1297.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2766//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2766//console
This message is automatically generated.
Miscellaneous Fair Scheduler speedups
-
Key: YARN-1297
URL: https://issues.apache.org/jira/browse/YARN-1297
Project: Hadoop YARN
Issue Type: Improvement
Components: scheduler
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-1297.patch, YARN-1297.patch
I ran the Fair Scheduler's core scheduling loop through a profiler to and
identified a bunch of minimally invasive changes that can shave off a few
milliseconds.
The main one is demoting a couple INFO log messages to DEBUG, which brought
my benchmark down from 16000 ms to 6000.
A few others (which had way less of an impact) were
* Most of the time in comparisons was being spent in Math.signum. I switched
this to direct ifs and elses and it halved the percent of time spent in
comparisons.
* I removed some unnecessary instantiations of Resource objects
* I made it so that queues' usage wasn't calculated from the applications up
each time getResourceUsage was called.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1029) Allow embedding leader election into the RM
[ https://issues.apache.org/jira/browse/YARN-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859821#comment-13859821 ] Karthik Kambatla commented on YARN-1029: [~vinodkv] - did you get a chance to look at the latest patch? Allow embedding leader election into the RM --- Key: YARN-1029 URL: https://issues.apache.org/jira/browse/YARN-1029 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Karthik Kambatla Attachments: embedded-zkfc-approach.patch, yarn-1029-0.patch, yarn-1029-0.patch, yarn-1029-1.patch, yarn-1029-2.patch, yarn-1029-3.patch, yarn-1029-4.patch, yarn-1029-5.patch, yarn-1029-6.patch, yarn-1029-7.patch, yarn-1029-7.patch, yarn-1029-8.patch, yarn-1029-9.patch, yarn-1029-approach.patch It should be possible to embed common ActiveStandyElector into the RM such that ZooKeeper based leader election and notification is in-built. In conjunction with a ZK state store, this configuration will be a simple deployment option. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1551) Allow user-specified reason for killApplication
[
https://issues.apache.org/jira/browse/YARN-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859830#comment-13859830
]
Hadoop QA commented on YARN-1551:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12620919/YARN-1551.v02.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.client.cli.TestYarnCLI
org.apache.hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2767//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2767//console
This message is automatically generated.
Allow user-specified reason for killApplication
---
Key: YARN-1551
URL: https://issues.apache.org/jira/browse/YARN-1551
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: YARN-1551.v01.patch, YARN-1551.v02.patch
This completes MAPREDUCE-5648
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
