[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[ https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327281#comment-14327281 ] Hudson commented on YARN-1514: -- FAILURE: Integrated in Hadoop-Yarn-trunk #843 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/843/]) YARN-1514. Utility to benchmark ZKRMStateStore#loadState for RM HA. Contributed by Tsuyoshi OZAWA (jianhe: rev 1c03376300a46722d4147f5b8f37242f68dba0a2) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStorePerf.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/test/YarnTestDriver.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java * hadoop-yarn-project/CHANGES.txt * hadoop-project/pom.xml Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA Key: YARN-1514 URL: https://issues.apache.org/jira/browse/YARN-1514 Project: Hadoop YARN Issue Type: Sub-task Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Fix For: 2.7.0 Attachments: YARN-1514.1.patch, YARN-1514.2.patch, YARN-1514.3.patch, YARN-1514.4.patch, YARN-1514.4.patch, YARN-1514.5.patch, YARN-1514.5.patch, YARN-1514.6.patch, YARN-1514.7.patch, YARN-1514.wip-2.patch, YARN-1514.wip.patch ZKRMStateStore is very sensitive to ZNode-related operations as discussed in YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is called when RM-HA cluster does failover. Therefore, its execution time impacts failover time of RM-HA. We need utility to benchmark time execution time of ZKRMStateStore#loadStore as development tool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[ https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327297#comment-14327297 ] Hudson commented on YARN-1514: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #109 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/109/]) YARN-1514. Utility to benchmark ZKRMStateStore#loadState for RM HA. Contributed by Tsuyoshi OZAWA (jianhe: rev 1c03376300a46722d4147f5b8f37242f68dba0a2) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/test/YarnTestDriver.java * hadoop-project/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStorePerf.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA Key: YARN-1514 URL: https://issues.apache.org/jira/browse/YARN-1514 Project: Hadoop YARN Issue Type: Sub-task Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Fix For: 2.7.0 Attachments: YARN-1514.1.patch, YARN-1514.2.patch, YARN-1514.3.patch, YARN-1514.4.patch, YARN-1514.4.patch, YARN-1514.5.patch, YARN-1514.5.patch, YARN-1514.6.patch, YARN-1514.7.patch, YARN-1514.wip-2.patch, YARN-1514.wip.patch ZKRMStateStore is very sensitive to ZNode-related operations as discussed in YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is called when RM-HA cluster does failover. Therefore, its execution time impacts failover time of RM-HA. We need utility to benchmark time execution time of ZKRMStateStore#loadStore as development tool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[ https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327622#comment-14327622 ] Hudson commented on YARN-1514: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2060 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2060/]) YARN-1514. Utility to benchmark ZKRMStateStore#loadState for RM HA. Contributed by Tsuyoshi OZAWA (jianhe: rev 1c03376300a46722d4147f5b8f37242f68dba0a2) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStorePerf.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/test/YarnTestDriver.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java * hadoop-project/pom.xml Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA Key: YARN-1514 URL: https://issues.apache.org/jira/browse/YARN-1514 Project: Hadoop YARN Issue Type: Sub-task Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Fix For: 2.7.0 Attachments: YARN-1514.1.patch, YARN-1514.2.patch, YARN-1514.3.patch, YARN-1514.4.patch, YARN-1514.4.patch, YARN-1514.5.patch, YARN-1514.5.patch, YARN-1514.6.patch, YARN-1514.7.patch, YARN-1514.wip-2.patch, YARN-1514.wip.patch ZKRMStateStore is very sensitive to ZNode-related operations as discussed in YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is called when RM-HA cluster does failover. Therefore, its execution time impacts failover time of RM-HA. We need utility to benchmark time execution time of ZKRMStateStore#loadStore as development tool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[ https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327454#comment-14327454 ] Hudson commented on YARN-1514: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2041 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2041/]) YARN-1514. Utility to benchmark ZKRMStateStore#loadState for RM HA. Contributed by Tsuyoshi OZAWA (jianhe: rev 1c03376300a46722d4147f5b8f37242f68dba0a2) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStorePerf.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/test/YarnTestDriver.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java * hadoop-yarn-project/CHANGES.txt * hadoop-project/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA Key: YARN-1514 URL: https://issues.apache.org/jira/browse/YARN-1514 Project: Hadoop YARN Issue Type: Sub-task Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Fix For: 2.7.0 Attachments: YARN-1514.1.patch, YARN-1514.2.patch, YARN-1514.3.patch, YARN-1514.4.patch, YARN-1514.4.patch, YARN-1514.5.patch, YARN-1514.5.patch, YARN-1514.6.patch, YARN-1514.7.patch, YARN-1514.wip-2.patch, YARN-1514.wip.patch ZKRMStateStore is very sensitive to ZNode-related operations as discussed in YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is called when RM-HA cluster does failover. Therefore, its execution time impacts failover time of RM-HA. We need utility to benchmark time execution time of ZKRMStateStore#loadStore as development tool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[ https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327489#comment-14327489 ] Hudson commented on YARN-1514: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #110 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/110/]) YARN-1514. Utility to benchmark ZKRMStateStore#loadState for RM HA. Contributed by Tsuyoshi OZAWA (jianhe: rev 1c03376300a46722d4147f5b8f37242f68dba0a2) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStorePerf.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/test/YarnTestDriver.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * hadoop-project/pom.xml * hadoop-yarn-project/CHANGES.txt Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA Key: YARN-1514 URL: https://issues.apache.org/jira/browse/YARN-1514 Project: Hadoop YARN Issue Type: Sub-task Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Fix For: 2.7.0 Attachments: YARN-1514.1.patch, YARN-1514.2.patch, YARN-1514.3.patch, YARN-1514.4.patch, YARN-1514.4.patch, YARN-1514.5.patch, YARN-1514.5.patch, YARN-1514.6.patch, YARN-1514.7.patch, YARN-1514.wip-2.patch, YARN-1514.wip.patch ZKRMStateStore is very sensitive to ZNode-related operations as discussed in YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is called when RM-HA cluster does failover. Therefore, its execution time impacts failover time of RM-HA. We need utility to benchmark time execution time of ZKRMStateStore#loadStore as development tool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[ https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327471#comment-14327471 ] Hudson commented on YARN-1514: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #100 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/100/]) YARN-1514. Utility to benchmark ZKRMStateStore#loadState for RM HA. Contributed by Tsuyoshi OZAWA (jianhe: rev 1c03376300a46722d4147f5b8f37242f68dba0a2) * hadoop-project/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStorePerf.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/test/YarnTestDriver.java Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA Key: YARN-1514 URL: https://issues.apache.org/jira/browse/YARN-1514 Project: Hadoop YARN Issue Type: Sub-task Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Fix For: 2.7.0 Attachments: YARN-1514.1.patch, YARN-1514.2.patch, YARN-1514.3.patch, YARN-1514.4.patch, YARN-1514.4.patch, YARN-1514.5.patch, YARN-1514.5.patch, YARN-1514.6.patch, YARN-1514.7.patch, YARN-1514.wip-2.patch, YARN-1514.wip.patch ZKRMStateStore is very sensitive to ZNode-related operations as discussed in YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is called when RM-HA cluster does failover. Therefore, its execution time impacts failover time of RM-HA. We need utility to benchmark time execution time of ZKRMStateStore#loadStore as development tool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[ https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14326877#comment-14326877 ] Tsuyoshi OZAWA commented on YARN-1514: -- Thanks Jian for your review! Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA Key: YARN-1514 URL: https://issues.apache.org/jira/browse/YARN-1514 Project: Hadoop YARN Issue Type: Sub-task Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Fix For: 2.7.0 Attachments: YARN-1514.1.patch, YARN-1514.2.patch, YARN-1514.3.patch, YARN-1514.4.patch, YARN-1514.4.patch, YARN-1514.5.patch, YARN-1514.5.patch, YARN-1514.6.patch, YARN-1514.7.patch, YARN-1514.wip-2.patch, YARN-1514.wip.patch ZKRMStateStore is very sensitive to ZNode-related operations as discussed in YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is called when RM-HA cluster does failover. Therefore, its execution time impacts failover time of RM-HA. We need utility to benchmark time execution time of ZKRMStateStore#loadStore as development tool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[ https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14326822#comment-14326822 ] Hudson commented on YARN-1514: -- SUCCESS: Integrated in Hadoop-trunk-Commit #7149 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7149/]) YARN-1514. Utility to benchmark ZKRMStateStore#loadState for RM HA. Contributed by Tsuyoshi OZAWA (jianhe: rev 1c03376300a46722d4147f5b8f37242f68dba0a2) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/test/YarnTestDriver.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * hadoop-yarn-project/CHANGES.txt * hadoop-project/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStorePerf.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA Key: YARN-1514 URL: https://issues.apache.org/jira/browse/YARN-1514 Project: Hadoop YARN Issue Type: Sub-task Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Fix For: 2.7.0 Attachments: YARN-1514.1.patch, YARN-1514.2.patch, YARN-1514.3.patch, YARN-1514.4.patch, YARN-1514.4.patch, YARN-1514.5.patch, YARN-1514.5.patch, YARN-1514.6.patch, YARN-1514.7.patch, YARN-1514.wip-2.patch, YARN-1514.wip.patch ZKRMStateStore is very sensitive to ZNode-related operations as discussed in YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is called when RM-HA cluster does failover. Therefore, its execution time impacts failover time of RM-HA. We need utility to benchmark time execution time of ZKRMStateStore#loadStore as development tool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[
https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14323315#comment-14323315
]
Hadoop QA commented on YARN-1514:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12699153/YARN-1514.7.patch
against trunk revision 814afa4.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6648//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6648//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6648//console
This message is automatically generated.
Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
Key: YARN-1514
URL: https://issues.apache.org/jira/browse/YARN-1514
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Fix For: 2.7.0
Attachments: YARN-1514.1.patch, YARN-1514.2.patch, YARN-1514.3.patch,
YARN-1514.4.patch, YARN-1514.4.patch, YARN-1514.5.patch, YARN-1514.5.patch,
YARN-1514.6.patch, YARN-1514.7.patch, YARN-1514.wip-2.patch,
YARN-1514.wip.patch
ZKRMStateStore is very sensitive to ZNode-related operations as discussed in
YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is
called when RM-HA cluster does failover. Therefore, its execution time
impacts failover time of RM-HA.
We need utility to benchmark time execution time of ZKRMStateStore#loadStore
as development tool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[ https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14323548#comment-14323548 ] Tsuyoshi OZAWA commented on YARN-1514: -- findbugs and test failure look not related to the patch. [~jianhe], could you take a look? Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA Key: YARN-1514 URL: https://issues.apache.org/jira/browse/YARN-1514 Project: Hadoop YARN Issue Type: Sub-task Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Fix For: 2.7.0 Attachments: YARN-1514.1.patch, YARN-1514.2.patch, YARN-1514.3.patch, YARN-1514.4.patch, YARN-1514.4.patch, YARN-1514.5.patch, YARN-1514.5.patch, YARN-1514.6.patch, YARN-1514.7.patch, YARN-1514.wip-2.patch, YARN-1514.wip.patch ZKRMStateStore is very sensitive to ZNode-related operations as discussed in YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is called when RM-HA cluster does failover. Therefore, its execution time impacts failover time of RM-HA. We need utility to benchmark time execution time of ZKRMStateStore#loadStore as development tool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[ https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14321261#comment-14321261 ] Tsuyoshi OZAWA commented on YARN-1514: -- Thanks for taking a look, Jian. I'll update the patch. Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA Key: YARN-1514 URL: https://issues.apache.org/jira/browse/YARN-1514 Project: Hadoop YARN Issue Type: Sub-task Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Fix For: 2.7.0 Attachments: YARN-1514.1.patch, YARN-1514.2.patch, YARN-1514.3.patch, YARN-1514.4.patch, YARN-1514.4.patch, YARN-1514.5.patch, YARN-1514.5.patch, YARN-1514.6.patch, YARN-1514.wip-2.patch, YARN-1514.wip.patch ZKRMStateStore is very sensitive to ZNode-related operations as discussed in YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is called when RM-HA cluster does failover. Therefore, its execution time impacts failover time of RM-HA. We need utility to benchmark time execution time of ZKRMStateStore#loadStore as development tool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[
https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319827#comment-14319827
]
Hadoop QA commented on YARN-1514:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12698679/YARN-1514.6.patch
against trunk revision ba3c80a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1152 javac
compiler warnings (more than the trunk's current 1151 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6631//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6631//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6631//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6631//console
This message is automatically generated.
Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
Key: YARN-1514
URL: https://issues.apache.org/jira/browse/YARN-1514
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Fix For: 2.7.0
Attachments: YARN-1514.1.patch, YARN-1514.2.patch, YARN-1514.3.patch,
YARN-1514.4.patch, YARN-1514.4.patch, YARN-1514.5.patch, YARN-1514.5.patch,
YARN-1514.6.patch, YARN-1514.wip-2.patch, YARN-1514.wip.patch
ZKRMStateStore is very sensitive to ZNode-related operations as discussed in
YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is
called when RM-HA cluster does failover. Therefore, its execution time
impacts failover time of RM-HA.
We need utility to benchmark time execution time of ZKRMStateStore#loadStore
as development tool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[ https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14320669#comment-14320669 ] Jian He commented on YARN-1514: --- thanks Tsuyoshi for working on this ! this is useful. Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA Key: YARN-1514 URL: https://issues.apache.org/jira/browse/YARN-1514 Project: Hadoop YARN Issue Type: Sub-task Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Fix For: 2.7.0 Attachments: YARN-1514.1.patch, YARN-1514.2.patch, YARN-1514.3.patch, YARN-1514.4.patch, YARN-1514.4.patch, YARN-1514.5.patch, YARN-1514.5.patch, YARN-1514.6.patch, YARN-1514.wip-2.patch, YARN-1514.wip.patch ZKRMStateStore is very sensitive to ZNode-related operations as discussed in YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is called when RM-HA cluster does failover. Therefore, its execution time impacts failover time of RM-HA. We need utility to benchmark time execution time of ZKRMStateStore#loadStore as development tool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[
https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14320666#comment-14320666
]
Jian He commented on YARN-1514:
---
- I tried to run from command line and specify appsize=5 attemptsize=10, it
fails with
{code}
15/02/13 11:44:42 INFO server.PrepRequestProcessor: Got user-level
KeeperException when processing sessionid:0x14b8478c39d type:multi
cxid:0x25 zxid:0x15 txntype:-1 reqpath:n/a aborting remaining multi ops. Error
Path:/Test/ZKRMStateRoot/RMAppRoot/application_1352994193343_0001/appattempt_1352994193343__00
Error:KeeperErrorCode = NoNode for
/Test/ZKRMStateRoot/RMAppRoot/application_1352994193343_0001/appattempt_1352994193343__00
15/02/13 11:44:42 INFO recovery.ZKRMStateStore: Exception while executing a ZK
operation.
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:949)
at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:915)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$4.run(ZKRMStateStore.java:907)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$4.run(ZKRMStateStore.java:904)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithCheck(ZKRMStateStore.java:1049)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:1070)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.doMultiWithRetries(ZKRMStateStore.java:904)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.removeApplicationStateInternal(ZKRMStateStore.java:698)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStorePerf.run(TestZKRMStateStorePerf.java:232)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStorePerf.main(TestZKRMStateStorePerf.java:266)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
15/02/13 11:44:42 INFO recovery.ZKRMStateStore: Maxed out ZK retries. Giving up!
failed to cleanup. : KeeperErrorCode = NoNode
{code}
Do you have the same issue ? not sure if it’s my local problem.
- ZK_TIMEOUT_MS is not used, we can remove
- {{ContainerId.newInstance(attemptId, 0)}} this looks still using the
deprecated method
- this is not important at all. but just feel a larger number of applications
and less number of attempts my be more similar to real scenario.
{code}
private int ZK_PERF_NUM_APP_DEFAULT = 100;
private int ZK_PERF_NUM_APPATTEMPT_PER_APP = 100;
{code}
- Regarding the excessive loggings, is it possible to suppress the LOG.info()
if running from command line ?
Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
Key: YARN-1514
URL: https://issues.apache.org/jira/browse/YARN-1514
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Fix For: 2.7.0
Attachments: YARN-1514.1.patch, YARN-1514.2.patch, YARN-1514.3.patch,
YARN-1514.4.patch, YARN-1514.4.patch, YARN-1514.5.patch, YARN-1514.5.patch,
YARN-1514.6.patch, YARN-1514.wip-2.patch, YARN-1514.wip.patch
ZKRMStateStore is very sensitive to ZNode-related operations as discussed in
YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is
called when RM-HA cluster does failover. Therefore, its execution time
impacts failover time of RM-HA.
We need utility to benchmark time execution time of ZKRMStateStore#loadStore
as development tool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[
https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319514#comment-14319514
]
Hadoop QA commented on YARN-1514:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12698612/YARN-1514.5.patch
against trunk revision 2f1e5dc.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6628//console
This message is automatically generated.
Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
Key: YARN-1514
URL: https://issues.apache.org/jira/browse/YARN-1514
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Fix For: 2.7.0
Attachments: YARN-1514.1.patch, YARN-1514.2.patch, YARN-1514.3.patch,
YARN-1514.4.patch, YARN-1514.4.patch, YARN-1514.5.patch, YARN-1514.5.patch,
YARN-1514.wip-2.patch, YARN-1514.wip.patch
ZKRMStateStore is very sensitive to ZNode-related operations as discussed in
YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is
called when RM-HA cluster does failover. Therefore, its execution time
impacts failover time of RM-HA.
We need utility to benchmark time execution time of ZKRMStateStore#loadStore
as development tool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[
https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229380#comment-14229380
]
Hadoop QA commented on YARN-1514:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12662518/YARN-1514.5.patch
against trunk revision 1556f86.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5958//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5958//console
This message is automatically generated.
Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
Key: YARN-1514
URL: https://issues.apache.org/jira/browse/YARN-1514
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Fix For: 2.7.0
Attachments: YARN-1514.1.patch, YARN-1514.2.patch, YARN-1514.3.patch,
YARN-1514.4.patch, YARN-1514.4.patch, YARN-1514.5.patch,
YARN-1514.wip-2.patch, YARN-1514.wip.patch
ZKRMStateStore is very sensitive to ZNode-related operations as discussed in
YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is
called when RM-HA cluster does failover. Therefore, its execution time
impacts failover time of RM-HA.
We need utility to benchmark time execution time of ZKRMStateStore#loadStore
as development tool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[ https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122073#comment-14122073 ] Tsuyoshi OZAWA commented on YARN-1514: -- Confirmed that v5 patch can be applied to trunk. Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA Key: YARN-1514 URL: https://issues.apache.org/jira/browse/YARN-1514 Project: Hadoop YARN Issue Type: Sub-task Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Fix For: 2.6.0 Attachments: YARN-1514.1.patch, YARN-1514.2.patch, YARN-1514.3.patch, YARN-1514.4.patch, YARN-1514.4.patch, YARN-1514.5.patch, YARN-1514.wip-2.patch, YARN-1514.wip.patch ZKRMStateStore is very sensitive to ZNode-related operations as discussed in YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is called when RM-HA cluster does failover. Therefore, its execution time impacts failover time of RM-HA. We need utility to benchmark time execution time of ZKRMStateStore#loadStore as development tool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[
https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100442#comment-14100442
]
Hadoop QA commented on YARN-1514:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12662441/YARN-1514.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStorePerf
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4661//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4661//console
This message is automatically generated.
Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
Key: YARN-1514
URL: https://issues.apache.org/jira/browse/YARN-1514
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Fix For: 2.6.0
Attachments: YARN-1514.1.patch, YARN-1514.2.patch, YARN-1514.3.patch,
YARN-1514.4.patch, YARN-1514.wip-2.patch, YARN-1514.wip.patch
ZKRMStateStore is very sensitive to ZNode-related operations as discussed in
YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is
called when RM-HA cluster does failover. Therefore, its execution time
impacts failover time of RM-HA.
We need utility to benchmark time execution time of ZKRMStateStore#loadStore
as development tool.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[
https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100499#comment-14100499
]
Hadoop QA commented on YARN-1514:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12662456/YARN-1514.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStorePerf
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4662//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4662//console
This message is automatically generated.
Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
Key: YARN-1514
URL: https://issues.apache.org/jira/browse/YARN-1514
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Fix For: 2.6.0
Attachments: YARN-1514.1.patch, YARN-1514.2.patch, YARN-1514.3.patch,
YARN-1514.4.patch, YARN-1514.4.patch, YARN-1514.wip-2.patch,
YARN-1514.wip.patch
ZKRMStateStore is very sensitive to ZNode-related operations as discussed in
YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is
called when RM-HA cluster does failover. Therefore, its execution time
impacts failover time of RM-HA.
We need utility to benchmark time execution time of ZKRMStateStore#loadStore
as development tool.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[
https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14101381#comment-14101381
]
Hadoop QA commented on YARN-1514:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12662518/YARN-1514.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4666//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4666//console
This message is automatically generated.
Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
Key: YARN-1514
URL: https://issues.apache.org/jira/browse/YARN-1514
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Fix For: 2.6.0
Attachments: YARN-1514.1.patch, YARN-1514.2.patch, YARN-1514.3.patch,
YARN-1514.4.patch, YARN-1514.4.patch, YARN-1514.5.patch,
YARN-1514.wip-2.patch, YARN-1514.wip.patch
ZKRMStateStore is very sensitive to ZNode-related operations as discussed in
YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is
called when RM-HA cluster does failover. Therefore, its execution time
impacts failover time of RM-HA.
We need utility to benchmark time execution time of ZKRMStateStore#loadStore
as development tool.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[ https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14045926#comment-14045926 ] Tsuyoshi OZAWA commented on YARN-1514: -- [~kkambatl], could you take a look at this JIRA? This per tools is useful and I hope to include this feature in 2.5.0 release. Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA Key: YARN-1514 URL: https://issues.apache.org/jira/browse/YARN-1514 Project: Hadoop YARN Issue Type: Sub-task Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Fix For: 2.5.0 Attachments: YARN-1514.1.patch, YARN-1514.2.patch, YARN-1514.wip-2.patch, YARN-1514.wip.patch ZKRMStateStore is very sensitive to ZNode-related operations as discussed in YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is called when RM-HA cluster does failover. Therefore, its execution time impacts failover time of RM-HA. We need utility to benchmark time execution time of ZKRMStateStore#loadStore as development tool. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[
https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026741#comment-14026741
]
Tsuyoshi OZAWA commented on YARN-1514:
--
Attached wip version 2 patch.
1. number of applications
2. number of application attempts
3. ZK connection configuration(host:port)
4. Added YarnTestDriver to launch this perf tool from {{bin/hadoop}} command.
What a latest patch doesn't solve is dependency problem. Currently, it doesn't
include dependency on zookeeper-tests, so we'll see the following class-path
error:
{quote}
bin/hadoop jar
../hadoop-common/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/hadoop-yarn-server-resourcemanager-3.0.0-SNAPSHOT-tests.jar
java.lang.NoClassDefFoundError: org/apache/zookeeper/ZKTestCase
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
{quote}
[~kkambatl], can we add zookeeper-tests' jar to the result of {{mvn package}}?
Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
Key: YARN-1514
URL: https://issues.apache.org/jira/browse/YARN-1514
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Fix For: 2.5.0
Attachments: YARN-1514.wip-2.patch, YARN-1514.wip.patch
ZKRMStateStore is very sensitive to ZNode-related operations as discussed in
YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is
called when RM-HA cluster does failover. Therefore, its execution time
impacts failover time of RM-HA.
We need utility to benchmark time execution time of ZKRMStateStore#loadStore
as development tool.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[
https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14027224#comment-14027224
]
Tsuyoshi OZAWA commented on YARN-1514:
--
Updated a patch. This patch includes following changes:
* Adding dependency on zookeeper-tests' jar.
* Made number of applications, number of application attempts , and ZK
connection configurable.
* Added YarnTestDriver to launch this perf tool from bin/hadoop command.
Example of running jar is as follows:
{code}
bin/hadoop jar
../hadoop-common/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/hadoop-yarn-server-resourcemanager-3.0.0-SNAPSHOT-tests.jar
TestZKRMStateStorePerf -appSize 100 -appattemptsize 100 -hostPort
localhost:2181
{code}
One problem is too much logging. If we run the command, the following logs are
generated:
{quote}
...
14/06/10 23:25:55 INFO recovery.RMStateStore: Storing info for app:
application_1352994193343_0099
14/06/10 23:26:31 INFO recovery.TestZKRMStateStore: ZKRMStateStore takes 2791
msec to loadState.
ZKRMStateStore takes 2791 msec to loadState.
...
14/06/10 23:26:31 INFO recovery.ZKRMStateStore: Watcher event type: NodeDeleted
with state:SyncConnected for
path:/Test/ZKRMStateRoot/RMAppRoot/application_1352994193343_ for Service
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore in state
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: STARTED
{quote}
This is because of ZKRMStateStore generates log via {{LOG.info()}}.
[~kkambatl], what do you think about this? Should we improve it on this JIRA?
Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
Key: YARN-1514
URL: https://issues.apache.org/jira/browse/YARN-1514
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Fix For: 2.5.0
Attachments: YARN-1514.1.patch, YARN-1514.wip-2.patch,
YARN-1514.wip.patch
ZKRMStateStore is very sensitive to ZNode-related operations as discussed in
YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is
called when RM-HA cluster does failover. Therefore, its execution time
impacts failover time of RM-HA.
We need utility to benchmark time execution time of ZKRMStateStore#loadStore
as development tool.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[
https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14027255#comment-14027255
]
Hadoop QA commented on YARN-1514:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12649700/YARN-1514.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1279 javac
compiler warnings (more than the trunk's current 1276 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3952//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/3952//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3952//console
This message is automatically generated.
Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
Key: YARN-1514
URL: https://issues.apache.org/jira/browse/YARN-1514
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Fix For: 2.5.0
Attachments: YARN-1514.1.patch, YARN-1514.wip-2.patch,
YARN-1514.wip.patch
ZKRMStateStore is very sensitive to ZNode-related operations as discussed in
YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is
called when RM-HA cluster does failover. Therefore, its execution time
impacts failover time of RM-HA.
We need utility to benchmark time execution time of ZKRMStateStore#loadStore
as development tool.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[
https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14027317#comment-14027317
]
Hadoop QA commented on YARN-1514:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12649713/YARN-1514.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3953//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3953//console
This message is automatically generated.
Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
Key: YARN-1514
URL: https://issues.apache.org/jira/browse/YARN-1514
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Fix For: 2.5.0
Attachments: YARN-1514.1.patch, YARN-1514.2.patch,
YARN-1514.wip-2.patch, YARN-1514.wip.patch
ZKRMStateStore is very sensitive to ZNode-related operations as discussed in
YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is
called when RM-HA cluster does failover. Therefore, its execution time
impacts failover time of RM-HA.
We need utility to benchmark time execution time of ZKRMStateStore#loadStore
as development tool.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[ https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14020686#comment-14020686 ] Tsuyoshi OZAWA commented on YARN-1514: -- Thank you for the comment, Karthik. Sure, I'll update a patch. Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA Key: YARN-1514 URL: https://issues.apache.org/jira/browse/YARN-1514 Project: Hadoop YARN Issue Type: Sub-task Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Fix For: 2.5.0 Attachments: YARN-1514.wip.patch ZKRMStateStore is very sensitive to ZNode-related operations as discussed in YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is called when RM-HA cluster does failover. Therefore, its execution time impacts failover time of RM-HA. We need utility to benchmark time execution time of ZKRMStateStore#loadStore as development tool. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[ https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14019561#comment-14019561 ] Karthik Kambatla commented on YARN-1514: The patch looks like a good first-cut. Could you add the configuration options as you mentioned? Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA Key: YARN-1514 URL: https://issues.apache.org/jira/browse/YARN-1514 Project: Hadoop YARN Issue Type: Sub-task Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Fix For: 2.5.0 Attachments: YARN-1514.wip.patch ZKRMStateStore is very sensitive to ZNode-related operations as discussed in YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is called when RM-HA cluster does failover. Therefore, its execution time impacts failover time of RM-HA. We need utility to benchmark time execution time of ZKRMStateStore#loadStore as development tool. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[
https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999258#comment-13999258
]
Tsuyoshi OZAWA commented on YARN-1514:
--
I'll make these parameters configurable:
1. number of applications
2. number of application attempts
3. ZK connection configuration(host:port)
A result message with the WIP patch is as follows:
{quote}
ZKRMStateStore takes 12644 msec to loadState.
{quote}
Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
Key: YARN-1514
URL: https://issues.apache.org/jira/browse/YARN-1514
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Fix For: 2.5.0
Attachments: YARN-1514.wip.patch
ZKRMStateStore is very sensitive to ZNode-related operations as discussed in
YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is
called when RM-HA cluster does failover. Therefore, its execution time
impacts failover time of RM-HA.
We need utility to benchmark time execution time of ZKRMStateStore#loadStore
as development tool.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1514) Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
[ https://issues.apache.org/jira/browse/YARN-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998660#comment-13998660 ] Tsuyoshi OZAWA commented on YARN-1514: -- Rough design: 1. Launch ZKRMStateStore and initialize ZooKeeper. 2. Creating znodes based on given option. 3. Run loadState() and show how much time it takes. Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA Key: YARN-1514 URL: https://issues.apache.org/jira/browse/YARN-1514 Project: Hadoop YARN Issue Type: Sub-task Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Fix For: 2.5.0 ZKRMStateStore is very sensitive to ZNode-related operations as discussed in YARN-1307, YARN-1378 and so on. Especially, ZKRMStateStore#loadState is called when RM-HA cluster does failover. Therefore, its execution time impacts failover time of RM-HA. We need utility to benchmark time execution time of ZKRMStateStore#loadStore as development tool. -- This message was sent by Atlassian JIRA (v6.2#6252)
