[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020848#comment-16020848
]
Hudson commented on HBASE-17786:
FAILURE: Integrated in Jenkins build HBase-HBASE-14614 #244 (See
[https://builds.apache.org/job/HBase-HBASE-14614/244/])
HBASE-17786 Create LoadBalancer perf-test tool to benchmark Load (busbey: rev
da68537ae63ffbfc784de4dd6159d5d24f23f262)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestFavoredStochasticLoadBalancer.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
* (add)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerPerformanceEvaluation.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch, HBASE-17786.002.patch,
> HBASE-17786.002.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16008866#comment-16008866
]
Umesh Agashe commented on HBASE-17786:
--
Thanks [~busbey] for reviewing and committing the changes. Thanks [~stack] for
reviewing the changes.
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch, HBASE-17786.002.patch,
> HBASE-17786.002.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16008529#comment-16008529
]
Hudson commented on HBASE-17786:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2998 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2998/])
HBASE-17786 Create LoadBalancer perf-test tool to benchmark Load (busbey: rev
da68537ae63ffbfc784de4dd6159d5d24f23f262)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestFavoredStochasticLoadBalancer.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java
* (add)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerPerformanceEvaluation.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch, HBASE-17786.002.patch,
> HBASE-17786.002.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16008132#comment-16008132
]
Sean Busbey commented on HBASE-17786:
-
I have time set aside today to do another round of review. That said, if things
are pressing for others I'm fine being left behind. :)
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch, HBASE-17786.002.patch,
> HBASE-17786.002.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007653#comment-16007653
]
stack commented on HBASE-17786:
---
+1 from me. [~busbey] Ok to commit sir?
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch, HBASE-17786.002.patch,
> HBASE-17786.002.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007333#comment-16007333
]
Umesh Agashe commented on HBASE-17786:
--
Errors doesn't look related!
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch, HBASE-17786.002.patch,
> HBASE-17786.002.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005724#comment-16005724
]
Hadoop QA commented on HBASE-17786:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 34s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 30s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
61m 23s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 199m 36s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
55s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 290m 25s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestRegionReplicaFailover |
| | hadoop.hbase.client.TestAsyncBalancerAdminApi |
| | hadoop.hbase.regionserver.TestPerColumnFamilyFlush |
| | hadoop.hbase.security.access.TestCoprocessorWhitelistMasterObserver |
| | hadoop.hbase.regionserver.TestCorruptedRegionStoreFile |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestFromClientSide |
| | org.apache.hadoop.hbase.TestFullLogReconstruction |
| | org.apache.hadoop.hbase.replication.regionserver.TestWALEntryStream |
| | org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot |
| | org.apache.hadoop.hbase.TestMovedRegionsCleaner |
| | org.apache.hadoop.hbase.TestRegionRebalancing |
| | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12867402/HBASE-17786.002.patch
|
| JIRA Issue | HBASE-17786 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux a8c0cb56780f 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / c5cc81d |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| unit |
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997737#comment-15997737
]
Hadoop QA commented on HBASE-17786:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 24s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
7s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 6s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
55m 2s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 110m 7s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
2s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 191m 35s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
| | org.apache.hadoop.hbase.TestPartialResultsFromClientSide |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12866513/HBASE-17786.002.patch
|
| JIRA Issue | HBASE-17786 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux f3a76649f0ee 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 01af270 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6701/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/6701/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6701/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997592#comment-15997592
]
Umesh Agashe commented on HBASE-17786:
--
Linked reviewboard: https://reviews.apache.org/r/59003/.
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch, HBASE-17786.002.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997589#comment-15997589
]
Sean Busbey commented on HBASE-17786:
-
thanks! can you link to the reviewboard here and I'll use it to look at v2?
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch, HBASE-17786.002.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997579#comment-15997579
]
Umesh Agashe commented on HBASE-17786:
--
bq. nit: the file header should use "/*" instead of "/**" because it's not
meant to be a javadoc parsed comment. I think we make this mistake in lots of
places already, so not a big deal if you leave it.
Makes sense. Fixed. 2428 out of 3528 Java files use /**. In future, git add
hook would be useful to check this.
bq. We've been inconsistent on wether non-Test classes in the test packages
need InterfaceAudience annotations. Since this is a tool, best to be on the
safe side and label it IA.LimitedPrivate(TOOL).
Fixed.
bq. So this is general cleanup while you're in the area?
Yes
bq. Comments like this would be good to have as output while running as DEBUG
or INFO messages (perhaps on stderr if we can't use a logger for some reason).
Fixed. Added a INFO log message for it.
bq. Why this variable rather than just using
HConstants.DEFAULT_REGIONSERVER_PORT?
Fixed.
bq. Class description should include an example of how we expect folks to
invoke this.
Fixed.
{quote}
bq. What's the rationale for using System.out directly instead of a logger?
Thinking about this more, and looking at e.g. the existing
PerformanceEvaluation tool, I think we should update this to use commons-logger
as most of the rest of our stuff does.
{quote}
I referred to tests HFilePerformanceEvaluation and
ProcedureWALPerformanceEvaluation which have System.out.print statements. Most
PerformanceEvaluation tools (9/12) have System.out.print statements. Looking at
the output I think the rational is: logs get printed to stderr while explicit
print statements print to stdout. stderr can be redirected to get user
presentable output with the tool.
{quote}
We should only need the stochastic load balancer settings if the load balancer
is the stochastic load balancer, I think?
Also, we should not set these values if the incoming configuration had them set
by the user running the perf tool.
{quote}
Removed. User can set these in conf, defaults will be used otherwise.
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997564#comment-15997564
]
Hadoop QA commented on HBASE-17786:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 26s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 19s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 3s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
60m 50s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 116m 12s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
57s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 203m 32s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
| | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
| | org.apache.hadoop.hbase.TestPartialResultsFromClientSide |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12866478/HBASE-17786.001.patch
|
| JIRA Issue | HBASE-17786 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 2b4546525e28 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0fb1fd1 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6700/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/6700/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6700/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997402#comment-15997402
]
Sean Busbey commented on HBASE-17786:
-
{code}
+ // Non-default configurations.
+ private void setupConf() {
+conf.setFloat("hbase.master.balancer.stochastic.maxMovePercent", 1.0f);
+conf.setLong(StochasticLoadBalancer.MAX_STEPS_KEY, 200L);
+conf.setFloat("hbase.master.balancer.stochastic.localityCost", 0);
+conf.setLong("hbase.master.balancer.stochastic.maxRunningTime", 300 *
1000); // 300 sec
+conf.setFloat("hbase.master.balancer.stochastic.minCostNeedBalance",
0.05f);
+
+conf.setClass(HConstants.HBASE_MASTER_LOADBALANCER_CLASS,
loadBalancerClazz, LoadBalancer.class);
+loadBalancer = LoadBalancerFactory.getLoadBalancer(conf);
+ }
{code}
We should only need the stochastic load balancer settings if the load balancer
is the stochastic load balancer, I think?
Also, we should not set these values if the incoming configuration had them set
by the user running the perf tool.
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997393#comment-15997393
]
Sean Busbey commented on HBASE-17786:
-
bq. What's the rationale for using System.out directly instead of a logger?
Thinking about this more, and looking at e.g. the existing
PerformanceEvaluation tool, I think we should update this to use commons-logger
as most of the rest of our stuff does.
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997390#comment-15997390
]
Sean Busbey commented on HBASE-17786:
-
{code}
+/**
+ * Tool to test performance of different {@link
org.apache.hadoop.hbase.master.LoadBalancer}
+ * implementations.
+ */
+public class LoadBalancerPerformanceEvaluation extends AbstractHBaseTool {
{code}
Class description should include an example of how we expect folks to invoke
this.
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997352#comment-15997352
]
Sean Busbey commented on HBASE-17786:
-
{code}
+ public static final int SERVER_PORT = 61000;
...
+ ServerName sn = ServerName.valueOf("srv" + i, SERVER_PORT, i);
{code}
Why this variable rather than just using
{{HConstants.DEFAULT_REGIONSERVER_PORT}}?
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997346#comment-15997346
]
Sean Busbey commented on HBASE-17786:
-
{code}
+// Do round-robin assignment
{code}
Comments like this would be good to have as output while running as DEBUG or
INFO messages (perhaps on stderr if we can't use a logger for some reason).
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997345#comment-15997345
]
Sean Busbey commented on HBASE-17786:
-
What's the rationale for using System.out directly instead of a logger?
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997333#comment-15997333
]
Sean Busbey commented on HBASE-17786:
-
{code}
+ public static final int DEFAULT_NUM_REGIONS = 100;
{code}
nit: all these defaults can start as private since we don't need to reference
them outside of the class.
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997327#comment-15997327
]
Sean Busbey commented on HBASE-17786:
-
{quote}
bq. Why this change to make it mutable?
I think all field declaration in the body of an interface are implicitly
public, static, and final.
{quote}
Ah, good point. Missed that we were in an interface. So this is general cleanup
while you're in the area? I noticed some unused import changes as well. Usually
we try to keep cleanup and functionality changes in different commits so that
it's easier to review the two separately.
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997322#comment-15997322
]
Sean Busbey commented on HBASE-17786:
-
{code}
+/**
+ * Tool to test performance of different {@link
org.apache.hadoop.hbase.master.LoadBalancer}
+ * implementations.
+ */
+public class LoadBalancerPerformanceEvaluation extends AbstractHBaseTool {
+
{code}
We've been inconsistent on wether non-Test classes in the test packages need
{{InterfaceAudience}} annotations. Since this is a tool, best to be on the safe
side and label it IA.LimitedPrivate(TOOL).
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997316#comment-15997316
]
Sean Busbey commented on HBASE-17786:
-
{code}
index
..afb67198c3d131bfc32d3a0581a78070f4f82e7c
--- /dev/null
+++
b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerPerformanceEvaluation.java
@@ -0,0 +1,169 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
{code}
nit: the file header should use "/\*" instead of "/\*\*" because it's not meant
to be a javadoc parsed comment. I think we make this mistake in lots of places
already, so not a big deal if you leave it.
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997318#comment-15997318
]
Umesh Agashe commented on HBASE-17786:
--
bq. Why this change to make it mutable?
I think all field declaration in the body of an interface are implicitly
public, static, and final.
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997306#comment-15997306
]
Sean Busbey commented on HBASE-17786:
-
{code}
// Used to signal to the caller that the region(s) cannot be assigned
- static final ServerName BOGUS_SERVER_NAME =
ServerName.valueOf("bogus.example.com,1,1");
+ ServerName BOGUS_SERVER_NAME = ServerName.valueOf("bogus.example.com,1,1");
{code}
Why this change to make it mutable?
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994017#comment-15994017
]
Umesh Agashe commented on HBASE-17786:
--
Quote from [~mbertozzi] on internal JIRA:
This is a perf-test on the balancer code. This is no different from HFilePerf
or WALPerf
The test should cover the two methods balancer.roundRobinAssignment() and
balancer.retainAssignment()
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
