[jira] [Commented] (HBASE-23949) refactor loadBalancer implements for rsgroup balance by table to achieve overallbalanced
[
https://issues.apache.org/jira/browse/HBASE-23949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067246#comment-17067246
]
Hudson commented on HBASE-23949:
Results for branch branch-2
[build #2566 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2566/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2566//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2566//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2566//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2566//JDK11_Nightly_Build_Report/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2566//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> refactor loadBalancer implements for rsgroup balance by table to achieve
> overallbalanced
> --
>
> Key: HBASE-23949
> URL: https://issues.apache.org/jira/browse/HBASE-23949
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Affects Versions: 2.2.0
>Reporter: niuyulin
>Assignee: niuyulin
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.4.0, 2.2.5
>
>
> now can not achieve overallbalanced when use rsgroup balancer and by table
> is on,
> because balance every table actually use the clusterload only contain one
> table's load.
> we should use clusterload contain all this rsgroup table's load to balance
> overall
>
> hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> {code:java}
> public boolean balance(boolean force) throws IOException {
> ..
> boolean isByTable =
> getConfiguration().getBoolean("hbase.master.loadbalance.bytable", false);
> Map>> assignments =
> this.assignmentManager.getRegionStates()
> .getAssignmentsForBalancer(tableStateManager,
> this.serverManager.getOnlineServersList(),
> isByTable);
> for (Map> serverMap : assignments.values()) {
>
> serverMap.keySet().removeAll(this.serverManager.getDrainingServersList());
> }
> //Give the balancer the current cluster state.
> this.balancer.setClusterMetrics(getClusterMetricsWithoutCoprocessor());
> this.balancer.setClusterLoad(assignments);
> List plans = new ArrayList<>();
> for (Entry>> e :
> assignments.entrySet()) {
> List partialPlans =
> this.balancer.balanceCluster(e.getKey(), e.getValue());
> if (partialPlans != null) {
> plans.addAll(partialPlans);
> }
> }
> {code}
> now do refactor:
> # add method 'balanceTable' in interface LoadBalancer
> # SimpleLoadBalancer and StochasticLoadBalancer do the real 'balanceTable' ,
> and 'balanceTable' is not support in BaseLoadBalancer and
> RSGroupBasedLoadBalancer
> # RSGroupBasedLoadBalancer invoke balanceCluster , and pass GroupClusterLoad
> to internal balacer by group
> # internal balancer balance cluster invoke 'balanceTable'
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23949) refactor loadBalancer implements for rsgroup balance by table to achieve overallbalanced
[
https://issues.apache.org/jira/browse/HBASE-23949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067239#comment-17067239
]
Hudson commented on HBASE-23949:
Results for branch branch-2.2
[build #825 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/825/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/825//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/825//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/825//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> refactor loadBalancer implements for rsgroup balance by table to achieve
> overallbalanced
> --
>
> Key: HBASE-23949
> URL: https://issues.apache.org/jira/browse/HBASE-23949
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Affects Versions: 2.2.0
>Reporter: niuyulin
>Assignee: niuyulin
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.4.0, 2.2.5
>
>
> now can not achieve overallbalanced when use rsgroup balancer and by table
> is on,
> because balance every table actually use the clusterload only contain one
> table's load.
> we should use clusterload contain all this rsgroup table's load to balance
> overall
>
> hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> {code:java}
> public boolean balance(boolean force) throws IOException {
> ..
> boolean isByTable =
> getConfiguration().getBoolean("hbase.master.loadbalance.bytable", false);
> Map>> assignments =
> this.assignmentManager.getRegionStates()
> .getAssignmentsForBalancer(tableStateManager,
> this.serverManager.getOnlineServersList(),
> isByTable);
> for (Map> serverMap : assignments.values()) {
>
> serverMap.keySet().removeAll(this.serverManager.getDrainingServersList());
> }
> //Give the balancer the current cluster state.
> this.balancer.setClusterMetrics(getClusterMetricsWithoutCoprocessor());
> this.balancer.setClusterLoad(assignments);
> List plans = new ArrayList<>();
> for (Entry>> e :
> assignments.entrySet()) {
> List partialPlans =
> this.balancer.balanceCluster(e.getKey(), e.getValue());
> if (partialPlans != null) {
> plans.addAll(partialPlans);
> }
> }
> {code}
> now do refactor:
> # add method 'balanceTable' in interface LoadBalancer
> # SimpleLoadBalancer and StochasticLoadBalancer do the real 'balanceTable' ,
> and 'balanceTable' is not support in BaseLoadBalancer and
> RSGroupBasedLoadBalancer
> # RSGroupBasedLoadBalancer invoke balanceCluster , and pass GroupClusterLoad
> to internal balacer by group
> # internal balancer balance cluster invoke 'balanceTable'
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23949) refactor loadBalancer implements for rsgroup balance by table to achieve overallbalanced
[
https://issues.apache.org/jira/browse/HBASE-23949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066852#comment-17066852
]
Hudson commented on HBASE-23949:
Results for branch branch-2.3
[build #7 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/7/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/7//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/7//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/7//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/7//JDK11_Nightly_Build_Report/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/7//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> refactor loadBalancer implements for rsgroup balance by table to achieve
> overallbalanced
> --
>
> Key: HBASE-23949
> URL: https://issues.apache.org/jira/browse/HBASE-23949
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Affects Versions: 2.2.0
>Reporter: niuyulin
>Assignee: niuyulin
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.4.0, 2.2.5
>
>
> now can not achieve overallbalanced when use rsgroup balancer and by table
> is on,
> because balance every table actually use the clusterload only contain one
> table's load.
> we should use clusterload contain all this rsgroup table's load to balance
> overall
>
> hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> {code:java}
> public boolean balance(boolean force) throws IOException {
> ..
> boolean isByTable =
> getConfiguration().getBoolean("hbase.master.loadbalance.bytable", false);
> Map>> assignments =
> this.assignmentManager.getRegionStates()
> .getAssignmentsForBalancer(tableStateManager,
> this.serverManager.getOnlineServersList(),
> isByTable);
> for (Map> serverMap : assignments.values()) {
>
> serverMap.keySet().removeAll(this.serverManager.getDrainingServersList());
> }
> //Give the balancer the current cluster state.
> this.balancer.setClusterMetrics(getClusterMetricsWithoutCoprocessor());
> this.balancer.setClusterLoad(assignments);
> List plans = new ArrayList<>();
> for (Entry>> e :
> assignments.entrySet()) {
> List partialPlans =
> this.balancer.balanceCluster(e.getKey(), e.getValue());
> if (partialPlans != null) {
> plans.addAll(partialPlans);
> }
> }
> {code}
> now do refactor:
> # add method 'balanceTable' in interface LoadBalancer
> # SimpleLoadBalancer and StochasticLoadBalancer do the real 'balanceTable' ,
> and 'balanceTable' is not support in BaseLoadBalancer and
> RSGroupBasedLoadBalancer
> # RSGroupBasedLoadBalancer invoke balanceCluster , and pass GroupClusterLoad
> to internal balacer by group
> # internal balancer balance cluster invoke 'balanceTable'
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23949) refactor loadBalancer implements for rsgroup balance by table to achieve overallbalanced
[
https://issues.apache.org/jira/browse/HBASE-23949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064977#comment-17064977
]
Hudson commented on HBASE-23949:
Results for branch master
[build #1676 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1676/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1676//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1676//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1676//JDK8_Nightly_Build_Report_(Hadoop3)/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1676//JDK11_Nightly_Build_Report/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/master/1676//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> refactor loadBalancer implements for rsgroup balance by table to achieve
> overallbalanced
> --
>
> Key: HBASE-23949
> URL: https://issues.apache.org/jira/browse/HBASE-23949
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Affects Versions: 2.2.0
>Reporter: niuyulin
>Assignee: niuyulin
>Priority: Major
>
> now can not achieve overallbalanced when use rsgroup balancer and by table
> is on,
> because balance every table actually use the clusterload only contain one
> table's load.
> we should use clusterload contain all this rsgroup table's load to balance
> overall
>
> hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> {code:java}
> public boolean balance(boolean force) throws IOException {
> ..
> boolean isByTable =
> getConfiguration().getBoolean("hbase.master.loadbalance.bytable", false);
> Map>> assignments =
> this.assignmentManager.getRegionStates()
> .getAssignmentsForBalancer(tableStateManager,
> this.serverManager.getOnlineServersList(),
> isByTable);
> for (Map> serverMap : assignments.values()) {
>
> serverMap.keySet().removeAll(this.serverManager.getDrainingServersList());
> }
> //Give the balancer the current cluster state.
> this.balancer.setClusterMetrics(getClusterMetricsWithoutCoprocessor());
> this.balancer.setClusterLoad(assignments);
> List plans = new ArrayList<>();
> for (Entry>> e :
> assignments.entrySet()) {
> List partialPlans =
> this.balancer.balanceCluster(e.getKey(), e.getValue());
> if (partialPlans != null) {
> plans.addAll(partialPlans);
> }
> }
> {code}
> now do refactor:
> # add method 'balanceTable' in interface LoadBalancer
> # SimpleLoadBalancer and StochasticLoadBalancer do the real 'balanceTable' ,
> and 'balanceTable' is not support in BaseLoadBalancer and
> RSGroupBasedLoadBalancer
> # RSGroupBasedLoadBalancer invoke balanceCluster , and pass GroupClusterLoad
> to internal balacer by group
> # internal balancer balance cluster invoke 'balanceTable'
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
