[jira] [Commented] (HBASE-19021) Restore a few important missing logics for balancer in 2.0
[
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217516#comment-16217516
]
Hudson commented on HBASE-19021:
FAILURE: Integrated in Jenkins build HBase-2.0 #743 (See
[https://builds.apache.org/job/HBase-2.0/743/])
HBASE-19021 Restore a few important missing logics for balancer in 2.0
(jerryjch: rev a49850e5c3a648a93ecca848d5d58ca7da9b2d4d)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
> Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
> Fix For: 2.0.0-alpha-4
>
> Attachments: HBASE-19021-master-v2.patch, HBASE-19021-master.patch,
> HBASE-19021-master.patch
>
>
> After looking at the code, and some testing, I see the following things are
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'.
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a
> {{testRebalanceOnRegionServerNumberChange}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19021) Restore a few important missing logics for balancer in 2.0
[
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217353#comment-16217353
]
Jerry He commented on HBASE-19021:
--
Thanks [~stack]. I got interrupted after committing.
Added Release Notes as well.
> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
> Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
> Fix For: 2.0.0-alpha-4
>
> Attachments: HBASE-19021-master-v2.patch, HBASE-19021-master.patch,
> HBASE-19021-master.patch
>
>
> After looking at the code, and some testing, I see the following things are
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'.
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a
> {{testRebalanceOnRegionServerNumberChange}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19021) Restore a few important missing logics for balancer in 2.0
[
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217180#comment-16217180
]
Hudson commented on HBASE-19021:
ABORTED: Integrated in Jenkins build HBase-Trunk_matrix #3942 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/3942/])
HBASE-19021 Restore a few important missing logics for balancer in 2.0
(jerryjch: rev 9716f62f43195ef024ac7a4bafb93a4716a7323e)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java
> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
> Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
> Fix For: 2.0.0-alpha-4
>
> Attachments: HBASE-19021-master-v2.patch, HBASE-19021-master.patch,
> HBASE-19021-master.patch
>
>
> After looking at the code, and some testing, I see the following things are
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'.
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a
> {{testRebalanceOnRegionServerNumberChange}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19021) Restore a few important missing logics for balancer in 2.0
[
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216394#comment-16216394
]
stack commented on HBASE-19021:
---
+1 Thanks [~jerryhe] for nice explanation. Shove it up in release notes where
folks will be more likely to find it.
> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
> Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
> Fix For: 2.0.0-alpha-4
>
> Attachments: HBASE-19021-master-v2.patch, HBASE-19021-master.patch,
> HBASE-19021-master.patch
>
>
> After looking at the code, and some testing, I see the following things are
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'.
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a
> {{testRebalanceOnRegionServerNumberChange}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19021) Restore a few important missing logics for balancer in 2.0
[
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215717#comment-16215717
]
Ted Yu commented on HBASE-19021:
lgtm
Fix whitespace on commit
> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
> Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
> Attachments: HBASE-19021-master-v2.patch, HBASE-19021-master.patch,
> HBASE-19021-master.patch
>
>
> After looking at the code, and some testing, I see the following things are
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'.
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a
> {{testRebalanceOnRegionServerNumberChange}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19021) Restore a few important missing logics for balancer in 2.0
[
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214850#comment-16214850
]
Hadoop QA commented on HBASE-19021:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
16s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
23s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
51m 2s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}157m
23s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}239m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:cb5c477 |
| JIRA Issue | HBASE-19021 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12893483/HBASE-19021-master-v2.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 316b4bb469ea 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 2ee8690 |
| Default Java | 1.8.0_141 |
| findbugs | v3.1.0-RC3 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/9343/artifact/patchprocess/whitespace-tabs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/9343/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/9343/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was
[jira] [Commented] (HBASE-19021) Restore a few important missing logics for balancer in 2.0
[
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214636#comment-16214636
]
Jerry He commented on HBASE-19021:
--
Are you good with the patch? [~stack], [~yuzhih...@gmail.com]
Any other comment from others?
> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
> Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
> Attachments: HBASE-19021-master-v2.patch, HBASE-19021-master.patch,
> HBASE-19021-master.patch
>
>
> After looking at the code, and some testing, I see the following things are
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'.
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a
> {{testRebalanceOnRegionServerNumberChange}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19021) Restore a few important missing logics for balancer in 2.0
[
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212029#comment-16212029
]
Hadoop QA commented on HBASE-19021:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
6s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
54s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
38m 57s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}116m 17s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}177m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestSplitTransactionOnCluster |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:af479c5 |
| JIRA Issue | HBASE-19021 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12893132/HBASE-19021-master.patch
|
| Optional Tests | asflicense shadedjars javac javadoc unit findbugs
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 6adc974bc543 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / af479c5 |
| Default Java | 1.8.0_141 |
| findbugs | v3.1.0-RC3 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/9247/artifact/patchprocess/whitespace-tabs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/9247/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/9247/testReport/ |
|
[jira] [Commented] (HBASE-19021) Restore a few important missing logics for balancer in 2.0
[
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211835#comment-16211835
]
Jerry He commented on HBASE-19021:
--
All the timeout tests passed locally for me. There is already a successful QA
run earlier.
But retry QA again.
> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
> Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
> Attachments: HBASE-19021-master.patch
>
>
> After looking at the code, and some testing, I see the following things are
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'.
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a
> {{testRebalanceOnRegionServerNumberChange}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19021) Restore a few important missing logics for balancer in 2.0
[
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211831#comment-16211831
]
Jerry He commented on HBASE-19021:
--
More explanation.
In the branch-1 RegionStates.getAssignmentsByTable()
https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java#L1115
there is a part to deal with servers w/o assignments and draining mode. This
is missing after AMv2.
But the draining mode is actually ok after a 'detour' in AMv2.
The balancer's balanceCluster() can pick a plan to move regions to the draining
servers. The regions will be 'unassigned'. But in the 'assign' phase, when
going thru retainAssignment check, the plan is checked against the server list
obtained from ServerManager.createDestinationServersList(). This list is a
good list without the draining servers. So it is like a detour, but the end
result is ok.
But I restored the branch-1 behavior, which is to take the draining servers out
of consideration from the beginning.
The balancer's retainAssignment, randomAssignment and roundRobinAssignment all
take a server list an parameters. We seem to be always calling
ServerManager.createDestinationServersList() to pass the server list. They are
all good. Only the big balanceCluster() call has the issue.
> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
> Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
> Attachments: HBASE-19021-master.patch
>
>
> After looking at the code, and some testing, I see the following things are
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'.
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a
> {{testRebalanceOnRegionServerNumberChange}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19021) Restore a few important missing logics for balancer in 2.0
[
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211787#comment-16211787
]
Jerry He commented on HBASE-19021:
--
Hi, [~stack]
bq. bytable is not implemented in AMv2 IIRC
It is implemented RegionStates, but just missing a call to it.
{code}
* @param forceByCluster a flag to force to aggregate the server-load to the
cluster level
* @return A clone of current assignments by table.
*/
public Map>
getAssignmentsByTable(
final boolean forceByCluster) {
if (!forceByCluster) return getAssignmentsByTable();
{code}
bq. You saying when you add a Server, it doesn't get Regions? (I don't recall
this in testing but perhaps so).
Yes. From looking at the code, and some testing. Tested it again today.
{noformat}
ServerName Start time Last contactVersion
Requests Per Second Num. Regions
throb1,16020,1508446443086 Thu Oct 19 13:54:03 PDT 20172 s
3.0.0-SNAPSHOT 0 0
throb1,45403,1508440236028 Thu Oct 19 12:10:36 PDT 20170 s
3.0.0-SNAPSHOT 0 6
Total:2 0 6
{noformat}
bq. You are fixing this over in another issue?
The draining issue is dealt with in this patch too.
> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
> Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
> Attachments: HBASE-19021-master.patch
>
>
> After looking at the code, and some testing, I see the following things are
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'.
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a
> {{testRebalanceOnRegionServerNumberChange}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19021) Restore a few important missing logics for balancer in 2.0
[
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207074#comment-16207074
]
Hadoop QA commented on HBASE-19021:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
11s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
56s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
37m 13s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 6s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}149m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.coprocessor.TestMasterObserver |
| | org.apache.hadoop.hbase.coprocessor.TestWALObserver |
| | org.apache.hadoop.hbase.TestClusterBootOrder |
| |
org.apache.hadoop.hbase.coprocessor.TestRegionObserverForAddingMutationsFromCoprocessors
|
| | org.apache.hadoop.hbase.TestGlobalMemStoreSize |
| | org.apache.hadoop.hbase.TestIOFencing |
| |
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove
|
| | org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass |
| | org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide |
| |
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort
|
| | org.apache.hadoop.hbase.TestPartialResultsFromClientSide |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:5d60123 |
| JIRA Issue | HBASE-19021 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12892523/HBASE-19021-master.patch
|
| Optional Tests | asflicense shadedjars javac javadoc unit findbugs
hadoopcheck hbaseanti checkstyle
[jira] [Commented] (HBASE-19021) Restore a few important missing logics for balancer in 2.0
[
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207029#comment-16207029
]
stack commented on HBASE-19021:
---
Thanks for taking a look [~jerryhe]
bq. hbase.master.loadbalance.bytable is not respected. It is always 'bytable'.
Previous default is cluster wide, not by table.
Thanks. bytable is not implemented in AMv2 IIRC.
bq. Servers with no assignments is not added for balance consideration.
You saying when you add a Server, it doesn't get Regions? (I don't recall this
in testing but perhaps so).
bq. Crashed server is not removed from the in-memory server map in
RegionStates, which affects balance.
Ok. Good.
bq. Draining marker is not respected when balance.
You are fixing this over in another issue?
Thanks.
I skimmed the patch. It looks great. Thanks.
> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
> Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
> Attachments: HBASE-19021-master.patch
>
>
> After looking at the code, and some testing, I see the following things are
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'.
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a
> {{testRebalanceOnRegionServerNumberChange}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19021) Restore a few important missing logics for balancer in 2.0
[
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206983#comment-16206983
]
Hadoop QA commented on HBASE-19021:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
39s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
37s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
52m 20s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}137m
55s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}217m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:5d60123 |
| JIRA Issue | HBASE-19021 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12892498/HBASE-19021-master.patch
|
| Optional Tests | asflicense shadedjars javac javadoc unit findbugs
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 6b7873fd0c55 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 51489b20 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/9146/artifact/patchprocess/whitespace-tabs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/9146/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/9146/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was
[jira] [Commented] (HBASE-19021) Restore a few important missing logics for balancer in 2.0
[
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206963#comment-16206963
]
Jerry He commented on HBASE-19021:
--
bq. Why the negation ?
{code}
* @param forceByCluster a flag to force to aggregate the server-load to the
cluster level
* @return A clone of current assignments by table.
*/
public Map>
getAssignmentsByTable(
final boolean forceByCluster) {
if (!forceByCluster) return getAssignmentsByTable();
{code}
!isByTable will be cluster level.
> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
> Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
>
> After looking at the code, and some testing, I see the following things are
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'.
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a
> {{testRebalanceOnRegionServerNumberChange}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19021) Restore a few important missing logics for balancer in 2.0
[
https://issues.apache.org/jira/browse/HBASE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206851#comment-16206851
]
Ted Yu commented on HBASE-19021:
{code}
1421
this.assignmentManager.getRegionStates().getAssignmentsByTable(!isByTable);
{code}
Why the negation ?
{code}
+for (Map table: result.values()) {
{code}
table is a Map. Consider renaming the variable for better readability.
{code}
+ if (this.balancerName.contains("StochasticLoadBalancer")) {
+ avgLoadPlusSlop++;
+ avgLoadMinusSlop--;
{code}
Indentation is off.
{code}
+try {
+Thread.sleep(200);
+ } catch (InterruptedException e) {}
{code}
Please handle InterruptedException properly.
> Restore a few important missing logics for balancer in 2.0
> --
>
> Key: HBASE-19021
> URL: https://issues.apache.org/jira/browse/HBASE-19021
> Project: HBase
> Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Critical
> Attachments: HBASE-19021-master.patch
>
>
> After looking at the code, and some testing, I see the following things are
> missing for balancer to work properly after AMv2.
> # hbase.master.loadbalance.bytable is not respected. It is always 'bytable'.
> Previous default is cluster wide, not by table.
> # Servers with no assignments is not added for balance consideration.
> # Crashed server is not removed from the in-memory server map in
> RegionStates, which affects balance.
> # Draining marker is not respected when balance.
> Also try to re-enable {{TestRegionRebalancing}}, which has a
> {{testRebalanceOnRegionServerNumberChange}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
