[jira] [Commented] (HBASE-23078) BaseLoadBalancer should consider region replicas when randomAssignment and roundRobinAssignment
[
https://issues.apache.org/jira/browse/HBASE-23078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952571#comment-16952571
]
Guanghao Zhang commented on HBASE-23078:
Opened new issue HBASE-23178 for branch-2.1.
> BaseLoadBalancer should consider region replicas when randomAssignment and
> roundRobinAssignment
> ---
>
> Key: HBASE-23078
> URL: https://issues.apache.org/jira/browse/HBASE-23078
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.2.2
>
>
> Copied the comment in HBASE-23035.
>
> There are two problems about the LoadBalancer.
>
> 1. The cluster means the cluster state of the whole cluster. But
> hasRegionReplica is false, so it only create clusterstate by the regions
> which need to assign, not the whole cluster...
> {code:java}
> Cluster cluster = createCluster(servers, regions, false);
> List unassignedRegions = new ArrayList<>();
> roundRobinAssignment(cluster, regions, unassignedRegions,
> servers, assignments);
> protected Cluster createCluster(List servers,
> Collection regions,
> boolean hasRegionReplica) {
> // Get the snapshot of the current assignments for the regions in
> question, and then create
> // a cluster out of it. Note that we might have replicas already assigned
> to some servers
> // earlier. So we want to get the snapshot to see those assignments, but
> this will only contain
> // replicas of the regions that are passed (for performance).
> Map> clusterState = null;
> if (!hasRegionReplica) {
> clusterState = getRegionAssignmentsByServer(regions);
> } else {
> // for the case where we have region replica it is better we get the
> entire cluster's snapshot
> clusterState = getRegionAssignmentsByServer(null);
> }for (ServerName server : servers) {
> if (!clusterState.containsKey(server)) {
> clusterState.put(server, EMPTY_REGION_LIST);
> }
> }
> return new Cluster(regions, clusterState, null, this.regionFinder,
> rackManager);
> }
> {code}
> 2. wouldLowerAvailability method only consider the primary regions. The
> replica region can't assign to same server with primary region. But can be
> assigned to same server with other replica regions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23078) BaseLoadBalancer should consider region replicas when randomAssignment and roundRobinAssignment
[
https://issues.apache.org/jira/browse/HBASE-23078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940634#comment-16940634
]
Guanghao Zhang commented on HBASE-23078:
Pushed to branch-2.2+. UT failure when cherry-pick to branch-2.1. Need to dig
more.
> BaseLoadBalancer should consider region replicas when randomAssignment and
> roundRobinAssignment
> ---
>
> Key: HBASE-23078
> URL: https://issues.apache.org/jira/browse/HBASE-23078
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
>
> Copied the comment in HBASE-23035.
>
> There are two problems about the LoadBalancer.
>
> 1. The cluster means the cluster state of the whole cluster. But
> hasRegionReplica is false, so it only create clusterstate by the regions
> which need to assign, not the whole cluster...
> {code:java}
> Cluster cluster = createCluster(servers, regions, false);
> List unassignedRegions = new ArrayList<>();
> roundRobinAssignment(cluster, regions, unassignedRegions,
> servers, assignments);
> protected Cluster createCluster(List servers,
> Collection regions,
> boolean hasRegionReplica) {
> // Get the snapshot of the current assignments for the regions in
> question, and then create
> // a cluster out of it. Note that we might have replicas already assigned
> to some servers
> // earlier. So we want to get the snapshot to see those assignments, but
> this will only contain
> // replicas of the regions that are passed (for performance).
> Map> clusterState = null;
> if (!hasRegionReplica) {
> clusterState = getRegionAssignmentsByServer(regions);
> } else {
> // for the case where we have region replica it is better we get the
> entire cluster's snapshot
> clusterState = getRegionAssignmentsByServer(null);
> }for (ServerName server : servers) {
> if (!clusterState.containsKey(server)) {
> clusterState.put(server, EMPTY_REGION_LIST);
> }
> }
> return new Cluster(regions, clusterState, null, this.regionFinder,
> rackManager);
> }
> {code}
> 2. wouldLowerAvailability method only consider the primary regions. The
> replica region can't assign to same server with primary region. But can be
> assigned to same server with other replica regions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23078) BaseLoadBalancer should consider region replicas when randomAssignment and roundRobinAssignment
[
https://issues.apache.org/jira/browse/HBASE-23078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940594#comment-16940594
]
Hudson commented on HBASE-23078:
Results for branch branch-2
[build #2306 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2306/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2306//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2306//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2306//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> BaseLoadBalancer should consider region replicas when randomAssignment and
> roundRobinAssignment
> ---
>
> Key: HBASE-23078
> URL: https://issues.apache.org/jira/browse/HBASE-23078
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
>
> Copied the comment in HBASE-23035.
>
> There are two problems about the LoadBalancer.
>
> 1. The cluster means the cluster state of the whole cluster. But
> hasRegionReplica is false, so it only create clusterstate by the regions
> which need to assign, not the whole cluster...
> {code:java}
> Cluster cluster = createCluster(servers, regions, false);
> List unassignedRegions = new ArrayList<>();
> roundRobinAssignment(cluster, regions, unassignedRegions,
> servers, assignments);
> protected Cluster createCluster(List servers,
> Collection regions,
> boolean hasRegionReplica) {
> // Get the snapshot of the current assignments for the regions in
> question, and then create
> // a cluster out of it. Note that we might have replicas already assigned
> to some servers
> // earlier. So we want to get the snapshot to see those assignments, but
> this will only contain
> // replicas of the regions that are passed (for performance).
> Map> clusterState = null;
> if (!hasRegionReplica) {
> clusterState = getRegionAssignmentsByServer(regions);
> } else {
> // for the case where we have region replica it is better we get the
> entire cluster's snapshot
> clusterState = getRegionAssignmentsByServer(null);
> }for (ServerName server : servers) {
> if (!clusterState.containsKey(server)) {
> clusterState.put(server, EMPTY_REGION_LIST);
> }
> }
> return new Cluster(regions, clusterState, null, this.regionFinder,
> rackManager);
> }
> {code}
> 2. wouldLowerAvailability method only consider the primary regions. The
> replica region can't assign to same server with primary region. But can be
> assigned to same server with other replica regions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23078) BaseLoadBalancer should consider region replicas when randomAssignment and roundRobinAssignment
[
https://issues.apache.org/jira/browse/HBASE-23078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940579#comment-16940579
]
Hudson commented on HBASE-23078:
Results for branch branch-2.2
[build #645 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/645/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/645//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/645//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/645//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> BaseLoadBalancer should consider region replicas when randomAssignment and
> roundRobinAssignment
> ---
>
> Key: HBASE-23078
> URL: https://issues.apache.org/jira/browse/HBASE-23078
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
>
> Copied the comment in HBASE-23035.
>
> There are two problems about the LoadBalancer.
>
> 1. The cluster means the cluster state of the whole cluster. But
> hasRegionReplica is false, so it only create clusterstate by the regions
> which need to assign, not the whole cluster...
> {code:java}
> Cluster cluster = createCluster(servers, regions, false);
> List unassignedRegions = new ArrayList<>();
> roundRobinAssignment(cluster, regions, unassignedRegions,
> servers, assignments);
> protected Cluster createCluster(List servers,
> Collection regions,
> boolean hasRegionReplica) {
> // Get the snapshot of the current assignments for the regions in
> question, and then create
> // a cluster out of it. Note that we might have replicas already assigned
> to some servers
> // earlier. So we want to get the snapshot to see those assignments, but
> this will only contain
> // replicas of the regions that are passed (for performance).
> Map> clusterState = null;
> if (!hasRegionReplica) {
> clusterState = getRegionAssignmentsByServer(regions);
> } else {
> // for the case where we have region replica it is better we get the
> entire cluster's snapshot
> clusterState = getRegionAssignmentsByServer(null);
> }for (ServerName server : servers) {
> if (!clusterState.containsKey(server)) {
> clusterState.put(server, EMPTY_REGION_LIST);
> }
> }
> return new Cluster(regions, clusterState, null, this.regionFinder,
> rackManager);
> }
> {code}
> 2. wouldLowerAvailability method only consider the primary regions. The
> replica region can't assign to same server with primary region. But can be
> assigned to same server with other replica regions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23078) BaseLoadBalancer should consider region replicas when randomAssignment and roundRobinAssignment
[
https://issues.apache.org/jira/browse/HBASE-23078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940576#comment-16940576
]
Hudson commented on HBASE-23078:
Results for branch master
[build #1487 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1487/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1487//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1487//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1487//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> BaseLoadBalancer should consider region replicas when randomAssignment and
> roundRobinAssignment
> ---
>
> Key: HBASE-23078
> URL: https://issues.apache.org/jira/browse/HBASE-23078
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
>
> Copied the comment in HBASE-23035.
>
> There are two problems about the LoadBalancer.
>
> 1. The cluster means the cluster state of the whole cluster. But
> hasRegionReplica is false, so it only create clusterstate by the regions
> which need to assign, not the whole cluster...
> {code:java}
> Cluster cluster = createCluster(servers, regions, false);
> List unassignedRegions = new ArrayList<>();
> roundRobinAssignment(cluster, regions, unassignedRegions,
> servers, assignments);
> protected Cluster createCluster(List servers,
> Collection regions,
> boolean hasRegionReplica) {
> // Get the snapshot of the current assignments for the regions in
> question, and then create
> // a cluster out of it. Note that we might have replicas already assigned
> to some servers
> // earlier. So we want to get the snapshot to see those assignments, but
> this will only contain
> // replicas of the regions that are passed (for performance).
> Map> clusterState = null;
> if (!hasRegionReplica) {
> clusterState = getRegionAssignmentsByServer(regions);
> } else {
> // for the case where we have region replica it is better we get the
> entire cluster's snapshot
> clusterState = getRegionAssignmentsByServer(null);
> }for (ServerName server : servers) {
> if (!clusterState.containsKey(server)) {
> clusterState.put(server, EMPTY_REGION_LIST);
> }
> }
> return new Cluster(regions, clusterState, null, this.regionFinder,
> rackManager);
> }
> {code}
> 2. wouldLowerAvailability method only consider the primary regions. The
> replica region can't assign to same server with primary region. But can be
> assigned to same server with other replica regions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23078) BaseLoadBalancer should consider region replicas when randomAssignment and roundRobinAssignment
[
https://issues.apache.org/jira/browse/HBASE-23078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940222#comment-16940222
]
Guanghao Zhang commented on HBASE-23078:
{quote}Or is is your supposition that this change is a bug fix and that now the
balancer will behave more properly?
{quote}
Sir, I thought this should be a bug fix, as hasRegionReplica always be false
even there have region replica..
> BaseLoadBalancer should consider region replicas when randomAssignment and
> roundRobinAssignment
> ---
>
> Key: HBASE-23078
> URL: https://issues.apache.org/jira/browse/HBASE-23078
> Project: HBase
> Issue Type: Task
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
>
> Copied the comment in HBASE-23035.
>
> There are two problems about the LoadBalancer.
>
> 1. The cluster means the cluster state of the whole cluster. But
> hasRegionReplica is false, so it only create clusterstate by the regions
> which need to assign, not the whole cluster...
> {code:java}
> Cluster cluster = createCluster(servers, regions, false);
> List unassignedRegions = new ArrayList<>();
> roundRobinAssignment(cluster, regions, unassignedRegions,
> servers, assignments);
> protected Cluster createCluster(List servers,
> Collection regions,
> boolean hasRegionReplica) {
> // Get the snapshot of the current assignments for the regions in
> question, and then create
> // a cluster out of it. Note that we might have replicas already assigned
> to some servers
> // earlier. So we want to get the snapshot to see those assignments, but
> this will only contain
> // replicas of the regions that are passed (for performance).
> Map> clusterState = null;
> if (!hasRegionReplica) {
> clusterState = getRegionAssignmentsByServer(regions);
> } else {
> // for the case where we have region replica it is better we get the
> entire cluster's snapshot
> clusterState = getRegionAssignmentsByServer(null);
> }for (ServerName server : servers) {
> if (!clusterState.containsKey(server)) {
> clusterState.put(server, EMPTY_REGION_LIST);
> }
> }
> return new Cluster(regions, clusterState, null, this.regionFinder,
> rackManager);
> }
> {code}
> 2. wouldLowerAvailability method only consider the primary regions. The
> replica region can't assign to same server with primary region. But can be
> assigned to same server with other replica regions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23078) BaseLoadBalancer should consider region replicas when randomAssignment and roundRobinAssignment
[
https://issues.apache.org/jira/browse/HBASE-23078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938774#comment-16938774
]
Michael Stack commented on HBASE-23078:
---
Nice patch. Release note how you think this will change the behavior of the
balancer. Will it shift behavior radically? Will operators be surprised? Or is
is your supposition that this change is a bug fix and that now the balancer
will behave more properly?
> BaseLoadBalancer should consider region replicas when randomAssignment and
> roundRobinAssignment
> ---
>
> Key: HBASE-23078
> URL: https://issues.apache.org/jira/browse/HBASE-23078
> Project: HBase
> Issue Type: Task
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
>
> Copied the comment in HBASE-23035.
>
> There are two problems about the LoadBalancer.
>
> 1. The cluster means the cluster state of the whole cluster. But
> hasRegionReplica is false, so it only create clusterstate by the regions
> which need to assign, not the whole cluster...
> {code:java}
> Cluster cluster = createCluster(servers, regions, false);
> List unassignedRegions = new ArrayList<>();
> roundRobinAssignment(cluster, regions, unassignedRegions,
> servers, assignments);
> protected Cluster createCluster(List servers,
> Collection regions,
> boolean hasRegionReplica) {
> // Get the snapshot of the current assignments for the regions in
> question, and then create
> // a cluster out of it. Note that we might have replicas already assigned
> to some servers
> // earlier. So we want to get the snapshot to see those assignments, but
> this will only contain
> // replicas of the regions that are passed (for performance).
> Map> clusterState = null;
> if (!hasRegionReplica) {
> clusterState = getRegionAssignmentsByServer(regions);
> } else {
> // for the case where we have region replica it is better we get the
> entire cluster's snapshot
> clusterState = getRegionAssignmentsByServer(null);
> }for (ServerName server : servers) {
> if (!clusterState.containsKey(server)) {
> clusterState.put(server, EMPTY_REGION_LIST);
> }
> }
> return new Cluster(regions, clusterState, null, this.regionFinder,
> rackManager);
> }
> {code}
> 2. wouldLowerAvailability method only consider the primary regions. The
> replica region can't assign to same server with primary region. But can be
> assigned to same server with other replica regions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
