[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Pierre Zemb (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17077965#comment-17077965
 ] 

Pierre Zemb commented on HBASE-22618:
-

We published a blogpost about that contribution on my company's blog: 
[https://www.ovh.com/blog/contributing-to-apache-hbase-custom-data-balancing/]

 

In it, you will find our context, why this patch was needed, and an example 
with our custom balance function.

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
> Attachments: HBASE-22618.branch-1.001.patch, 
> HBASE-22618.branch-1.002.patch, HBASE-22618.branch-2.001.patch, 
> HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> according to the rules.* In this case, it will move regions from the second 
> group to the first.
> The balancer will:
>  * compute how many regions needs to be moved. In our example, by moving 36 
> regions on 

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Nick Dimiduk (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17077578#comment-17077578
 ] 

Nick Dimiduk commented on HBASE-22618:
--

Hi folks. I'm trying to understand what this patch does, from a user's point of 
view. It's already committed, so we cannot change the subject, but would one of 
you mind adding a release note that describes the actual fix? Looking at the 
PR, I think it's making the load balancer better handle heterogeneous hardware 
profiles. I have that right?

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
> Attachments: HBASE-22618.branch-1.001.patch, 
> HBASE-22618.branch-1.002.patch, HBASE-22618.branch-2.001.patch, 
> HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> according to the rules.* In this case, it will move regions from the second 
> group to the first.
> The balancer will:
>  * c

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Pierre Zemb (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16941255#comment-16941255
 ] 

Pierre Zemb commented on HBASE-22618:
-

I created the associated JIRA here: 
https://issues.apache.org/jira/browse/HBASE-23073

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
> Attachments: HBASE-22618.branch-1.001.patch, 
> HBASE-22618.branch-1.002.patch, HBASE-22618.branch-2.001.patch, 
> HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> according to the rules.* In this case, it will move regions from the second 
> group to the first.
> The balancer will:
>  * compute how many regions needs to be moved. In our example, by moving 36 
> regions on rs10, we could go from 120.0% to 46.0%
>  * select regions with lowest data-locality
>  * try to find an appropriate RS for the region. We will take the lowest 
> available RS.
>

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Pierre Zemb (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925235#comment-16925235
 ] 

Pierre Zemb commented on HBASE-22618:
-

Yes I can do that! How can I provide you some Release Notes?

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
> Attachments: HBASE-22618.branch-1.001.patch, 
> HBASE-22618.branch-1.002.patch, HBASE-22618.branch-2.001.patch, 
> HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> according to the rules.* In this case, it will move regions from the second 
> group to the first.
> The balancer will:
>  * compute how many regions needs to be moved. In our example, by moving 36 
> regions on rs10, we could go from 120.0% to 46.0%
>  * select regions with lowest data-locality
>  * try to find an appropriate RS for the region. We will take the lowest 
> available RS.
> h2. Other implementations

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Anoop Sam John (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16922438#comment-16922438
 ] 

Anoop Sam John commented on HBASE-22618:


Can we have a Release Notes here which explains the usage/configs etc.

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
> Attachments: HBASE-22618.branch-1.001.patch, 
> HBASE-22618.branch-1.002.patch, HBASE-22618.branch-2.001.patch, 
> HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> according to the rules.* In this case, it will move regions from the second 
> group to the first.
> The balancer will:
>  * compute how many regions needs to be moved. In our example, by moving 36 
> regions on rs10, we could go from 120.0% to 46.0%
>  * select regions with lowest data-locality
>  * try to find an appropriate RS for the region. We will take the lowest 
> available RS.
> h2. Other

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Wellington Chevreuil (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918401#comment-16918401
 ] 

Wellington Chevreuil commented on HBASE-22618:
--

Hi [~PierreZ], yes, branch-1 is targeted for the next 1.x minor release, which 
happens to be 1.5 now. The next 1.4.y maintenance release is targeted to be 
1.4.11. To target this change to that release, we need to commit it on 
branch-1.4 as well. Maintenance releases should mostly concern to bug fixes, 
but since this improvement seems more like a new feature,  I'm wondering if it 
would be the right decision to include this in maintenance releases. 

 
{quote} * The second patch is missing, making the Jira not resolved. First 
patch was about opening the possibility to add cost functions, but we want to 
also provide an example of a cost function, by adding one that is allowing for 
Heterogeneous cluster, as stated by the ticket. Should I create another 
Jira?{quote}
Yeah, let's keep things separate. Now that this Jira allows for adding custom 
cost functions, let's track custom cost functions development on its specific 
jira. Once you create this new jira, please link to this one.

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
> Attachments: HBASE-22618.branch-1.001.patch, 
> HBASE-22618.branch-1.002.patch, HBASE-22618.branch-2.001.patch, 
> HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Hudson (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918147#comment-16918147
 ] 

Hudson commented on HBASE-22618:


Results for branch branch-1
[build #1037 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1037/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(x) {color:red}-1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1037//General_Nightly_Build_Report/]


(x) {color:red}-1 jdk7 checks{color}
-- Something went wrong running this stage, please [check relevant console 
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1037//console].


(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1037//JDK8_Nightly_Build_Report_(Hadoop2)/]




(/) {color:green}+1 source release artifact{color}
-- See build output for details.


> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
> Attachments: HBASE-22618.branch-1.001.patch, 
> HBASE-22618.branch-1.002.patch, HBASE-22618.branch-2.001.patch, 
> HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The secon

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Pierre Zemb (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917988#comment-16917988
 ] 

Pierre Zemb commented on HBASE-22618:
-

Hi [~wchevreuil] ! Thanks for the help getting my patch into the different 
branches! I do have some questions:


 * 1.5 is the only target for 1.X? Can we target 1.4?
 * The second patch is missing, making the Jira not resolved. First patch was 
about opening the possibility to add cost functions, but we want to also 
provide an example of a cost function, by adding one that is allowing for 
Heterogeneous cluster, as stated by the ticket. Should I create another Jira?

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
> Attachments: HBASE-22618.branch-1.001.patch, 
> HBASE-22618.branch-1.002.patch, HBASE-22618.branch-2.001.patch, 
> HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> acc

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Wellington Chevreuil (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917570#comment-16917570
 ] 

Wellington Chevreuil commented on HBASE-22618:
--

Latest test failure should be unrelated, have the same passing locally. Also, 
the shaded and javadoc failures don't seem valid to me. Shaded is complaining 
about hbase-annotations dependency on jdk1.8, but this patch clearly doesn't 
add any change on hbase-annotations. As for the javadoc warnings, the current 
complain was not added in this patch. Pushed branch-1.002.patch to branch-1. 

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-22618.branch-1.001.patch, 
> HBASE-22618.branch-1.002.patch, HBASE-22618.branch-2.001.patch, 
> HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> according to the rules.* In this case, it will move regions from the seco

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Hudson (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917469#comment-16917469
 ] 

Hudson commented on HBASE-22618:


Results for branch branch-2
[build #2200 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2200/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2200//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2192//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2166//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-22618.branch-1.001.patch, 
> HBASE-22618.branch-1.002.patch, HBASE-22618.branch-2.001.patch, 
> HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quot

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[HBase QA (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916830#comment-16916830
 ] 

HBase QA commented on HBASE-22618:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 33m 
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
30s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
59s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
35s{color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red}  0m 
18s{color} | {color:red} branch has 10 errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
49s{color} | {color:green} branch-1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red}  0m 
16s{color} | {color:red} patch has 10 errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}  
6m 56s{color} | {color:green} Patch does not cause any errors with Hadoop 2.8.5 
2.9.2. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
46s{color} | {color:red} hbase-server generated 3 new + 5 unchanged - 0 fixed = 
8 total (was 5) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 36m 37s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}102m 29s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.executor.TestExecutorService |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 base: 
https://builds.apache.org/job/PreCommit-HBASE-Build/839/artifact/patchprocess/Dockerfile
 |
| JIRA Issue | HBASE-22618 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12978696/HBASE-22618.branch-1.002.patch
 |
| Optional Tests |  dupname  asflicense  javac  javadoc  unit  findbugs  
shadedjars  hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux a95e8a863176 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-1 / 3d22604 |
| maven | version: Apache Maven 3.0.5 |
| Default Java | 1.8.0_222 |
| shadedjars | 
https://builds.apache.org/job/PreCommit-HBASE-Build/839/artifact/patchprocess/branch-shadedjars.txt
 |
| shadedjars | 
https://builds.apache.org/job/PreCommit-HBASE-Build/839/artifact/patchprocess/patch-shadedjars.txt
 |
| javadoc | 
https://builds.apache.org/job/PreCommit-HBASE-Build/839/artifact/patchprocess/diff-javadoc-javadoc-hbase-server.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/839/artifact/patchprocess/patch-unit-hbase-ser

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Wellington Chevreuil (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916726#comment-16916726
 ] 

Wellington Chevreuil commented on HBASE-22618:
--

Committed latest branch-2 patch to related branch. Noticed pre-commit job did 
not run for latest branch-1 patch, after it had failed to apply cleanly on 
branch-1. Maybe because it has been submitted as same name? Trying attaching it 
as version 002. I'll wait for the pre-commit job results, then.

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.branch-1.001.patch, 
> HBASE-22618.branch-1.002.patch, HBASE-22618.branch-2.001.patch, 
> HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> according to the rules.* In this case, it will move regions from the second 
> group to the first.
> The balancer will:
>  * compute how many regions needs to be moved. In our example, by moving 36 
> reg

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[HBase QA (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16915093#comment-16915093
 ] 

HBase QA commented on HBASE-22618:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
14s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
48s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
11s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
 2s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
47s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
30s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
 5s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
15m 37s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.8.5 2.9.2 or 3.1.2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}298m 
10s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  1m 
10s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}351m  8s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 base: 
https://builds.apache.org/job/PreCommit-HBASE-Build/823/artifact/patchprocess/Dockerfile
 |
| JIRA Issue | HBASE-22618 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12978510/HBASE-22618.branch-2.001.patch
 |
| Optional Tests |  dupname  asflicense  javac  javadoc  unit  findbugs  
shadedjars  hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 523271eab616 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2 / 15b0bab120 |
| maven | version: Apache Maven 3.5.4 
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.11 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/823/testReport/ |
| Max. process+thread count | 4545 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/823/console |
| Powered by | Apache Yetus 0.9.0 http://yetus.apache.org |


This message was automatically generated.



> Pr

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[HBase QA (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16915083#comment-16915083
 ] 

HBase QA commented on HBASE-22618:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m  
6s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
31s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
3s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
41s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
55s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
46s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
41s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
 1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
22s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
19m  3s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.8.5 2.9.2 or 3.1.2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}241m 
48s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
36s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}308m 13s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 base: 
https://builds.apache.org/job/PreCommit-HBASE-Build/821/artifact/patchprocess/Dockerfile
 |
| JIRA Issue | HBASE-22618 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12978510/HBASE-22618.branch-2.001.patch
 |
| Optional Tests |  dupname  asflicense  javac  javadoc  unit  findbugs  
shadedjars  hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 4a7dc26c4481 4.4.0-154-generic #181-Ubuntu SMP Tue Jun 25 
05:29:03 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2 / 15b0bab120 |
| maven | version: Apache Maven 3.5.4 
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.11 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/821/testReport/ |
| Max. process+thread count | 4248 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/821/console |
| Powered by | Apache Yetus 0.9.0 http://yetus.apache.org |


This message was automatically generated.



> P

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[HBase QA (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16914421#comment-16914421
 ] 

HBase QA commented on HBASE-22618:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  8s{color} 
| {color:red} HBASE-22618 does not apply to branch-1. Rebase required? Wrong 
Branch? See 
https://yetus.apache.org/documentation/in-progress/precommit-patchnames for 
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-22618 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12978427/HBASE-22618.branch-1.001.patch
 |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/817/console |
| Powered by | Apache Yetus 0.9.0 http://yetus.apache.org |


This message was automatically generated.



> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.branch-1.001.patch, 
> HBASE-22618.branch-2.001.patch, HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that 

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Pierre Zemb (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913395#comment-16913395
 ] 

Pierre Zemb commented on HBASE-22618:
-

Yeah sorry, I need to have a look at this. I will try to fix it ASAP :)

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.branch-1.001.patch, 
> HBASE-22618.branch-2.001.patch, HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> according to the rules.* In this case, it will move regions from the second 
> group to the first.
> The balancer will:
>  * compute how many regions needs to be moved. In our example, by moving 36 
> regions on rs10, we could go from 120.0% to 46.0%
>  * select regions with lowest data-locality
>  * try to find an appropriate RS for the region. We will take the lowest 
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quot

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Wellington Chevreuil (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16911784#comment-16911784
 ] 

Wellington Chevreuil commented on HBASE-22618:
--

I had tried both branch-1 and branch-2 patches locally and build failed for 
both. Can you verify on those failed tests, [~PierreZ]?

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.branch-1.001.patch, 
> HBASE-22618.branch-2.001.patch, HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> according to the rules.* In this case, it will move regions from the second 
> group to the first.
> The balancer will:
>  * compute how many regions needs to be moved. In our example, by moving 36 
> regions on rs10, we could go from 120.0% to 46.0%
>  * select regions with lowest data-locality
>  * try to find an appropriate RS for the region. We will take the lowest 
> available RS.
> h2. Other imple

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[HBase QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16910069#comment-16910069
 ] 

HBase QA commented on HBASE-22618:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
16s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
38s{color} | {color:green} branch-1 passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
44s{color} | {color:green} branch-1 passed with JDK v1.7.0_232 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
19s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  2m 
48s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
29s{color} | {color:green} branch-1 passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
37s{color} | {color:green} branch-1 passed with JDK v1.7.0_232 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
 0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
40s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed with JDK v1.7.0_232 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  2m 
51s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}  
3m 59s{color} | {color:green} Patch does not cause any errors with Hadoop 2.8.5 
2.9.2. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
30s{color} | {color:red} hbase-server-jdk1.8.0_222 with JDK v1.8.0_222 
generated 3 new + 5 unchanged - 0 fixed = 8 total (was 5) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
39s{color} | {color:red} hbase-server-jdk1.7.0_232 with JDK v1.7.0_232 
generated 3 new + 5 unchanged - 0 fixed = 8 total (was 5) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}117m 21s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}142m 25s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.TestRegionRebalancing |
|   | hadoop.hbase.master.balancer.TestStochasticLoadBalancer |
|   | hadoop.hbase.master.balancer.TestStochasticLoadBalancer2 |
|   | hadoop.hbase.client.TestMultiParallel |
|   | hadoop.hbase.coprocessor.TestMasterObserver |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 base: 
https://builds.apache.o

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[HBase QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16910048#comment-16910048
 ] 

HBase QA commented on HBASE-22618:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
58s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
50s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 9s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
 6s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
40s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
29s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
 4s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
14m 51s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.8.5 2.9.2 or 3.1.2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 30m  1s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 19s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestMultiActionMetricsFromClient |
|   | hadoop.hbase.client.TestAsyncAccessControlAdminApi |
|   | hadoop.hbase.TestClientClusterMetrics |
|   | hadoop.hbase.coprocessor.TestCoreMasterCoprocessor |
|   | hadoop.hbase.master.TestCatalogJanitor |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 base: 
https://builds.apache.org/job/PreCommit-HBASE-Build/799/artifact/patchprocess/Dockerfile
 |
| JIRA Issue | HBASE-22618 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12977897/HBASE-22618.branch-2.001.patch
 |
| Optional Tests |  dupname  asflicense  javac  javadoc  unit  findbugs  
shadedjars  hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux c47fd252cbc5 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2 / 29ed4157b3 |
| maven | version: Apache Maven 3.5.4 
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.11 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/799/artifact/patchproc

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Pierre Zemb (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16910038#comment-16910038
 ] 

Pierre Zemb commented on HBASE-22618:
-

Done!

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.branch-1.001.patch, 
> HBASE-22618.branch-2.001.patch, HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> according to the rules.* In this case, it will move regions from the second 
> group to the first.
> The balancer will:
>  * compute how many regions needs to be moved. In our example, by moving 36 
> regions on rs10, we could go from 120.0% to 46.0%
>  * select regions with lowest data-locality
>  * try to find an appropriate RS for the region. We will take the lowest 
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}Could it work to have the stochastic load balance

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Wellington Chevreuil (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16910032#comment-16910032
 ] 

Wellington Chevreuil commented on HBASE-22618:
--

{quote}Should I be working on the branch-1 as well? It is my main target after 
all.
{quote}
Sure, just attach it here too once you are done.

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.branch-2.001.patch, 
> HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> according to the rules.* In this case, it will move regions from the second 
> group to the first.
> The balancer will:
>  * compute how many regions needs to be moved. In our example, by moving 36 
> regions on rs10, we could go from 120.0% to 46.0%
>  * select regions with lowest data-locality
>  * try to find an appropriate RS for the region. We will take the lowest 
> available RS.
> h2. Other implementations and ideas
> 

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Pierre Zemb (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16910027#comment-16910027
 ] 

Pierre Zemb commented on HBASE-22618:
-

Wow, thanks for the merge [~wchevreuil]!

 

Should I be working on the branch-1 as well? It is my main target after all.

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> according to the rules.* In this case, it will move regions from the second 
> group to the first.
> The balancer will:
>  * compute how many regions needs to be moved. In our example, by moving 36 
> regions on rs10, we could go from 120.0% to 46.0%
>  * select regions with lowest data-locality
>  * try to find an appropriate RS for the region. We will take the lowest 
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909906#comment-16909906
 ] 

Hudson commented on HBASE-22618:


Results for branch master
[build #1343 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1343/]: (/) 
*{color:green}+1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1343//General_Nightly_Build_Report/]




(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1343//JDK8_Nightly_Build_Report_(Hadoop2)/]


(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1343//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can h

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Wellington Chevreuil (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909792#comment-16909792
 ] 

Wellington Chevreuil commented on HBASE-22618:
--

Latest test failures are unrelated. Had pushed it to master, but current patch 
does not apply on branch-2. [~PierreZ], can you attach a branch-2 compatible 
patch?

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> according to the rules.* In this case, it will move regions from the second 
> group to the first.
> The balancer will:
>  * compute how many regions needs to be moved. In our example, by moving 36 
> regions on rs10, we could go from 120.0% to 46.0%
>  * select regions with lowest data-locality
>  * try to find an appropriate RS for the region. We will take the lowest 
> available RS.
> h2. Other implementations and ideas
> Clay Baenzig

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[HBase QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909134#comment-16909134
 ] 

HBase QA commented on HBASE-22618:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m  
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
20s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
39s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
15m 51s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.8.5 2.9.2 or 3.1.2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}292m 19s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
35s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}347m 35s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.TestMasterShutdown |
|   | hadoop.hbase.replication.TestReplicationSmallTestsSync |
|   | hadoop.hbase.client.TestSnapshotTemporaryDirectory |
|   | hadoop.hbase.client.TestFromClientSide |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 base: 
https://builds.apache.org/job/PreCommit-HBASE-Build/786/artifact/patchprocess/Dockerfile
 |
| JIRA Issue | HBASE-22618 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/1293/HBASE-22618.master.001.patch
 |
| Optional Tests |  dupname  asflicense  javac  javadoc  unit  findbugs  
shadedjars  hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux c7e9d7a1c47e 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 3eb602c7f7 |
| maven | version: Apache Maven 3.5.4 
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.11 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/786/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apa

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Wellington Chevreuil (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898994#comment-16898994
 ] 

Wellington Chevreuil commented on HBASE-22618:
--

Thanks [~PierreZ], had made some comments on the PR.

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.1.6, 1.4.11
>Reporter: Pierre Zemb
>Priority: Major
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> according to the rules.* In this case, it will move regions from the second 
> group to the first.
> The balancer will:
>  * compute how many regions needs to be moved. In our example, by moving 36 
> regions on rs10, we could go from 120.0% to 46.0%
>  * select regions with lowest data-locality
>  * try to find an appropriate RS for the region. We will take the lowest 
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}Could it work to have the stochastic load balancer use 
> [pluggable cost functions instead of this static list of cost 
> functions|[https://github.com/apache/hbase/bl

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Pierre Zemb (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898356#comment-16898356
 ] 

Pierre Zemb commented on HBASE-22618:
-

I just opened the PR [https://github.com/apache/hbase/pull/439]

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.1.6, 1.4.11
>Reporter: Pierre Zemb
>Priority: Major
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> according to the rules.* In this case, it will move regions from the second 
> group to the first.
> The balancer will:
>  * compute how many regions needs to be moved. In our example, by moving 36 
> regions on rs10, we could go from 120.0% to 46.0%
>  * select regions with lowest data-locality
>  * try to find an appropriate RS for the region. We will take the lowest 
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}Could it work to have the stochastic load balancer use 
> [pluggable cost functions instead of this static list of cost 
> functions|[https://github.com/apache/hbase/blob/baf3

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Pierre Zemb (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894997#comment-16894997
 ] 

Pierre Zemb commented on HBASE-22618:
-

Hi!

I'm back from my holidays, I will rework it using a Map, this is will simplify 
the implementation indeed, thanks for the suggestion. And I will create the PR 
:)

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.1.6, 1.4.11
>Reporter: Pierre Zemb
>Priority: Major
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> according to the rules.* In this case, it will move regions from the second 
> group to the first.
> The balancer will:
>  * compute how many regions needs to be moved. In our example, by moving 36 
> regions on rs10, we could go from 120.0% to 46.0%
>  * select regions with lowest data-locality
>  * try to find an appropriate RS for the region. We will take the lowest 
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}Could it work to have the stochastic load balancer use 
> [pluggable co

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Wellington Chevreuil (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16883966#comment-16883966
 ] 

Wellington Chevreuil commented on HBASE-22618:
--

{quote}This new branch is based on master this time. I decided to go through 
the first technical point, which is how we are exposing custom cost 
functions.{quote}
Can you do a pull request of this branch of yours to master? The you can push 
your future commits iteratively.

{quote}This first implementation is only capable of adding new cost functions. 
I tried to load them in a dynamic manner, but I don't like how the code 
turned-up{quote}
Hum, so each cost function class only need to be instantiated once? Maybe we 
can make usage of a _Map_ for tracking already created 
instances? Then, instead of the _switch/case_, you do just something such as 
below:
{code}
...
CostFunction costFunction = functionsMap.get(className);
if(costFunction==null){
...
//code that instantiates the function reflectively
...
functionsMap.put(className, costFunction);
}
return costFunction;
...
{code}



> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.1.6, 1.4.11
>Reporter: Pierre Zemb
>Priority: Major
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousB

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Pierre Zemb (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16883853#comment-16883853
 ] 

Pierre Zemb commented on HBASE-22618:
-

Hi!

I worked a bit on the PR. I created branch from master and started it again. 
This [new 
branch|https://github.com/PierreZ/hbase/commit/6b2edab4e5059913090e7b0db2e90d67afb15729]
 is based on master this time. I decided to go through the first technical 
point, which is how we are exposing custom cost functions.

 

This first implementation is only capable of adding new cost functions. I tried 
to load them in a dynamic manner, but I don't like how the code turned-up: as 
some functions are needed elsewhere, there is an ugly switch case matching for 
already loaded functions, visible here: 
[https://gist.github.com/PierreZ/1dc5ba6e9217cb97308cbe25d5b3#file-testfullyconfigurable-java-L212]

 

I decided to make only extension available, as every cost function implements a 
multiplier that can disabled it. What do you think about this?

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.1.6, 1.4.11
>Reporter: Pierre Zemb
>Priority: Major
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Pierre Zemb (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880361#comment-16880361
 ] 

Pierre Zemb commented on HBASE-22618:
-

Cool, I will make the implementation based on master then. Thanks for the tips!

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.1.6, 1.4.11
>Reporter: Pierre Zemb
>Priority: Major
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> according to the rules.* In this case, it will move regions from the second 
> group to the first.
> The balancer will:
>  * compute how many regions needs to be moved. In our example, by moving 36 
> regions on rs10, we could go from 120.0% to 46.0%
>  * select regions with lowest data-locality
>  * try to find an appropriate RS for the region. We will take the lowest 
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}Could it work to have the stochastic load balancer use 
> [pluggable cost functions instead of this static list of cost 
> functions|[https://github.com/apache

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Wellington Chevreuil (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880342#comment-16880342
 ] 

Wellington Chevreuil commented on HBASE-22618:
--

Oh, sorry for the confusion, thought this was based out of master.
{quote}Master branch corresponds to Hbase 2.X release right?
{quote}
No, master branch is for the next potential major release. 
{quote} I must admit that my target cluster is in 1.4
{quote}
Right, and how difficult would be to apply this on master first? We could then 
backport it back to branch-2 and branch-1, once master implementation is 
finished.

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.1.6, 1.4.11
>Reporter: Pierre Zemb
>Priority: Major
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> according to the rules.* In this case, it will move regions from the second 
> group to the first.
> The balancer will:
>  * compute how many regions needs to be moved. In our example, by moving 36 
> regions on rs10, we could go from 120.0% to 46.0%
>  * select regions with lowest data-locality
>  *

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Pierre Zemb (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880297#comment-16880297
 ] 

Pierre Zemb commented on HBASE-22618:
-

Thank you [~wchevreuil] for your comments!

 

Master branch corresponds to Hbase 2.X release right? Is it possible to 
backport the patch to 1.4? I must admit that my target cluster is in 1.4 :)

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.1.6, 1.4.11
>Reporter: Pierre Zemb
>Priority: Major
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> according to the rules.* In this case, it will move regions from the second 
> group to the first.
> The balancer will:
>  * compute how many regions needs to be moved. In our example, by moving 36 
> regions on rs10, we could go from 120.0% to 46.0%
>  * select regions with lowest data-locality
>  * try to find an appropriate RS for the region. We will take the lowest 
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}Could it work to have the stochastic load b

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Wellington Chevreuil (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880174#comment-16880174
 ] 

Wellington Chevreuil commented on HBASE-22618:
--

Thanks for sharing this, [~PierreZ]. Had left some comments on the your github 
fork. Once you are done with your next changes, [can you submit a PR to master 
branch|https://help.github.com/en/articles/creating-a-pull-request-from-a-fork] 
and link it here? 

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.1.6, 1.4.11
>Reporter: Pierre Zemb
>Priority: Major
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> according to the rules.* In this case, it will move regions from the second 
> group to the first.
> The balancer will:
>  * compute how many regions needs to be moved. In our example, by moving 36 
> regions on rs10, we could go from 120.0% to 46.0%
>  * select regions with lowest data-locality
>  * try to find an appropriate RS for the region. We will take the lowest 
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea 

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Pierre Zemb (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880118#comment-16880118
 ] 

Pierre Zemb commented on HBASE-22618:
-

Hi! Thanks [~apurtell] for your comment. I worked a bit on it last week:

 
 * I added the possibility to load a CostFunction 
[here|[https://github.com/PierreZ/hbase/commit/9ddf356ee12f0b39ee0d33211834a718f0dd6194].]
 It can only load a single function for now
 * I reimplemented my balancer as a cost function. Moving from a full balancer 
to a single cost function was a huge benefit for us, as we just need to 
implement [cost 
title|[https://github.com/PierreZ/hbase/commit/ebe2a1501dda4deb150308a3b380de3bef5961ee#diff-53043f78e2be40cfbf3ff4344bb30bd0R69]]

I will now backport my tests, and also add the possibility to load multiple 
cost functions

 

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.1.6, 1.4.11
>Reporter: Pierre Zemb
>Priority: Major
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> according to the rules.* In this case, it will move regions from the second 
> gro

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16878594#comment-16878594
 ] 

Andrew Purtell commented on HBASE-22618:


Pluggable cost functions for the load balancer would be high leverage work 
useful for a variety of things. If that can serve to satisfy the requirements 
here too it’s a good approach and a patch would be welcome. 

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.1.6, 1.4.11
>Reporter: Pierre Zemb
>Priority: Major
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> according to the rules.* In this case, it will move regions from the second 
> group to the first.
> The balancer will:
>  * compute how many regions needs to be moved. In our example, by moving 36 
> regions on rs10, we could go from 120.0% to 46.0%
>  * select regions with lowest data-locality
>  * try to find an appropriate RS for the region. We will take the lowest 
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}Could it work to 

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Pierre Zemb (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16870472#comment-16870472
 ] 

Pierre Zemb commented on HBASE-22618:
-

Our main consideration for the patch is being able to balance regions according 
to the max number of regions per RS. As you said, hardware is evolving, and 
this is why we ended-up with different kind of hardware. 

We will decommissioned some old hardware at some point, but having the 
possibility to have both new hardware and old will ease migration to it! 

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.1.6, 1.4.11
>Reporter: Pierre Zemb
>Priority: Major
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> according to the rules.* In this case, it will move regions from the second 
> group to the first.
> The balancer will:
>  * compute how many regions needs to be moved. In our example, by moving 36 
> regions on rs10, we could go from 120.0% to 46.0%
>  * select regions with lowest data-locality
>  * try to find an appropriate RS for the region. We will take the lowest 
> ava

[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment

[Vladimir Rodionov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16870394#comment-16870394
 ] 

Vladimir Rodionov commented on HBASE-22618:
---

This is, kind of, not what I expect from a next big feature.  The world of a 
big data has been moving towards disaggregated, elastic compute-storage-network 
environments. HDDs are almost dead today and will be completely dead for sure 
in 2-3 years at least as a hot/warm storage, SSD are cheap as ice cream in 
Mexico, data locality is no longer an issue when you have 25/40/100GBs 
networks, the advent of Optane PMEM DIMM will significantly impact new storage 
architectures. What concerns you today, tomorrow will no longer be the problem, 
but, of course, you are free to submit the patch *smile*

> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.1.6, 1.4.11
>Reporter: Pierre Zemb
>Priority: Major
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have 
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase 
> cluster running different kind of hardware.
> h2. Why?
>  * Cloud deployments means that we may not be able to have the same hardware 
> throughout the years
>  * Some tables may need special requirements such as SSD whereas others 
> should be using hard-drives
>  * {color:#22} {color}*in our usecase*{color:#22}(single table, 
> dedicated HBase and Hadoop tuned for our usecase, good key 
> distribution){color}*, the number of regions per RS was the real limit for 
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop 
> tuned for our usecase, good key distribution)*, the number of regions per RS 
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new 
> machines, we ended-up with differents groups of hardware: some servers can 
> handle only 180 regions, whereas the biggest can handle more than 900. 
> Because of such a difference, we had to disable the LoadBalancing to avoid 
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are 
> responsible for load balancing regions across RegionServers. That was 1.5 
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  and some early tests 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>  
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
>  and 
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
>  We wrote the balancer for our use-case, which means that:
>  * there is one table
>  * there is no region-replica
>  * good key dispersion
>  * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is 
> composed of a regexp for hostname, and a limit. For example, we could have:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> RegionServers with hostname matching the first rules will have a limit of 
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for 
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will 
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
>  * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and 
> each can handle 200 regions.
>  * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and 
> each can support 50 regions.
> Based on the following rules:
>  
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>  
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we 
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will 
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this 
> information, we will then *try to put all the RegionServers to ~48% of load 
> according to the rules.* In this case, it will move regions from the second 
> group to the first.
> The balancer will:
>  *