[jira] [Commented] (HBASE-12890) Provide a way to throttle the number of regions moved by the balancer
[ https://issues.apache.org/jira/browse/HBASE-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702228#comment-15702228 ] Dave Latham commented on HBASE-12890: - Watchers may be interested in a new attempt at HBASE-17178 > Provide a way to throttle the number of regions moved by the balancer > - > > Key: HBASE-12890 > URL: https://issues.apache.org/jira/browse/HBASE-12890 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.10 >Reporter: churro morales >Assignee: churro morales > Attachments: HBASE-12890.patch > > > We have a very large cluster and we frequently add remove quite a few > regionservers from our cluster. Whenever we do this the balancer moves > thousands of regions at once. Instead we provide a configuration parameter: > hbase.balancer.max.regions. This limits the number of regions that are > balanced per iteration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12890) Provide a way to throttle the number of regions moved by the balancer
[ https://issues.apache.org/jira/browse/HBASE-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704383#comment-14704383 ] Anoop Sam John commented on HBASE-12890: This has to be a balancer implementation detail. If the balancer in use is not having support, will be best to extend it and use the new one. Provide a way to throttle the number of regions moved by the balancer - Key: HBASE-12890 URL: https://issues.apache.org/jira/browse/HBASE-12890 Project: HBase Issue Type: Improvement Affects Versions: 0.98.10 Reporter: churro morales Assignee: churro morales Fix For: 2.0.0, 1.3.0, 0.98.15 Attachments: HBASE-12890.patch We have a very large cluster and we frequently add remove quite a few regionservers from our cluster. Whenever we do this the balancer moves thousands of regions at once. Instead we provide a configuration parameter: hbase.balancer.max.regions. This limits the number of regions that are balanced per iteration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12890) Provide a way to throttle the number of regions moved by the balancer
[ https://issues.apache.org/jira/browse/HBASE-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14703188#comment-14703188 ] churro morales commented on HBASE-12890: Agreed, it does work for the stochastic load balancer. Reason why i suggested the patch was this will work for any balancer you decide to go with. The patch however has the downfall where the respective balancer creates plans for everything then we filter away..but we use the table level and not stochastic. Provide a way to throttle the number of regions moved by the balancer - Key: HBASE-12890 URL: https://issues.apache.org/jira/browse/HBASE-12890 Project: HBase Issue Type: Improvement Affects Versions: 0.98.10 Reporter: churro morales Assignee: churro morales Fix For: 2.0.0, 1.3.0, 0.98.15 Attachments: HBASE-12890.patch We have a very large cluster and we frequently add remove quite a few regionservers from our cluster. Whenever we do this the balancer moves thousands of regions at once. Instead we provide a configuration parameter: hbase.balancer.max.regions. This limits the number of regions that are balanced per iteration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12890) Provide a way to throttle the number of regions moved by the balancer
[ https://issues.apache.org/jira/browse/HBASE-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694404#comment-14694404 ] Andrew Purtell commented on HBASE-12890: Ok, going to commit shortly. Provide a way to throttle the number of regions moved by the balancer - Key: HBASE-12890 URL: https://issues.apache.org/jira/browse/HBASE-12890 Project: HBase Issue Type: Improvement Affects Versions: 0.98.10 Reporter: churro morales Assignee: churro morales Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: HBASE-12890.patch We have a very large cluster and we frequently add remove quite a few regionservers from our cluster. Whenever we do this the balancer moves thousands of regions at once. Instead we provide a configuration parameter: hbase.balancer.max.regions. This limits the number of regions that are balanced per iteration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12890) Provide a way to throttle the number of regions moved by the balancer
[
https://issues.apache.org/jira/browse/HBASE-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694426#comment-14694426
]
Elliott Clark commented on HBASE-12890:
---
We already have something pretty close to this in the balancer. Why not put
that in there?
{code}hbase.master.balancer.stochastic.maxMovePercent{code}
That sets the max percent. Then we also will only move Max of the percent and
600 regions at a time. We could just make the 600 configurable. That would make
the balancer smarter rather than generating plans that won't do much.
Provide a way to throttle the number of regions moved by the balancer
-
Key: HBASE-12890
URL: https://issues.apache.org/jira/browse/HBASE-12890
Project: HBase
Issue Type: Improvement
Affects Versions: 0.98.10
Reporter: churro morales
Assignee: churro morales
Fix For: 2.0.0, 0.98.14, 1.3.0
Attachments: HBASE-12890.patch
We have a very large cluster and we frequently add remove quite a few
regionservers from our cluster. Whenever we do this the balancer moves
thousands of regions at once. Instead we provide a configuration parameter:
hbase.balancer.max.regions. This limits the number of regions that are
balanced per iteration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12890) Provide a way to throttle the number of regions moved by the balancer
[ https://issues.apache.org/jira/browse/HBASE-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14652192#comment-14652192 ] Dave Latham commented on HBASE-12890: - Thanks, Ted and Andrew. Can it be committed? Provide a way to throttle the number of regions moved by the balancer - Key: HBASE-12890 URL: https://issues.apache.org/jira/browse/HBASE-12890 Project: HBase Issue Type: Improvement Affects Versions: 0.98.10 Reporter: churro morales Assignee: churro morales Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: HBASE-12890.patch We have a very large cluster and we frequently add remove quite a few regionservers from our cluster. Whenever we do this the balancer moves thousands of regions at once. Instead we provide a configuration parameter: hbase.balancer.max.regions. This limits the number of regions that are balanced per iteration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12890) Provide a way to throttle the number of regions moved by the balancer
[ https://issues.apache.org/jira/browse/HBASE-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14650424#comment-14650424 ] Andrew Purtell commented on HBASE-12890: +1 Provide a way to throttle the number of regions moved by the balancer - Key: HBASE-12890 URL: https://issues.apache.org/jira/browse/HBASE-12890 Project: HBase Issue Type: Improvement Affects Versions: 0.98.10 Reporter: churro morales Assignee: churro morales Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: HBASE-12890.patch We have a very large cluster and we frequently add remove quite a few regionservers from our cluster. Whenever we do this the balancer moves thousands of regions at once. Instead we provide a configuration parameter: hbase.balancer.max.regions. This limits the number of regions that are balanced per iteration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12890) Provide a way to throttle the number of regions moved by the balancer
[ https://issues.apache.org/jira/browse/HBASE-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14650374#comment-14650374 ] Ted Yu commented on HBASE-12890: lgtm Provide a way to throttle the number of regions moved by the balancer - Key: HBASE-12890 URL: https://issues.apache.org/jira/browse/HBASE-12890 Project: HBase Issue Type: Improvement Affects Versions: 0.98.10 Reporter: churro morales Assignee: churro morales Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: HBASE-12890.patch We have a very large cluster and we frequently add remove quite a few regionservers from our cluster. Whenever we do this the balancer moves thousands of regions at once. Instead we provide a configuration parameter: hbase.balancer.max.regions. This limits the number of regions that are balanced per iteration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12890) Provide a way to throttle the number of regions moved by the balancer
[
https://issues.apache.org/jira/browse/HBASE-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14635722#comment-14635722
]
Hadoop QA commented on HBASE-12890:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12693449/HBASE-12890.patch
against master branch at commit 5ec5552be0534dbf4b07ef6607741ae6f9ab0495.
ATTACHMENT ID: 12693449
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestImportExport
org.apache.hadoop.hbase.util.TestProcessBasedCluster
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.sentry.tests.e2e.hive.TestPrivilegesAtTableScope.testTableViewJoin4(TestPrivilegesAtTableScope.java:401)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14853//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14853//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14853//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14853//console
This message is automatically generated.
Provide a way to throttle the number of regions moved by the balancer
-
Key: HBASE-12890
URL: https://issues.apache.org/jira/browse/HBASE-12890
Project: HBase
Issue Type: Improvement
Affects Versions: 0.98.10
Reporter: churro morales
Assignee: churro morales
Fix For: 2.0.0, 0.98.14, 1.3.0
Attachments: HBASE-12890.patch
We have a very large cluster and we frequently add remove quite a few
regionservers from our cluster. Whenever we do this the balancer moves
thousands of regions at once. Instead we provide a configuration parameter:
hbase.balancer.max.regions. This limits the number of regions that are
balanced per iteration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12890) Provide a way to throttle the number of regions moved by the balancer
[ https://issues.apache.org/jira/browse/HBASE-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14286088#comment-14286088 ] Elliott Clark commented on HBASE-12890: --- H not sure how that happened. I could have sworn that it used to be there. It should be added back. Blah sorry about that. Good catch. Provide a way to throttle the number of regions moved by the balancer - Key: HBASE-12890 URL: https://issues.apache.org/jira/browse/HBASE-12890 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0, 0.98.10, 1.1.0 Reporter: churro morales Assignee: churro morales Attachments: HBASE-12890.patch We have a very large cluster and we frequently add remove quite a few regionservers from our cluster. Whenever we do this the balancer moves thousands of regions at once. Instead we provide a configuration parameter: hbase.balancer.max.regions. This limits the number of regions that are balanced per iteration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12890) Provide a way to throttle the number of regions moved by the balancer
[
https://issues.apache.org/jira/browse/HBASE-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14286004#comment-14286004
]
churro morales commented on HBASE-12890:
Hi Elliott,
{code} hbase.master.balancer.stochastic.maxMoveRegions {code}
is never used anywhere, i noticed that there was a comment in the header of the
StochasticLoadBalancer but never mentioned anywhere else.
Provide a way to throttle the number of regions moved by the balancer
-
Key: HBASE-12890
URL: https://issues.apache.org/jira/browse/HBASE-12890
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0, 0.98.10, 1.1.0
Reporter: churro morales
Assignee: churro morales
Attachments: HBASE-12890.patch
We have a very large cluster and we frequently add remove quite a few
regionservers from our cluster. Whenever we do this the balancer moves
thousands of regions at once. Instead we provide a configuration parameter:
hbase.balancer.max.regions. This limits the number of regions that are
balanced per iteration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12890) Provide a way to throttle the number of regions moved by the balancer
[ https://issues.apache.org/jira/browse/HBASE-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14286261#comment-14286261 ] churro morales commented on HBASE-12890: No worries at all. I did not want to put this code in HMaster but the added bonus is it will work regardless of which implementation of balancer you use. Provide a way to throttle the number of regions moved by the balancer - Key: HBASE-12890 URL: https://issues.apache.org/jira/browse/HBASE-12890 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0, 0.98.10, 1.1.0 Reporter: churro morales Assignee: churro morales Attachments: HBASE-12890.patch We have a very large cluster and we frequently add remove quite a few regionservers from our cluster. Whenever we do this the balancer moves thousands of regions at once. Instead we provide a configuration parameter: hbase.balancer.max.regions. This limits the number of regions that are balanced per iteration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12890) Provide a way to throttle the number of regions moved by the balancer
[
https://issues.apache.org/jira/browse/HBASE-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14285308#comment-14285308
]
Elliott Clark commented on HBASE-12890:
---
I think the best way to do this is to make the balancer know how to slowly move
the regions.
{code}hbase.master.balancer.stochastic.maxMoveRegions{code} is the setting for
to limit the max number of regions the balancer will move in one invocation.
Provide a way to throttle the number of regions moved by the balancer
-
Key: HBASE-12890
URL: https://issues.apache.org/jira/browse/HBASE-12890
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0, 0.98.10, 1.1.0
Reporter: churro morales
Assignee: churro morales
Attachments: HBASE-12890.patch
We have a very large cluster and we frequently add remove quite a few
regionservers from our cluster. Whenever we do this the balancer moves
thousands of regions at once. Instead we provide a configuration parameter:
hbase.balancer.max.regions. This limits the number of regions that are
balanced per iteration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
