subject:"\[jira\] \[Commented\] $HBASE\-13103$ \[ergonomics\] add region size balancing as a feature of master"


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597260#comment-14597260
 ] 

Hudson commented on HBASE-13103:


FAILURE: Integrated in HBase-TRUNK #6591 (See 
[https://builds.apache.org/job/HBase-TRUNK/6591/])
HBASE-13103 [ergonomics] add region size balancing as a feature of master 
(antonov: rev fd37ccb63c545850c08c132b2f6470354a6629f9)
* hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
* hbase-common/src/main/resources/hbase-default.xml
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/MergeNormalizationPlan.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizerChore.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/NormalizationPlan.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizerOnCluster.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/EmptyNormalizationPlan.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizerFactory.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizer.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SplitNormalizationPlan.java


 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Improvement
  Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 1.2.0

 Attachments: HBASE-13103-branch-1.v3.patch, HBASE-13103-v0.patch, 
 HBASE-13103-v1.patch, HBASE-13103-v2.patch, HBASE-13103-v3.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master

2015-06-23 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597230#comment-14597230
 ] 

Hadoop QA commented on HBASE-13103:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12741223/HBASE-13103-branch-1.v3.patch
  against branch-1 branch at commit 6a537eb8545c7dd6c01c0d911ad12e789eeab3ae.
  ATTACHMENT ID: 12741223

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 8 new 
or modified tests.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/14521//console

This message is automatically generated.

 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Improvement
  Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-13103-branch-1.v3.patch, HBASE-13103-v0.patch, 
 HBASE-13103-v1.patch, HBASE-13103-v2.patch, HBASE-13103-v3.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597337#comment-14597337
 ] 

Hudson commented on HBASE-13103:


SUCCESS: Integrated in HBase-1.3 #10 (See 
[https://builds.apache.org/job/HBase-1.3/10/])
HBASE-13103 [ergonomics] add region size balancing as a feature of master 
(antonov: rev 84675ef6159692b0a8da219df5abcf111fe46845)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/EmptyNormalizationPlan.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizerFactory.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
* hbase-common/src/main/resources/hbase-default.xml
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/NormalizationPlan.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizerChore.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizerOnCluster.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/MergeNormalizationPlan.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SplitNormalizationPlan.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizer.java


 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Improvement
  Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 1.2.0

 Attachments: HBASE-13103-branch-1.v3.patch, HBASE-13103-v0.patch, 
 HBASE-13103-v1.patch, HBASE-13103-v2.patch, HBASE-13103-v3.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597364#comment-14597364
 ] 

Hudson commented on HBASE-13103:


SUCCESS: Integrated in HBase-1.2-IT #17 (See 
[https://builds.apache.org/job/HBase-1.2-IT/17/])
HBASE-13103 [ergonomics] add region size balancing as a feature of master 
(antonov: rev 5d1603f7591d22c212c2869d4cc820790a0a2f11)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
* hbase-common/src/main/resources/hbase-default.xml
* hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/EmptyNormalizationPlan.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizerFactory.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizerChore.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SplitNormalizationPlan.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizerOnCluster.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/NormalizationPlan.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizer.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/MergeNormalizationPlan.java


 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Improvement
  Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 1.2.0

 Attachments: HBASE-13103-branch-1.v3.patch, HBASE-13103-v0.patch, 
 HBASE-13103-v1.patch, HBASE-13103-v2.patch, HBASE-13103-v3.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597385#comment-14597385
 ] 

Hudson commented on HBASE-13103:


SUCCESS: Integrated in HBase-1.3-IT #2 (See 
[https://builds.apache.org/job/HBase-1.3-IT/2/])
HBASE-13103 [ergonomics] add region size balancing as a feature of master 
(antonov: rev 84675ef6159692b0a8da219df5abcf111fe46845)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SplitNormalizationPlan.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizerFactory.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/EmptyNormalizationPlan.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/NormalizationPlan.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/MergeNormalizationPlan.java
* hbase-common/src/main/resources/hbase-default.xml
* hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizerOnCluster.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizerChore.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizer.java


 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Improvement
  Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 1.2.0

 Attachments: HBASE-13103-branch-1.v3.patch, HBASE-13103-v0.patch, 
 HBASE-13103-v1.patch, HBASE-13103-v2.patch, HBASE-13103-v3.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597378#comment-14597378
 ] 

Hudson commented on HBASE-13103:


FAILURE: Integrated in HBase-1.2 #24 (See 
[https://builds.apache.org/job/HBase-1.2/24/])
HBASE-13103 [ergonomics] add region size balancing as a feature of master 
(antonov: rev 5d1603f7591d22c212c2869d4cc820790a0a2f11)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
* hbase-common/src/main/resources/hbase-default.xml
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/NormalizationPlan.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizerFactory.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizerChore.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizer.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/MergeNormalizationPlan.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SplitNormalizationPlan.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizerOnCluster.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/EmptyNormalizationPlan.java


 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Improvement
  Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 1.2.0

 Attachments: HBASE-13103-branch-1.v3.patch, HBASE-13103-v0.patch, 
 HBASE-13103-v1.patch, HBASE-13103-v2.patch, HBASE-13103-v3.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master

2015-06-22 Thread Sean Busbey (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596185#comment-14596185
]

Sean Busbey commented on HBASE-13103:
-

I'd like to see this in 1.2, but feature freeze is nigh. I'll leave this
targeting until I actually cut the RC this afternoon/evening. Feel free to bump
out to 1.3 if you don't think things will be ready.

[ergonomics] add region size balancing as a feature of master
-

Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch,
HBASE-13103-v2.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master

2015-06-22 Thread Nick Dimiduk (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596251#comment-14596251
 ] 

Nick Dimiduk commented on HBASE-13103:
--

Looks great [~mantonov], +1 ship it. Only thing is you're using java language 
{{assert}} in a couple places in test; instead use JUnit's {{assertTrue}}, but 
that can be fixed on commit.

 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Improvement
  Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch, 
 HBASE-13103-v2.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14595466#comment-14595466
 ] 

Mikhail Antonov commented on HBASE-13103:
-

Test failures don't seem to be related (TestRegionRebalancing is generally 
flaky and fails for me on and off, and visibility labels tests pass on my 
local). Checkstyles (lack of final on 2 classes) I'll add in next version of 
patch.

 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Improvement
  Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch, 
 HBASE-13103-v2.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596309#comment-14596309
 ] 

Mikhail Antonov commented on HBASE-13103:
-

Thanks [~ndimiduk] (will fix this remaining nits on commit), [~busbey] - 
thanks! I'll commit this shortly in an hour or two unless there's objections.

 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Improvement
  Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch, 
 HBASE-13103-v2.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master

2015-06-22 Thread Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597046#comment-14597046
 ] 

Ted Yu commented on HBASE-13103:


Latest patch should be good to go.
There is room for improvement which can be addressed in follow-on JIRAs.

 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Improvement
  Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch, 
 HBASE-13103-v2.patch, HBASE-13103-v3.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597089#comment-14597089
 ] 

Mikhail Antonov commented on HBASE-13103:
-

Thanks [~te...@apache.org], agree there's room to further improve. I'm going to 
commit v3 to master, branch-1 and branch-1.2 then.

 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Improvement
  Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch, 
 HBASE-13103-v2.patch, HBASE-13103-v3.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master

2015-06-22 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596982#comment-14596982
 ] 

Hadoop QA commented on HBASE-13103:
---

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12741160/HBASE-13103-v3.patch
  against master branch at commit d51a184051d968dc3bdc00b1c9257c0a9e5ff8a6.
  ATTACHMENT ID: 12741160

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 8 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.1 2.5.2 2.6.0)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/14511//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/14511//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/14511//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/14511//console

This message is automatically generated.

 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Improvement
  Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch, 
 HBASE-13103-v2.patch, HBASE-13103-v3.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597180#comment-14597180
 ] 

Mikhail Antonov commented on HBASE-13103:
-

Committed to master, will commit version for branch-1 and branch-1.2 shortly

 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Improvement
  Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch, 
 HBASE-13103-v2.patch, HBASE-13103-v3.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master

2015-06-20 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594427#comment-14594427
 ] 

Hadoop QA commented on HBASE-13103:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12740774/HBASE-13103-v2.patch
  against master branch at commit db08013ebeeaa85802d9795cc72b4c29c5338a47.
  ATTACHMENT ID: 12740774

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 8 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.1 2.5.2 2.6.0)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:red}-1 checkstyle{color}.  The applied patch generated 
1908 checkstyle errors (more than the master's current 1906 errors).

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   org.apache.hadoop.hbase.TestRegionRebalancing

 {color:red}-1 core zombie tests{color}.  There are 5 zombie test(s):   
at 
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testVisibilityLabelsThatDoesNotPassTheCriteria(TestVisibilityLabels.java:231)
at 
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testVisibilityLabelsInGetThatDoesNotMatchAnyDefinedLabels(TestVisibilityLabels.java:400)
at 
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes.testVisibilityLabelsWithDeleteColumnsWithNoMatchVisExpWithMultipleVersionsNoTimestamp(TestVisibilityLabelsWithDeletes.java:376)

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/14477//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/14477//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/14477//artifact/patchprocess/checkstyle-aggregate.html

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/14477//console

This message is automatically generated.

 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Improvement
  Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch, 
 HBASE-13103-v2.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master

2015-06-19 Thread Mikhail Antonov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594050#comment-14594050
 ] 

Mikhail Antonov commented on HBASE-13103:
-

bq. Yeah, there should be some upper bound on the total number of regions, 
which I assume would be something like $MAX_REGIONS_PER_SERVER * $NUM_SERVERS, 
where max regions per server is configurable.

I thought the limit is the other way around, i.e. the total number of regions 
is more or less fixed (as assignment manager won't handle that properly), 
hence, increasing number of region servers would inevitably decrease number of 
regions per RS - larger and larger regions?

 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Improvement
  Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master

2015-06-19 Thread Nick Dimiduk (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594149#comment-14594149
]

Nick Dimiduk commented on HBASE-13103:
--

Yes there's a global upper bound as well, much more discussion over on
HBASE-11165.

I'm talking about a local upper bound on an individual region server's ability
to maintain active regions online for reads and writes. Usually this is
confined by memory pressure due to active memstores of open regions.

I dunno though. Maybe [~lhofhansl] and [~phobos182] want to meter this upper
bound differently?

[ergonomics] add region size balancing as a feature of master
-

Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master

2015-06-19 Thread Mikhail Antonov (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593220#comment-14593220
]

Mikhail Antonov commented on HBASE-13103:
-

Yeah. We used to mention here that region has some ideal size and we should
try to get each region to this size, and I think we mentioned that ideal size
might be a fixed fraction of max size or something like that. May'be needs to
be more configurable.

I guess you assume here that every large table is supposed to be spread across
all RSs, and not just some subset (group?) of them? Also, to make sure I
understand right, when you say 250 regions per RS, you mean 250regions of
each table, or across all tables? Also this number of regions per RS.. I
suppose we can derive it dynamically like (max number of regions total in
cluster, as limited by AM performance, see issue about scaling to 1M regions) /
# of RS? Total max number of regions could be set in config,like 100k or 300k?

I'm thinking about roughly same logic for lower and upper ends (for lower end
another implicit threshold would be max size of each region, and for upper
limit I think there should be 2 more guards - 1) should check that total number
of regions doesn't approach the limits of AM and 2) we don't break table into
ridiculously small regions (less than N hdfs blocks?).

[ergonomics] add region size balancing as a feature of master
-

Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master

2015-06-19 Thread Mikhail Antonov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593235#comment-14593235
 ] 

Mikhail Antonov commented on HBASE-13103:
-

Thanks for review, will fix remaining items and update patch (do you think 
what's discussed here about ideal size should go there, or in subsequent 
ticket?)

 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Improvement
  Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master

2015-06-19 Thread Nick Dimiduk (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593598#comment-14593598
]

Nick Dimiduk commented on HBASE-13103:
--

Max of 250 total regions on a region server, not per table. This is a rough
guideline, and will vary based on individual cluster configuration. Yes, this
is definitely related to the 1M regions ticket.

bq. 1) should check that total number of regions doesn't approach the limits of
AM

Yeah, there should be some upper bound on the total number of regions, which I
assume would be something like {{$MAX_REGIONS_PER_SERVER * $NUM_SERVERS}},
where max regions per server is configurable.

bq. 2) we don't break table into ridiculously small regions (less than N hdfs
blocks?)

Generally yes, but there is the counter case example i mentioned above, where
I'm new to HBase and my big table is only a single region on a single host.
We want the beginners to have a good experience too. More, smaller regions
spread over an overpowered cluster should result in everything being cached and
a better intro experience.

bq. do you think what's discussed here about ideal size should go there, or in
subsequent ticket?

I'm fine with improvements on the normalizer algorithms going in with
subsequent patches. I think your harness here is enough to let people get
started -- for instance, Nasron from the user list thread titled Stochastic
Balancer by tables.

[ergonomics] add region size balancing as a feature of master
-

Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master

2015-06-18 Thread Nick Dimiduk (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14592689#comment-14592689
 ] 

Nick Dimiduk commented on HBASE-13103:
--

Left some comments on RB.

Further thinking about the use-case, this chore is aiming for an ideal state of 
even cluster utilization. We seem to think of this in terms of (1) evenly 
distributed load and (2) region servers are not hosting more regions than they 
can hold -- regions are sized just right. We assume schema design results in 
a natural application load over keys, so (1) can be approximated by uniform 
region size and count. Uniform count/server is handled by the Balancer, which 
leaves the Normalizer to worry about overall count and size. Too few overall 
and you have unused hosts (i just stood up a 10 node cluster but only one host 
is doing work!), too many and you end up with 1k regions/server.

At the lower end, we probably want to split relatively empty tables toward a 
goal of {{# of regions = 2x number of region servers}}. Or maybe 3x or 5x?

At the upper end, we want to push toward a target of ~250 regions per region 
server and those regions being of uniform size if possible.

 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Improvement
  Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master

2015-06-18 Thread Nick Dimiduk (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14592702#comment-14592702
 ] 

Nick Dimiduk commented on HBASE-13103:
--

Oh, and probably we'll want coprocessor hooks pre- and post balancer 
invocation, but that can be a follow-on ticket.

 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Improvement
  Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master

2015-06-14 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14585051#comment-14585051
 ] 

Hadoop QA commented on HBASE-13103:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12739481/HBASE-13103-v1.patch
  against master branch at commit 682b8ab8a542a903e5807053282693e3a96bad2d.
  ATTACHMENT ID: 12739481

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 8 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.1 2.5.2 2.6.0)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 1 
warning messages.

{color:red}-1 checkstyle{color}.  The applied patch generated 
1908 checkstyle errors (more than the master's current 1907 errors).

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   
org.apache.hadoop.hbase.master.regionnormalizer.TestSimpleRegionNormalizerOnCluster

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/14403//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/14403//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/14403//artifact/patchprocess/checkstyle-aggregate.html

Javadoc warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/14403//artifact/patchprocess/patchJavadocWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/14403//console

This message is automatically generated.

 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Improvement
  Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master

2015-06-14 Thread Mikhail Antonov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14585016#comment-14585016
 ] 

Mikhail Antonov commented on HBASE-13103:
-

bq. it'll be good to get this out for folks to start playing with it in 1.1.0.
That's what I'm thinking too. Apparently nobody is going to turn it on in 
production env (yet); thinking what would be the most conservative yet usable 
strategy folks may want to play with in some sandbox clusters?

 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Improvement
  Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master

2015-05-21 Thread Nick Dimiduk (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14554925#comment-14554925
 ] 

Nick Dimiduk commented on HBASE-13103:
--

Ping [~toffer], this is the ticket I mentioned.

In another conversation I had recently, it occurred to me that this would be 
really handy for folks running elastic deployments, environments like EC2, 
YARN/Slider or Mesos where clusters are intentionally growing and shrinking 
capacity as business requirements change (cc [~clehene], [~stmcpherson], 
[~ste...@apache.org])

 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Improvement
  Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-13103-v0.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master

2015-04-28 Thread Mikhail Antonov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14516547#comment-14516547
 ] 

Mikhail Antonov commented on HBASE-13103:
-

Yeah :( that still needs some more work from my side to incorporate the 
feedbacks and probably several more rounds of reviews to get something ready 
for folks to try.

 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Improvement
  Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-13103-v0.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master

2015-04-16 Thread Mikhail Antonov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497774#comment-14497774
 ] 

Mikhail Antonov commented on HBASE-13103:
-

All right, let me revise the suggestions here and I'll try to roll out next 
version of patch next week or so.

 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Brainstorming
  Components: Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-13103-v0.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master

2015-04-16 Thread Nick Dimiduk (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14498445#comment-14498445
 ] 

Nick Dimiduk commented on HBASE-13103:
--

Thanks [~mantonov], it'll be good to get this out for folks to start playing 
with it in 1.1.0.

 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Brainstorming
  Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-13103-v0.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486831#comment-14486831
 ] 

Mikhail Antonov commented on HBASE-13103:
-

Also could have a variation of reshaping which doesn't really take any 
action, but writes down recommended merge/splits command into a file and makes 
it available somewhere?

 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Brainstorming
  Components: Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-13103-v0.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master

[
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486884#comment-14486884
]

Mikhail Antonov commented on HBASE-13103:
-

bq. Yeah, meant the reshaping after I identified that something is odd/bad
about a table. But maybe it's better to just automate, otherwise nobody would
use it, as you say.

I could have a switch like auto (chore + admin rpc calls accepted), manual
(no chore, admin calls accepted), disabled (no chore, no rpc calls allowed)
in hbase config for master. Or just auto and manual. Also thinking may be
exposing more params to adjust the aggressiveness of reshaping would help
people to adopt it. Probably better have policy which improves cluster state
little bit, which many people are willing to turn on and forget about, rather
than a policy, which could theoretically improve cluster state a lot, which
most of production users would be afraid to turn on.

As you said (and many users would likely agree!) that you'd be hesitant to turn
it on unless you know that it takes nearly perfect decision. What if we try to
formalize these rules, like -

- only normalize tables which opted in (like in table descriptor)
- don't touch regions which served writes in last N minutes, or served more
than X reads last hour
- don't normalize if balancer is in progress, or any splits/merges are in
progress
- don't normalize if RS hosting regions we want to split/merge is under high
load (need to define it)

May be you could list some more? Thanks for highlighting that point. W/o
proper/configurable safeguarding probably many people won't have it enabled.

[ergonomics] add region size balancing as a feature of master
-

Key: HBASE-13103
URL: https://issues.apache.org/jira/browse/HBASE-13103
Project: HBase
Issue Type: Brainstorming
Components: Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
Fix For: 2.0.0, 1.1.0

Attachments: HBASE-13103-v0.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master

2015-04-09 Thread Lars Hofhansl (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486812#comment-14486812
]

Lars Hofhansl commented on HBASE-13103:
---

Just reading through the comments here. Unless the reshaper perfectly takes
all factors into account I'd be very hesitant to run it on our clusters. With
perfect I mean that it knows about load, disk IO, etc. Since that's hard (or
impossible) I think I'd prefer to trigger this manually, as suggested in the
description. But maybe I am overly cautious.

Split decisions can be made locally and are rarely bad (unless really
excessive). Merge decisions need (a) global knowledge - not all regions may on
the same server and (b) can possibly lead to worse performance (hot regions
merged together, etc)
What we can automate is merging empty regions away.

[ergonomics] add region size balancing as a feature of master
-

Attachments: HBASE-13103-v0.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master


[ 
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486827#comment-14486827
 ] 

Mikhail Antonov commented on HBASE-13103:
-

[~larsh]

bq. I think I'd prefer to trigger this manually
You mean - you'd prefer to do splits and merges manually, or you'd prefer to 
kick the reshaping manually (via admin command, rather than letting it run as 
a chore)?

I'm thinking what could be done to make it safer and more conservative, while 
still reliving cluster admin of at least some housekeeping tasks. If this isn't 
safe, most people probably just won't turn it on..

 - since as you said, split decisions are generally safer than merge decisions, 
could have policy which is much more conservative in merging, than it is in 
splitting
 - regarding the load..What's there in ServerLoad and RegionLoad won't suffice 
you think? Would it help it we grab some OS-level info in ServerLoad (or 
similar class)?

 [ergonomics] add region size balancing as a feature of master
 -

 Key: HBASE-13103
 URL: https://issues.apache.org/jira/browse/HBASE-13103
 Project: HBase
  Issue Type: Brainstorming
  Components: Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
 Fix For: 2.0.0, 1.1.0

 Attachments: HBASE-13103-v0.patch


 Often enough, folks miss-judge split points or otherwise end up with a 
 suboptimal number of regions. We should have an automated, reliable way to 
 reshape or balance a table's region boundaries. This would be for tables 
 that contain existing data. This might look like:
 {noformat}
 Admin#reshapeTable(TableName, int numSplits);
 {noformat}
 or from the shell:
 {noformat}
  reshape TABLE, numSplits
 {noformat}
 Better still would be to have a maintenance process, similar to the existing 
 Balancer that runs AssignmentManager on an interval, to run the above 
 reshape operation on an interval. That way, the cluster will automatically 
 self-correct toward a desirable state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master

2015-04-09 Thread Lars Hofhansl (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486841#comment-14486841
]

Lars Hofhansl commented on HBASE-13103:
---

Yeah, meant the reshaping after I identified that something is odd/bad about
a table.
But maybe it's better to just automate, otherwise nobody would use it, as you
say.

Splits already happen automatically with nice simple local-only logic do we
need more logic for those? (but we could get rid of
IncreasingToUpperBoundRegionSplitPolicy and combine it all in one class, which
would be nice).

bq. could have policy which is much more conservative in merging, than it is in
splitting
I think that'd be nice. With IncreasingToUpperBoundRegionSplitPolicy it's
possible that we get a 2x size difference between regions for a bit. Hard to
say whether a region will be written to in the future, and avoid an early
merge. Maybe we can track the age of a region? And then favor older regions for
merges unless they're hot...

bq. ServerLoad and RegionLoad won't suffice you think?
You're right, that's probably all the information we need. And if not, we'd add
it.

[ergonomics] add region size balancing as a feature of master
-

Attachments: HBASE-13103-v0.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master