[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15102602#comment-15102602 ] Mikhail Antonov commented on HBASE-15065: - [~te...@apache.org] do you want to prevent situation when too many splits are triggered at the same time? > SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15065-v1.txt, 15065-v2.txt, 15065.addendum > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15102621#comment-15102621 ] Ted Yu commented on HBASE-15065: Yes, we should. Do we use number of regions in transition as a metric for throttling decision making ? Any suggestion ? > SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15065-v1.txt, 15065-v2.txt, 15065.addendum > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092820#comment-15092820 ] Ted Yu commented on HBASE-15065: Currently normalizer chore would run all the normalization plans. I think we should provide some throttling in certain scenarios. e.g. when normalization is enabled for an existing table which has many (close to) empty regions to be merged or, at cluster startup, the number of normalization plans for all the tables in the cluster could be quite high Throttling in the above scenarios would make the cluster more stable. [~enis] [~mantonov]: What's your opinion ? > SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15065-v1.txt, 15065-v2.txt, 15065.addendum > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[
https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090122#comment-15090122
]
Ted Yu commented on HBASE-15065:
Looks like test environment had issue:
{code}
Running org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 22.11 sec <<<
FAILURE! - in org.apache.hadoop.hbase.mapred.TestTableMapReduce
testCombiner(org.apache.hadoop.hbase.mapred.TestTableMapReduce) Time elapsed:
6.884 sec <<< ERROR!
java.io.IOException: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: Error while running command to get file permissions
: ExitCodeException exitCode=127: /bin/ls: error while loading shared
libraries: libselinux.so.1: failed to map segment from shared object:
Permission denied
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:815)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:798)
{code}
> SimpleRegionNormalizer should return multiple normalization plans in one run
>
>
> Key: HBASE-15065
> URL: https://issues.apache.org/jira/browse/HBASE-15065
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 15065-v1.txt, 15065-v2.txt, 15065.addendum
>
>
> This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer
> Outline for enhancements:
> * adjustment to the period when SimpleRegionNormalizer runs
> * explore merge opportunities among all neighboring region pairs
> * return multiple normalization plans
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090252#comment-15090252 ] Hudson commented on HBASE-15065: SUCCESS: Integrated in HBase-1.2-IT #386 (See [https://builds.apache.org/job/HBase-1.2-IT/386/]) HBASE-15065 SimpleRegionNormalizer should return multiple normalization (tedyu: rev a5438f4a55bc6ff56a1eded712c567141f679600) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizer.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizerOnCluster.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizerChore.java HBASE-15065 Addendum fixes checkstyle warnings (tedyu: rev 2e08227b4b337e45dfb8282c3c657b95061073c1) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java > SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15065-v1.txt, 15065-v2.txt, 15065.addendum > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090242#comment-15090242 ] Hudson commented on HBASE-15065: FAILURE: Integrated in HBase-Trunk_matrix #622 (See [https://builds.apache.org/job/HBase-Trunk_matrix/622/]) HBASE-15065 SimpleRegionNormalizer should return multiple normalization (tedyu: rev 6e781a1f342de069868a8a1f95f6a403a6d4c499) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizerOnCluster.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizerChore.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java > SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15065-v1.txt, 15065-v2.txt, 15065.addendum > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090070#comment-15090070 ] Hudson commented on HBASE-15065: FAILURE: Integrated in HBase-1.3 #488 (See [https://builds.apache.org/job/HBase-1.3/488/]) HBASE-15065 SimpleRegionNormalizer should return multiple normalization (tedyu: rev 5a0658377d7a496767acb7f32b393a42798b40df) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizerChore.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizerOnCluster.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizer.java > SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15065-v1.txt, 15065-v2.txt > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090114#comment-15090114 ] Ted Yu commented on HBASE-15065: TestDeleteTableProcedure test failure was not related to the patch - normalization is not enabled for the table to be tested. Checkstyle warnings would be addressed in addendum. > SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15065-v1.txt, 15065-v2.txt > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089055#comment-15089055 ] Enis Soztutar commented on HBASE-15065: --- bq. Haven't yet read the patch, but regarding supporting the old interface - HBASE-13103 added it since 1.2 onwards, so it's first GA release it would go in, no backward compatibility issues Good to know. bq. One factor we should consider is that normalization should't cause burst of region splitting / merging activities. I would prefer the execution to be serial. Alright, lets start with this. If in practice we need to execute multiple plans in parallel, we can do a follow up. bq. I think we don't need to compute new average for each iteration. Sounds good. +1 for the patch. > SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15065-v1.txt > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[
https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090088#comment-15090088
]
Hadoop QA commented on HBASE-15065:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 7s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/latest/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 4s
{color} | {color:red} hbase-server in master has 85 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 24s
{color} | {color:red} Patch generated 3 new checkstyle issues in hbase-server
(total was 58, now 61). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 18s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 31s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 41s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 239m 15s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_91 Failed junit tests |
hadoop.hbase.master.procedure.TestDeleteTableProcedure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090187#comment-15090187 ] Hudson commented on HBASE-15065: SUCCESS: Integrated in HBase-1.3-IT #430 (See [https://builds.apache.org/job/HBase-1.3-IT/430/]) HBASE-15065 SimpleRegionNormalizer should return multiple normalization (tedyu: rev 5a0658377d7a496767acb7f32b393a42798b40df) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizerOnCluster.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizerChore.java > SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15065-v1.txt, 15065-v2.txt, 15065.addendum > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090338#comment-15090338 ] Hudson commented on HBASE-15065: SUCCESS: Integrated in HBase-1.2 #497 (See [https://builds.apache.org/job/HBase-1.2/497/]) HBASE-15065 SimpleRegionNormalizer should return multiple normalization (tedyu: rev a5438f4a55bc6ff56a1eded712c567141f679600) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizerOnCluster.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizerChore.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java HBASE-15065 Addendum fixes checkstyle warnings (tedyu: rev 2e08227b4b337e45dfb8282c3c657b95061073c1) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java > SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15065-v1.txt, 15065-v2.txt, 15065.addendum > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090402#comment-15090402 ] Hudson commented on HBASE-15065: FAILURE: Integrated in HBase-Trunk_matrix #623 (See [https://builds.apache.org/job/HBase-Trunk_matrix/623/]) HBASE-15065 Addendum fixes checkstyle warnings (tedyu: rev 4a26644245bad6a661442f0eede6850e12d07cc9) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java > SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15065-v1.txt, 15065-v2.txt, 15065.addendum > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090367#comment-15090367 ] Hudson commented on HBASE-15065: SUCCESS: Integrated in HBase-1.3 #489 (See [https://builds.apache.org/job/HBase-1.3/489/]) HBASE-15065 Addendum fixes checkstyle warnings (tedyu: rev d277c153309c1ad5d5b08820c5dd1f6038f6ab2b) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java > SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15065-v1.txt, 15065-v2.txt, 15065.addendum > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090372#comment-15090372 ] Hudson commented on HBASE-15065: SUCCESS: Integrated in HBase-1.3-IT #431 (See [https://builds.apache.org/job/HBase-1.3-IT/431/]) HBASE-15065 Addendum fixes checkstyle warnings (tedyu: rev d277c153309c1ad5d5b08820c5dd1f6038f6ab2b) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java > SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15065-v1.txt, 15065-v2.txt, 15065.addendum > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086396#comment-15086396 ] Mikhail Antonov commented on HBASE-15065: - On the patch.. - do we not need regionSizeComparator anymore, and Collections.sort(regionsWithSize, regionSizeComparator) before we consider candidates for the merge? Now we just merge any two adjacent (since the original list of regions is sorted the way so they are all adjacent, right?) regions where r1+r2 SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15065-v1.txt > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086422#comment-15086422 ] Ted Yu commented on HBASE-15065: Thanks for detailed comment. bq. with this logic we merge 2 out of 3 10-size regions and split 60 After merging 2 of the size 10 regions, average becomes 90 / 3 = 30. So both the merge and split are still fine. If size 60 region gets split first, average becomes 90 / 5 = 18. The merging doesn't seem necessary. However, size 60 region comes last. Meaning, potentially the normalization process may have O(N) complexity where N is the number of regions in the table. I think we don't need to compute new average for each iteration. bq. we can run them concurrently more than 1 at a time The splitting / merging request is asynchronous. So by executing multiple plans in one round, the above is achieved. > SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15065-v1.txt > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086520#comment-15086520 ] Mikhail Antonov commented on HBASE-15065: - Fair enough. I think avg computation logic is part of implementation of already private audience stuff, so should be easy to modify even in patch release. If issuing async requests turns out to be bottleneck we can fix it later, now I can't really judge. > SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15065-v1.txt > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086188#comment-15086188 ] Mikhail Antonov commented on HBASE-15065: - Haven't yet read the patch, but regarding supporting the old interface - HBASE-13103 added it since 1.2 onwards, so it's first GA release it would go in, no backward compatibility issues :) > SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15065-v1.txt > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082820#comment-15082820 ] Enis Soztutar commented on HBASE-15065: --- Thanks Ted for working on this. This approach looks more realistic and useful for the general case. A couple of questions: - Do we have to support the old RegionNormalizer interface? Sorry I did not check whether it is a client-public interface or not. - Executing the region normalization plans is serial. If we end up with hundreds of plans, will this become a bottleneck? - Why sort the splits plans before merge plans for execution? [~mantonov] what do you think? > SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15065-v1.txt > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083267#comment-15083267 ] Ted Yu commented on HBASE-15065: RegionNormalizer is marked Private. bq. If we end up with hundreds of plans One factor we should consider is that normalization should't cause burst of region splitting / merging activities. I would prefer the execution to be serial. bq. Why sort the splits plans before merge plans for execution? The original patch from Mikhail gave more weight to splitting than to merging. The above is consistent with that practice. > SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15065-v1.txt > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[
https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082548#comment-15082548
]
Hadoop QA commented on HBASE-15065:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12780463/15065-v1.txt
against master branch at commit 9997e4ee52136a681faa825dea46bd1162d71eca.
ATTACHMENT ID: 12780463
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
new checkstyle errors. Check build console for list of new errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn post-site goal
to fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/17121//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/17121//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/17121//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/17121//console
This message is automatically generated.
> SimpleRegionNormalizer should return multiple normalization plans in one run
>
>
> Key: HBASE-15065
> URL: https://issues.apache.org/jira/browse/HBASE-15065
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 15065-v1.txt
>
>
> This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer
> Outline for enhancements:
> * adjustment to the period when SimpleRegionNormalizer runs
> * explore merge opportunities among all neighboring region pairs
> * return multiple normalization plans
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
