[jira] [Commented] (HBASE-16832) Reduce the default number of versions in Meta table for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15577239#comment-15577239 ] binlijin commented on HBASE-16832: -- Thanks all for the review and commit. > Reduce the default number of versions in Meta table for branch-1 > > > Key: HBASE-16832 > URL: https://issues.apache.org/jira/browse/HBASE-16832 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.3.0 >Reporter: binlijin >Assignee: binlijin > Fix For: 1.4.0 > > Attachments: 16832.branch-1.patch, HBASE-16832.branch-1.patch, > rpc-handler.png, rpc_processingCallTime.png, rpc_processingCallTimeV2.png, > rpc_qps.png, rpc_queueLength.png, rpc_queueSize.png, rpc_scan_latency.png, > rpc_scan_latencyV2.png, rpc_totalcalltime.png, rpc_totalcalltimeV2.png > > > I find the DEFAULT_HBASE_META_VERSIONS is still 10 in branch-1, and in master > version DEFAULT_HBASE_META_VERSIONS is 3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16832) Reduce the default number of versions in Meta table for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15577161#comment-15577161 ] Hudson commented on HBASE-16832: FAILURE: Integrated in Jenkins build HBase-1.4 #469 (See [https://builds.apache.org/job/HBase-1.4/469/]) HBASE-16832 Reduce the default number of versions in Meta table for (tedyu: rev 05b010cacb83a6449c205393e3c15b048b84b0a2) * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java > Reduce the default number of versions in Meta table for branch-1 > > > Key: HBASE-16832 > URL: https://issues.apache.org/jira/browse/HBASE-16832 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.3.0 >Reporter: binlijin >Assignee: binlijin > Fix For: 1.4.0 > > Attachments: 16832.branch-1.patch, HBASE-16832.branch-1.patch, > rpc-handler.png, rpc_processingCallTime.png, rpc_processingCallTimeV2.png, > rpc_qps.png, rpc_queueLength.png, rpc_queueSize.png, rpc_scan_latency.png, > rpc_scan_latencyV2.png, rpc_totalcalltime.png, rpc_totalcalltimeV2.png > > > I find the DEFAULT_HBASE_META_VERSIONS is still 10 in branch-1, and in master > version DEFAULT_HBASE_META_VERSIONS is 3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16832) Reduce the default number of versions in Meta table for branch-1
[
https://issues.apache.org/jira/browse/HBASE-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575996#comment-15575996
]
Hadoop QA commented on HBASE-16832:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 30m 23s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
50s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 42s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 2s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 54s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.1 Server=1.12.1 Image:yetus/hbase:b2c5d84 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12833404/16832.branch-1.patch |
| JIRA Issue | HBASE-16832 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 81327df9d105 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
| git revision | branch-1 / 57d3e9e |
[jira] [Commented] (HBASE-16832) Reduce the default number of versions in Meta table for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573956#comment-15573956 ] Heng Chen commented on HBASE-16832: --- Thanks [~aoxiang]. +1 for it. > Reduce the default number of versions in Meta table for branch-1 > > > Key: HBASE-16832 > URL: https://issues.apache.org/jira/browse/HBASE-16832 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.3.0 >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-16832.branch-1.patch, rpc-handler.png, > rpc_processingCallTime.png, rpc_processingCallTimeV2.png, rpc_qps.png, > rpc_queueLength.png, rpc_queueSize.png, rpc_scan_latency.png, > rpc_scan_latencyV2.png, rpc_totalcalltime.png, rpc_totalcalltimeV2.png > > > I find the DEFAULT_HBASE_META_VERSIONS is still 10 in branch-1, and in master > version DEFAULT_HBASE_META_VERSIONS is 3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16832) Reduce the default number of versions in Meta table for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573944#comment-15573944 ] binlijin commented on HBASE-16832: -- HBASE-12016 set DEFAULT_HBASE_META_VERSIONS to 3 on master. And HBASE-12204 backport to branch-1 miss it. > Reduce the default number of versions in Meta table for branch-1 > > > Key: HBASE-16832 > URL: https://issues.apache.org/jira/browse/HBASE-16832 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.3.0 >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-16832.branch-1.patch, rpc-handler.png, > rpc_processingCallTime.png, rpc_processingCallTimeV2.png, rpc_qps.png, > rpc_queueLength.png, rpc_queueSize.png, rpc_scan_latency.png, > rpc_scan_latencyV2.png, rpc_totalcalltime.png, rpc_totalcalltimeV2.png > > > I find the DEFAULT_HBASE_META_VERSIONS is still 10 in branch-1, and in master > version DEFAULT_HBASE_META_VERSIONS is 3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16832) Reduce the default number of versions in Meta table for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573940#comment-15573940 ] binlijin commented on HBASE-16832: -- Upload rpc_totalcalltimeV2.png，you can see it in this PNG. > Reduce the default number of versions in Meta table for branch-1 > > > Key: HBASE-16832 > URL: https://issues.apache.org/jira/browse/HBASE-16832 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.3.0 >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-16832.branch-1.patch, rpc-handler.png, > rpc_processingCallTime.png, rpc_processingCallTimeV2.png, rpc_qps.png, > rpc_queueLength.png, rpc_queueSize.png, rpc_scan_latency.png, > rpc_scan_latencyV2.png, rpc_totalcalltime.png, rpc_totalcalltimeV2.png > > > I find the DEFAULT_HBASE_META_VERSIONS is still 10 in branch-1, and in master > version DEFAULT_HBASE_META_VERSIONS is 3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16832) Reduce the default number of versions in Meta table for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573922#comment-15573922 ] Heng Chen commented on HBASE-16832: --- [~te...@apache.org] do you know why DEFAULT_HBASE_META_VERSIONS is 10 for branch-1 and set back to be 3 on master? > Reduce the default number of versions in Meta table for branch-1 > > > Key: HBASE-16832 > URL: https://issues.apache.org/jira/browse/HBASE-16832 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.3.0 >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-16832.branch-1.patch, rpc-handler.png, > rpc_processingCallTime.png, rpc_qps.png, rpc_scan_latency.png, > rpc_totalcalltime.png > > > I find the DEFAULT_HBASE_META_VERSIONS is still 10 in branch-1, and in master > version DEFAULT_HBASE_META_VERSIONS is 3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16832) Reduce the default number of versions in Meta table for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573916#comment-15573916 ] Ted Yu commented on HBASE-16832: +1 > Reduce the default number of versions in Meta table for branch-1 > > > Key: HBASE-16832 > URL: https://issues.apache.org/jira/browse/HBASE-16832 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.3.0 >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-16832.branch-1.patch, rpc-handler.png, > rpc_processingCallTime.png, rpc_qps.png, rpc_scan_latency.png, > rpc_totalcalltime.png > > > I find the DEFAULT_HBASE_META_VERSIONS is still 10 in branch-1, and in master > version DEFAULT_HBASE_META_VERSIONS is 3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16832) Reduce the default number of versions in Meta table for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573912#comment-15573912 ] Ted Yu commented on HBASE-16832: In rpc_totalcalltime.png, I don't see the green and yellow lines. > Reduce the default number of versions in Meta table for branch-1 > > > Key: HBASE-16832 > URL: https://issues.apache.org/jira/browse/HBASE-16832 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.3.0 >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-16832.branch-1.patch, rpc-handler.png, > rpc_processingCallTime.png, rpc_qps.png, rpc_scan_latency.png, > rpc_totalcalltime.png > > > I find the DEFAULT_HBASE_META_VERSIONS is still 10 in branch-1, and in master > version DEFAULT_HBASE_META_VERSIONS is 3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16832) Reduce the default number of versions in Meta table for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573841#comment-15573841 ] binlijin commented on HBASE-16832: -- Yes. > Reduce the default number of versions in Meta table for branch-1 > > > Key: HBASE-16832 > URL: https://issues.apache.org/jira/browse/HBASE-16832 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.3.0 >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-16832.branch-1.patch, rpc-handler.png, > rpc_processingCallTime.png, rpc_qps.png, rpc_scan_latency.png, > rpc_totalcalltime.png > > > I find the DEFAULT_HBASE_META_VERSIONS is still 10 in branch-1, and in master > version DEFAULT_HBASE_META_VERSIONS is 3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16832) Reduce the default number of versions in Meta table for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573810#comment-15573810 ] Heng Chen commented on HBASE-16832: --- The default value of major_compaction for your cluster is 7 days? > Reduce the default number of versions in Meta table for branch-1 > > > Key: HBASE-16832 > URL: https://issues.apache.org/jira/browse/HBASE-16832 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.3.0 >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-16832.branch-1.patch, rpc-handler.png, > rpc_processingCallTime.png, rpc_qps.png, rpc_scan_latency.png, > rpc_totalcalltime.png > > > I find the DEFAULT_HBASE_META_VERSIONS is still 10 in branch-1, and in master > version DEFAULT_HBASE_META_VERSIONS is 3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16832) Reduce the default number of versions in Meta table for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573671#comment-15573671 ] binlijin commented on HBASE-16832: -- The cluster have 600+ machines and 260k+ regions. > Reduce the default number of versions in Meta table for branch-1 > > > Key: HBASE-16832 > URL: https://issues.apache.org/jira/browse/HBASE-16832 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.3.0 >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-16832.branch-1.patch, rpc-handler.png, > rpc_processingCallTime.png, rpc_qps.png, rpc_scan_latency.png, > rpc_totalcalltime.png > > > I find the DEFAULT_HBASE_META_VERSIONS is still 10 in branch-1, and in master > version DEFAULT_HBASE_META_VERSIONS is 3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16832) Reduce the default number of versions in Meta table for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573664#comment-15573664 ] binlijin commented on HBASE-16832: -- The qps from less 3K to 170K. > Reduce the default number of versions in Meta table for branch-1 > > > Key: HBASE-16832 > URL: https://issues.apache.org/jira/browse/HBASE-16832 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.3.0 >Reporter: binlijin >Assignee: binlijin > Attachments: rpc-handler.png, rpc_processingCallTime.png, > rpc_qps.png, rpc_scan_latency.png, rpc_totalcalltime.png > > > I find the DEFAULT_HBASE_META_VERSIONS is still 10 in branch-1, and in master > version DEFAULT_HBASE_META_VERSIONS is 3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16832) Reduce the default number of versions in Meta table for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573658#comment-15573658 ] binlijin commented on HBASE-16832: -- The meta region size before major_compact is 1.01 GB and after major_compact is132 MB. > Reduce the default number of versions in Meta table for branch-1 > > > Key: HBASE-16832 > URL: https://issues.apache.org/jira/browse/HBASE-16832 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.3.0 >Reporter: binlijin >Assignee: binlijin > Attachments: rpc-handler.png, rpc_processingCallTime.png, > rpc_qps.png, rpc_scan_latency.png, rpc_totalcalltime.png > > > I find the DEFAULT_HBASE_META_VERSIONS is still 10 in branch-1, and in master > version DEFAULT_HBASE_META_VERSIONS is 3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16832) Reduce the default number of versions in Meta table for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573650#comment-15573650 ] binlijin commented on HBASE-16832: -- The major_compact done after 20min, because there is no chance to be executed. > Reduce the default number of versions in Meta table for branch-1 > > > Key: HBASE-16832 > URL: https://issues.apache.org/jira/browse/HBASE-16832 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.3.0 >Reporter: binlijin >Assignee: binlijin > Attachments: rpc-handler.png, rpc_processingCallTime.png, > rpc_qps.png, rpc_scan_latency.png, rpc_totalcalltime.png > > > I find the DEFAULT_HBASE_META_VERSIONS is still 10 in branch-1, and in master > version DEFAULT_HBASE_META_VERSIONS is 3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16832) Reduce the default number of versions in Meta table for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573644#comment-15573644 ] binlijin commented on HBASE-16832: -- When one of our production cluster encounter the meta performance problem, many of the getRegionLocation calls timeout. First i pick up a regionserver and change hbase.meta.versions=1 and hbase.regionserver.metahandler.count=90. Then restart the regionserver and move meta region to this regionserver, so there is only the meta region on this regionserver, then major_compact the meta table. hbase(main):002:0> major_compact 'hbase:meta' 0 row(s) in 1284.1590 seconds hbase(main):003:0> > Reduce the default number of versions in Meta table for branch-1 > > > Key: HBASE-16832 > URL: https://issues.apache.org/jira/browse/HBASE-16832 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.3.0 >Reporter: binlijin >Assignee: binlijin > Attachments: rpc-handler.png, rpc_processingCallTime.png, > rpc_qps.png, rpc_scan_latency.png, rpc_totalcalltime.png > > > I find the DEFAULT_HBASE_META_VERSIONS is still 10 in branch-1, and in master > version DEFAULT_HBASE_META_VERSIONS is 3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16832) Reduce the default number of versions in Meta table for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573610#comment-15573610 ] binlijin commented on HBASE-16832: -- We have encounter the meta performance problem. And i think the fast way to improve the meta performance is set DEFAULT_HBASE_META_VERSIONS=1 and constantly major_compact the meta. I also have the number of the performance. > Reduce the default number of versions in Meta table for branch-1 > > > Key: HBASE-16832 > URL: https://issues.apache.org/jira/browse/HBASE-16832 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.3.0 >Reporter: binlijin >Assignee: binlijin > > I find the DEFAULT_HBASE_META_VERSIONS is still 10 in branch-1, and in master > version DEFAULT_HBASE_META_VERSIONS is 3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
