[jira] [Commented] (HBASE-15169) Backport HBASE-14362 'TestWALProcedureStoreOnHDFS is super duper flaky' to branch-1.1
[
https://issues.apache.org/jira/browse/HBASE-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120359#comment-15120359
]
Hadoop QA commented on HBASE-15169:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} branch-1.1 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} branch-1.1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 50s
{color} | {color:red} hbase-server in branch-1.1 has 80 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 26s
{color} | {color:red} hbase-server in branch-1.1 failed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 3m
55s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 24s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 53s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 101m 38s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 131m 7s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.hbase.http.TestHttpServerLifecycle |
| JDK v1.7.0_91 Timed out junit tests |
org.apache.hadoop.hbase.namespace.TestNamespaceAuditor |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-01-27 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12784716/HBASE-15169-branch-1.1.patch
|
| JIRA Issue | HBASE-15169 |
| Optional Tests | asflicense javac javadoc unit findbugs
[jira] [Commented] (HBASE-15177) Reduce garbage created under high load
[ https://issues.apache.org/jira/browse/HBASE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120458#comment-15120458 ] stack commented on HBASE-15177: --- Nice one [~enis] Yeah, I remember CIS 4k buffer and going out of our way to avoid its allocation. That it is unavoidable for BB is interesting. bq. Maybe we need to re-write CodedInputStream? What HBaseZeroCopyByteString does for LiteralByteString? (CIS looks open, subclassable? ... but the static methods are probably used all over pb.. haven't looked). bq. AnnotationReadingPriorityFunction This thing has been a pain since day one. If you are able to kill it, good riddance (smile). > Reduce garbage created under high load > -- > > Key: HBASE-15177 > URL: https://issues.apache.org/jira/browse/HBASE-15177 > Project: HBase > Issue Type: Improvement >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0 > > Attachments: Screen Shot 2016-01-26 at 10.03.48 PM.png, Screen Shot > 2016-01-26 at 10.03.56 PM.png, Screen Shot 2016-01-26 at 10.06.16 PM.png, > Screen Shot 2016-01-26 at 10.15.15 PM.png, hbase-15177_v0.patch > > > I have been doing some profiling of the garbage being created. The idea was > to follow up on HBASE-14490 and experiment with offheap IPC byte buffers and > byte buffer re-use. However, without changing the IPC byte buffers for now, > there are a couple of (easy) improvements that I've identified from > profiling: > 1. RPCServer.Connection.processRequest() should work with ByteBuffer instead > of byte[] and not-recreate CodedInputStream a few times. > 2. RSRpcServices.getRegion() allocates two byte arrays for region, while only > 1 is needed. > 3. AnnotationReadingPriorityFunction is very expensive in allocations. Mainly > it allocates the regionName byte[] to get the table name. We already set the > priority for most of the operations (multi, get, increment, etc) but we are > only reading the priority in case of multi. We should use the priority from > the client side. > Lets do the simple improvements in this patch, we can get to IPC buffer > re-use in HBASE-14490. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15135) Add metrics for storefile age
[
https://issues.apache.org/jira/browse/HBASE-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120555#comment-15120555
]
Hadoop QA commented on HBASE-15135:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
7s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
51s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 8s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 11s
{color} | {color:red} hbase-hadoop2-compat in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 27s
{color} | {color:red} Patch generated 1 new checkstyle issues in hbase-server
(total was 145, now 145). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 8s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s
{color} | {color:green} hbase-hadoop2-compat in the patch passed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s
{color} | {color:green} hbase-hadoop-compat in the patch passed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 32s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s
{color} | {color:green} hbase-hadoop2-compat in the patch passed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s
{color} | {color:green} hbase-hadoop-compat in the patch passed with JDK
v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 48s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green}
[jira] [Commented] (HBASE-15135) Add metrics for storefile age
[ https://issues.apache.org/jira/browse/HBASE-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120564#comment-15120564 ] Mikhail Antonov commented on HBASE-15135: - Forgot to add flag to enforce ordering :( re-queuing. Seems like we can't manually run the same patch twice though. > Add metrics for storefile age > - > > Key: HBASE-15135 > URL: https://issues.apache.org/jira/browse/HBASE-15135 > Project: HBase > Issue Type: New Feature >Reporter: Elliott Clark >Assignee: Mikhail Antonov > Attachments: HBASE-15135-v2.patch, HBASE-15135-v3.patch, > HBASE-15135.patch > > > In order to make sure that compactions are fully up to date it would be nice > to have metrics on: > * Max storefile age > * Min storefile age > * Average storefile age > * Number of reference files > If we could have those metrics per region and per regionserver it would be > awesome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15135) Add metrics for storefile age
[ https://issues.apache.org/jira/browse/HBASE-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120583#comment-15120583 ] Mikhail Antonov commented on HBASE-15135: - [~busbey] hmm.. could it be that the patch gets stuck somehow and can't be run, even if renamed? I try to restart the patch manually but it fails immediately. [EnvInject] - Variables injected successfully. [PreCommit-HBASE-Build] $ /bin/bash -e /tmp/hudson920775254521547136.sh [WARN] patch process already existed '/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess' % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 1680 1680 0450 0 --:--:-- --:--:-- --:--:-- 451 tar: This does not look like a tar archive > Add metrics for storefile age > - > > Key: HBASE-15135 > URL: https://issues.apache.org/jira/browse/HBASE-15135 > Project: HBase > Issue Type: New Feature >Reporter: Elliott Clark >Assignee: Mikhail Antonov > Attachments: HBASE-15135-v2.patch, HBASE-15135-v3.patch, > HBASE-15135-v4.patch, HBASE-15135.patch > > > In order to make sure that compactions are fully up to date it would be nice > to have metrics on: > * Max storefile age > * Min storefile age > * Average storefile age > * Number of reference files > If we could have those metrics per region and per regionserver it would be > awesome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15173) Execute mergeRegions RPC call as the request user
[
https://issues.apache.org/jira/browse/HBASE-15173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120451#comment-15120451
]
Hadoop QA commented on HBASE-15173:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 50s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 11s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 45s
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 13s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 1s
{color} | {color:green} hbase-client in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 87m 52s
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 229m 15s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Timed out junit tests |
org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster |
| JDK v1.7.0_91 Timed out junit tests |
[jira] [Commented] (HBASE-15128) Disable region splits and merges in HBCK
[ https://issues.apache.org/jira/browse/HBASE-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120552#comment-15120552 ] Matteo Bertozzi commented on HBASE-15128: - what are all these Tracker that we are adding? and why are we using zk to store persistent data when we said that zk should be used only for non transient states and the only exception is replication (and we are trying to get rid of that)? can't we just merge this with the RegionNormalizerTracker? or if you want to make a generic dynamic configuration system, why don't pick up the dynamic configuration work in HBASE-13936? > Disable region splits and merges in HBCK > > > Key: HBASE-15128 > URL: https://issues.apache.org/jira/browse/HBASE-15128 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15128.patch, HBASE-15128_v1.patch, > HBASE-15128_v3.patch > > > In large clusters where region splits are frequent, and HBCK runs take > longer, the concurrent splits cause further problems in HBCK since HBCK > assumes a static state for the region partition map. We have just seen a case > where HBCK undo's a concurrently splitting region causing number of > inconsistencies to go up. > We can have a mode in master where splits and merges are disabled like the > balancer and catalog janitor switches. Master will reject the split requests > if regionservers decide to split. This switch can be turned on / off by the > admins and also automatically by HBCK while it is running (similar to > balancer switch being disabled by HBCK). > HBCK should also disable the Catalog Janitor just in case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15159) Fix merge of MVCC and SequenceID performance regression in branch-1.0 for checkAnd*
[ https://issues.apache.org/jira/browse/HBASE-15159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120267#comment-15120267 ] Heng Chen commented on HBASE-15159: --- If you have not start, i can take it. [~stack] :) PS: What about append? Does it need same trick? > Fix merge of MVCC and SequenceID performance regression in branch-1.0 for > checkAnd* > --- > > Key: HBASE-15159 > URL: https://issues.apache.org/jira/browse/HBASE-15159 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack > > Do the HBASE-15031 trick -- loosen reliance on mvcc for increments -- for > checkAnd* too in branch-1. Should be quick enough to do. Only prereq would > be HBASE-15157, tooling we have to do anyways, so we can show we've made > improvement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15135) Add metrics for storefile age
[ https://issues.apache.org/jira/browse/HBASE-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-15135: Status: Patch Available (was: Open) > Add metrics for storefile age > - > > Key: HBASE-15135 > URL: https://issues.apache.org/jira/browse/HBASE-15135 > Project: HBase > Issue Type: New Feature >Reporter: Elliott Clark >Assignee: Mikhail Antonov > Attachments: HBASE-15135-v2.patch, HBASE-15135-v3.patch, > HBASE-15135.patch > > > In order to make sure that compactions are fully up to date it would be nice > to have metrics on: > * Max storefile age > * Min storefile age > * Average storefile age > * Number of reference files > If we could have those metrics per region and per regionserver it would be > awesome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15159) Fix merge of MVCC and SequenceID performance regression in branch-1.0 for checkAnd* and Append
[ https://issues.apache.org/jira/browse/HBASE-15159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120366#comment-15120366 ] stack commented on HBASE-15159: --- Yeah, needs same trick. Was going to try and do the perf testing tool first so could tell if are making things better or not. HBASE-15157 > Fix merge of MVCC and SequenceID performance regression in branch-1.0 for > checkAnd* and Append > -- > > Key: HBASE-15159 > URL: https://issues.apache.org/jira/browse/HBASE-15159 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack > > Do the HBASE-15031 trick -- loosen reliance on mvcc for increments -- for > checkAnd* too in branch-1. Should be quick enough to do. Only prereq would > be HBASE-15157, tooling we have to do anyways, so we can show we've made > improvement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-15159) Fix merge of MVCC and SequenceID performance regression in branch-1.0 for checkAnd*
[ https://issues.apache.org/jira/browse/HBASE-15159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-15159: -- Comment: was deleted (was: what about append? Does it need same trick? If you have not start for this issue, i can take it. [~stack]) > Fix merge of MVCC and SequenceID performance regression in branch-1.0 for > checkAnd* > --- > > Key: HBASE-15159 > URL: https://issues.apache.org/jira/browse/HBASE-15159 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack > > Do the HBASE-15031 trick -- loosen reliance on mvcc for increments -- for > checkAnd* too in branch-1. Should be quick enough to do. Only prereq would > be HBASE-15157, tooling we have to do anyways, so we can show we've made > improvement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15173) Execute mergeRegions RPC call as the request user
[ https://issues.apache.org/jira/browse/HBASE-15173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15173: --- Attachment: HBASE-15173.v3.patch > Execute mergeRegions RPC call as the request user > - > > Key: HBASE-15173 > URL: https://issues.apache.org/jira/browse/HBASE-15173 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15173.v1.patch, HBASE-15173.v2.patch, > HBASE-15173.v2.patch, HBASE-15173.v3.patch, HBASE-15173.v3.patch, > HBASE-15173.v3.patch, HBASE-15173.v3.patch > > > This is follow up to HBASE-15132 > Master currently sends mergeRegions RPC to region server under user 'hbase'. > This issue is to execute mergeRegions RPC call as the request user > See tail of HBASE-15132 for related discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15135) Add metrics for storefile age
[ https://issues.apache.org/jira/browse/HBASE-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-15135: Attachment: HBASE-15135-v4.patch > Add metrics for storefile age > - > > Key: HBASE-15135 > URL: https://issues.apache.org/jira/browse/HBASE-15135 > Project: HBase > Issue Type: New Feature >Reporter: Elliott Clark >Assignee: Mikhail Antonov > Attachments: HBASE-15135-v2.patch, HBASE-15135-v3.patch, > HBASE-15135-v4.patch, HBASE-15135.patch > > > In order to make sure that compactions are fully up to date it would be nice > to have metrics on: > * Max storefile age > * Min storefile age > * Average storefile age > * Number of reference files > If we could have those metrics per region and per regionserver it would be > awesome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15171) Avoid counting duplicate kv and generating lots of small hfiles in PutSortReducer
[
https://issues.apache.org/jira/browse/HBASE-15171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120292#comment-15120292
]
Hudson commented on HBASE-15171:
FAILURE: Integrated in HBase-1.3 #517 (See
[https://builds.apache.org/job/HBase-1.3/517/])
HBASE-15171 Avoid counting duplicate kv and generating lots of small (tedyu:
rev 630ad95c923f642d006274b9b1a14397a6713412)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/PutSortReducer.java
> Avoid counting duplicate kv and generating lots of small hfiles in
> PutSortReducer
> -
>
> Key: HBASE-15171
> URL: https://issues.apache.org/jira/browse/HBASE-15171
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0, 1.1.2, 0.98.17
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-15171.patch, HBASE-15171.patch, HBASE-15171.patch
>
>
> Once there was one of our online user writing huge number of duplicated kvs
> during bulkload, and we found it generated lots of small hfiles and slows
> down the whole process.
> After debugging, we found in PutSortReducer#reduce, although it already tried
> to handle the pathological case by setting a threshold for single-row size
> and having a TreeMap to avoid writing out duplicated kv, it forgot to exclude
> duplicated kv from the accumulated size. As shown in below code segment:
> {code}
> while (iter.hasNext() && curSize < threshold) {
> Put p = iter.next();
> for (List cells: p.getFamilyCellMap().values()) {
> for (Cell cell: cells) {
> KeyValue kv = KeyValueUtil.ensureKeyValue(cell);
> map.add(kv);
> curSize += kv.heapSize();
> }
> }
> }
> {code}
> We should move the {{curSize += kv.heapSize();}} line out of the outer for
> loop
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15093) Replication can report incorrect size of log queue for the global source when multiwal is enabled
[
https://issues.apache.org/jira/browse/HBASE-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120318#comment-15120318
]
Ashu Pachauri commented on HBASE-15093:
---
{quote}
The failed tests seems relevant though.
{quote}
Sorry, I was busy with other stuff. The tests should not be directly related to
this patch and they pass on my system. Looking at the failures, it seems like
all of them failed due to timeouts. I will put another patch fixing the
checkstyle issues and see if tests pass this time. I wonder if timeouts in some
of these tests are too strict!!
> Replication can report incorrect size of log queue for the global source when
> multiwal is enabled
> -
>
> Key: HBASE-15093
> URL: https://issues.apache.org/jira/browse/HBASE-15093
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.2.0, 1.2.1
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
>Priority: Minor
> Attachments: HBASE-15093-V0.patch
>
>
> Replication can report incorrect size for the size of log queue for the
> global source when multiwal is enabled. This happens because the method
> MetricsSource#setSizeofLogQueue performs non-trivial operations in a
> multithreaded world, even though it is not synchronized.
> We can simply divide MetricsSource#setSizeofLogQueue into
> MetricsSource#incrSizeofLogQueue and MetricsSource#decrSizeofLogQueue. Not
> sure why we are currently directly setting the size instead of
> incrementing/decrementing it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14916) Add checkstyle_report.py to other branches
[ https://issues.apache.org/jira/browse/HBASE-14916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120556#comment-15120556 ] Appy commented on HBASE-14916: -- Seems to be working good. Closing this issue as obsolete. > Add checkstyle_report.py to other branches > -- > > Key: HBASE-14916 > URL: https://issues.apache.org/jira/browse/HBASE-14916 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: HBASE-14916-branch-1-v2.patch, > HBASE-14916-branch-1-v3.patch, HBASE-14916-branch-1.patch > > > Given test-patch.sh is always run from master, and that it now uses > checkstyle_report.py, we should pull back the script to other branches too. > Otherwise we see error like: > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/jenkins.build/dev-support/test-patch.sh: > line 662: > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/dev-support/checkstyle_report.py: > No such file or directory > [reference|https://builds.apache.org/job/PreCommit-HBASE-Build/16734//consoleFull] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15159) Fix merge of MVCC and SequenceID performance regression in branch-1.0 for checkAnd*
[ https://issues.apache.org/jira/browse/HBASE-15159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120264#comment-15120264 ] Heng Chen commented on HBASE-15159: --- what about append? Does it need same trick? If you have not start for this issue, i can take it. [~stack] > Fix merge of MVCC and SequenceID performance regression in branch-1.0 for > checkAnd* > --- > > Key: HBASE-15159 > URL: https://issues.apache.org/jira/browse/HBASE-15159 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack > > Do the HBASE-15031 trick -- loosen reliance on mvcc for increments -- for > checkAnd* too in branch-1. Should be quick enough to do. Only prereq would > be HBASE-15157, tooling we have to do anyways, so we can show we've made > improvement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15135) Add metrics for storefile age
[ https://issues.apache.org/jira/browse/HBASE-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-15135: Status: Open (was: Patch Available) > Add metrics for storefile age > - > > Key: HBASE-15135 > URL: https://issues.apache.org/jira/browse/HBASE-15135 > Project: HBase > Issue Type: New Feature >Reporter: Elliott Clark >Assignee: Mikhail Antonov > Attachments: HBASE-15135-v2.patch, HBASE-15135-v3.patch, > HBASE-15135.patch > > > In order to make sure that compactions are fully up to date it would be nice > to have metrics on: > * Max storefile age > * Min storefile age > * Average storefile age > * Number of reference files > If we could have those metrics per region and per regionserver it would be > awesome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15135) Add metrics for storefile age
[ https://issues.apache.org/jira/browse/HBASE-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-15135: Attachment: HBASE-15135-v3.patch Fixed division by 0 which was causing several tests to fail if there'a no store files in the store. > Add metrics for storefile age > - > > Key: HBASE-15135 > URL: https://issues.apache.org/jira/browse/HBASE-15135 > Project: HBase > Issue Type: New Feature >Reporter: Elliott Clark >Assignee: Mikhail Antonov > Attachments: HBASE-15135-v2.patch, HBASE-15135-v3.patch, > HBASE-15135.patch > > > In order to make sure that compactions are fully up to date it would be nice > to have metrics on: > * Max storefile age > * Min storefile age > * Average storefile age > * Number of reference files > If we could have those metrics per region and per regionserver it would be > awesome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15159) Fix merge of MVCC and SequenceID performance regression in branch-1.0 for checkAnd* and Append
[ https://issues.apache.org/jira/browse/HBASE-15159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15159: -- Summary: Fix merge of MVCC and SequenceID performance regression in branch-1.0 for checkAnd* and Append (was: Fix merge of MVCC and SequenceID performance regression in branch-1.0 for checkAnd*) > Fix merge of MVCC and SequenceID performance regression in branch-1.0 for > checkAnd* and Append > -- > > Key: HBASE-15159 > URL: https://issues.apache.org/jira/browse/HBASE-15159 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack > > Do the HBASE-15031 trick -- loosen reliance on mvcc for increments -- for > checkAnd* too in branch-1. Should be quick enough to do. Only prereq would > be HBASE-15157, tooling we have to do anyways, so we can show we've made > improvement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15128) Disable region splits and merges in HBCK
[
https://issues.apache.org/jira/browse/HBASE-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120433#comment-15120433
]
Hadoop QA commented on HBASE-15128:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 28s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 26s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 9m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 55s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 15s
{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 24s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 25s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 25s {color} |
{color:red} hbase-server in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 25s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 23s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_91.
{color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 23s {color} |
{color:red} hbase-server in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 23s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 49s
{color} | {color:red} Patch generated 4 new checkstyle issues in hbase-client
(total was 473, now 476). {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 10s
{color} | {color:red} Patch generated 4 new checkstyle issues in hbase-server
(total was 338, now 341). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m 13s
{color} | {color:red} The applied patch generated 35 new rubocop issues (total
was 828, now 860). {color} |
| {color:red}-1{color} | {color:red} ruby-lint {color} | {color:red} 0m 4s
{color} | {color:red} The applied patch generated 54 new ruby-lint issues
(total was 530, now 584). {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 28s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 7s
{color} |
[jira] [Updated] (HBASE-15171) Avoid counting duplicate kv and generating lots of small hfiles in PutSortReducer
[
https://issues.apache.org/jira/browse/HBASE-15171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-15171:
--
Attachment: HBASE-15171.addendum.patch
The addendum patch to adopt [~ram_krish]'s suggestion, thanks Ram for the note.
Thanks [~tedyu] and [~stack] for review.
> Avoid counting duplicate kv and generating lots of small hfiles in
> PutSortReducer
> -
>
> Key: HBASE-15171
> URL: https://issues.apache.org/jira/browse/HBASE-15171
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0, 1.1.2, 0.98.17
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-15171.addendum.patch, HBASE-15171.patch,
> HBASE-15171.patch, HBASE-15171.patch
>
>
> Once there was one of our online user writing huge number of duplicated kvs
> during bulkload, and we found it generated lots of small hfiles and slows
> down the whole process.
> After debugging, we found in PutSortReducer#reduce, although it already tried
> to handle the pathological case by setting a threshold for single-row size
> and having a TreeMap to avoid writing out duplicated kv, it forgot to exclude
> duplicated kv from the accumulated size. As shown in below code segment:
> {code}
> while (iter.hasNext() && curSize < threshold) {
> Put p = iter.next();
> for (List cells: p.getFamilyCellMap().values()) {
> for (Cell cell: cells) {
> KeyValue kv = KeyValueUtil.ensureKeyValue(cell);
> map.add(kv);
> curSize += kv.heapSize();
> }
> }
> }
> {code}
> We should move the {{curSize += kv.heapSize();}} line out of the outer for
> loop
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14877) maven archetype: client application
[ https://issues.apache.org/jira/browse/HBASE-14877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120738#comment-15120738 ] Daniel Vimont commented on HBASE-14877: --- I think I've got the README.md formatting looking the way I want it to now. I posted it into a test project over on github to view it via their rendering: https://github.com/dvimont/test_hbasearchetypes_readme > maven archetype: client application > --- > > Key: HBASE-14877 > URL: https://issues.apache.org/jira/browse/HBASE-14877 > Project: HBase > Issue Type: Sub-task > Components: build, Usability >Affects Versions: 2.0.0 >Reporter: Nick Dimiduk >Assignee: Daniel Vimont > Labels: archetype, beginner, maven > Attachments: HBASE-14877-v2.patch, HBASE-14877-v3.patch, > HBASE-14877-v4.patch, HBASE-14877.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14842) PrefixTree write path should work with BytebufferedCells (during compaction)
[ https://issues.apache.org/jira/browse/HBASE-14842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John resolved HBASE-14842. Resolution: Later Assignee: (was: ramkrishna.s.vasudevan) > PrefixTree write path should work with BytebufferedCells (during compaction) > > > Key: HBASE-14842 > URL: https://issues.apache.org/jira/browse/HBASE-14842 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: ramkrishna.s.vasudevan > > This is again related to HBASE-14832 where any Prefix Tree cell during > compaction should work with BBCells. For now changing the value/tag part > alone is enough since only the value/tag is going to be offheap and all > others are going to be onheap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-11425) Cell/DBB end-to-end on the read-path
[ https://issues.apache.org/jira/browse/HBASE-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan resolved HBASE-11425. Resolution: Fixed Hadoop Flags: Reviewed Moved out the Dictionary (HBASE-14841) and Prefix Tree (HBASE-14842) to work with ByteBuffer to HBASE-15179 as it is mostly concerned with writes. Hence resolving this parent JIRA as fixed. > Cell/DBB end-to-end on the read-path > > > Key: HBASE-11425 > URL: https://issues.apache.org/jira/browse/HBASE-11425 > Project: HBase > Issue Type: Umbrella > Components: regionserver, Scanners >Affects Versions: 0.99.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Attachments: BenchmarkTestCode.zip, Benchmarks_Tests.docx, GC pics > with evictions_4G heap.png, HBASE-11425-E2E-NotComplete.patch, > HBASE-11425.patch, Offheap reads in HBase using BBs_V2.pdf, Offheap reads in > HBase using BBs_final.pdf, Screen Shot 2015-10-16 at 5.13.22 PM.png, gc.png, > gets.png, heap.png, load.png, median.png, ram.log > > > Umbrella jira to make sure we can have blocks cached in offheap backed cache. > In the entire read path, we can refer to this offheap buffer and avoid onheap > copying. > The high level items I can identify as of now are > 1. Avoid the array() call on BB in read path.. (This is there in many > classes. We can handle class by class) > 2. Support Buffer based getter APIs in cell. In read path we will create a > new Cell with backed by BB. Will need in CellComparator, Filter (like SCVF), > CPs etc. > 3. Avoid KeyValue.ensureKeyValue() calls in read path - This make byte copy. > 4. Remove all CP hooks (which are already deprecated) which deal with KVs. > (In read path) > Will add subtasks under this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11425) Cell/DBB end-to-end on the read-path
[ https://issues.apache.org/jira/browse/HBASE-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-11425: --- Fix Version/s: 2.0.0 > Cell/DBB end-to-end on the read-path > > > Key: HBASE-11425 > URL: https://issues.apache.org/jira/browse/HBASE-11425 > Project: HBase > Issue Type: Umbrella > Components: regionserver, Scanners >Affects Versions: 0.99.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: BenchmarkTestCode.zip, Benchmarks_Tests.docx, GC pics > with evictions_4G heap.png, HBASE-11425-E2E-NotComplete.patch, > HBASE-11425.patch, Offheap reads in HBase using BBs_V2.pdf, Offheap reads in > HBase using BBs_final.pdf, Screen Shot 2015-10-16 at 5.13.22 PM.png, gc.png, > gets.png, heap.png, load.png, median.png, ram.log > > > Umbrella jira to make sure we can have blocks cached in offheap backed cache. > In the entire read path, we can refer to this offheap buffer and avoid onheap > copying. > The high level items I can identify as of now are > 1. Avoid the array() call on BB in read path.. (This is there in many > classes. We can handle class by class) > 2. Support Buffer based getter APIs in cell. In read path we will create a > new Cell with backed by BB. Will need in CellComparator, Filter (like SCVF), > CPs etc. > 3. Avoid KeyValue.ensureKeyValue() calls in read path - This make byte copy. > 4. Remove all CP hooks (which are already deprecated) which deal with KVs. > (In read path) > Will add subtasks under this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
[
https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120788#comment-15120788
]
Hadoop QA commented on HBASE-15180:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
42s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 50s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 6s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 16s
{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 29s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 26s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 26s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 29s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_91.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 29s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 11s
{color} | {color:red} Patch generated 2 new checkstyle issues in hbase-common
(total was 100, now 102). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 35s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 52s
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 43s
{color} | {color:green} hbase-common in the patch passed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 29s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 59s
{color} | {color:green} hbase-client in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} |
[jira] [Commented] (HBASE-15135) Add metrics for storefile age
[
https://issues.apache.org/jira/browse/HBASE-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120661#comment-15120661
]
Hadoop QA commented on HBASE-15135:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 1s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 29s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 50s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 29s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 16s
{color} | {color:green} hbase-hadoop-compat in the patch passed with JDK
v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 17s
{color} | {color:green} hbase-hadoop2-compat in the patch passed with JDK
v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 108m 32s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s
{color} | {color:green} hbase-hadoop-compat in the patch passed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 25s
{color} | {color:green} hbase-hadoop2-compat in the patch passed with JDK
[jira] [Commented] (HBASE-15128) Disable region splits and merges in HBCK
[ https://issues.apache.org/jira/browse/HBASE-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120683#comment-15120683 ] Enis Soztutar commented on HBASE-15128: --- bq. what are all these Tracker that we are adding? and why are we using zk to store persistent data when we said that zk should be used only for non transient states and the only exception is replication (and we are trying to get rid of that)? I think the patch follows the same model that balancer, catalog janitor and normalizer already use. That data is still at zookeeper. Agreed that this data should not be in zk, but it is a different issue. > Disable region splits and merges in HBCK > > > Key: HBASE-15128 > URL: https://issues.apache.org/jira/browse/HBASE-15128 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15128.patch, HBASE-15128_v1.patch, > HBASE-15128_v3.patch > > > In large clusters where region splits are frequent, and HBCK runs take > longer, the concurrent splits cause further problems in HBCK since HBCK > assumes a static state for the region partition map. We have just seen a case > where HBCK undo's a concurrently splitting region causing number of > inconsistencies to go up. > We can have a mode in master where splits and merges are disabled like the > balancer and catalog janitor switches. Master will reject the split requests > if regionservers decide to split. This switch can be turned on / off by the > admins and also automatically by HBCK while it is running (similar to > balancer switch being disabled by HBCK). > HBCK should also disable the Catalog Janitor just in case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14841) Allow Dictionary to work with BytebufferedCells
[ https://issues.apache.org/jira/browse/HBASE-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-14841: --- Status: Open (was: Patch Available) > Allow Dictionary to work with BytebufferedCells > --- > > Key: HBASE-14841 > URL: https://issues.apache.org/jira/browse/HBASE-14841 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: HBASE-14841.patch, HBASE-14841_1.patch, > HBASE-14841_2.patch, HBASE-14841_3.patch, HBASE-14841_4.patch, > HBASE-14841_5.patch > > > This is part of HBASE-14832 where we need to ensure that while BBCells are > getting compacted the TagCompression part should be working with BBCells. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15021) hadoopqa doing false positives
[
https://issues.apache.org/jira/browse/HBASE-15021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120819#comment-15120819
]
Nick Dimiduk commented on HBASE-15021:
--
Nope, no problem. Best to keep up everywhere we can to avoid backport issues
later. We're square :)
> hadoopqa doing false positives
> --
>
> Key: HBASE-15021
> URL: https://issues.apache.org/jira/browse/HBASE-15021
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: 15021.patch, 15021.thrownpe.patch, 15021.thrownpe.patch,
> 15021.thrownpe.patch, 15021.thrownpe.patch
>
>
> https://builds.apache.org/job/PreCommit-HBASE-Build/16930/consoleText says:
> {color:green}+1 core tests{color}. The patch passed unit tests in .
> ...but here is what happened:
> {code}
> ...
> Results :
> Tests in error:
> org.apache.hadoop.hbase.regionserver.TestRSStatusServlet.testBasic(org.apache.hadoop.hbase.regionserver.TestRSStatusServlet)
> Run 1: TestRSStatusServlet.testBasic:105 � NullPointer
> Run 2: TestRSStatusServlet.testBasic:105 � NullPointer
> Run 3: TestRSStatusServlet.testBasic:105 � NullPointer
> org.apache.hadoop.hbase.regionserver.TestRSStatusServlet.testWithRegions(org.apache.hadoop.hbase.regionserver.TestRSStatusServlet)
> Run 1: TestRSStatusServlet.testWithRegions:119 � NullPointer
> Run 2: TestRSStatusServlet.testWithRegions:119 � NullPointer
> Run 3: TestRSStatusServlet.testWithRegions:119 � NullPointer
> Tests run: 1033, Failures: 0, Errors: 2, Skipped: 21
> ...
> [INFO] Apache HBase - Server . FAILURE
> [17:54.559s]
> ...
> {code}
> Why we reporting pass when it failed?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13590) TestEnableTableHandler.testEnableTableWithNoRegionServers is flakey
[ https://issues.apache.org/jira/browse/HBASE-13590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-13590: - Attachment: HBASE-13590.branch-1.1.patch Reattaching patch for branch-1.1 QA Bot. > TestEnableTableHandler.testEnableTableWithNoRegionServers is flakey > --- > > Key: HBASE-13590 > URL: https://issues.apache.org/jira/browse/HBASE-13590 > Project: HBase > Issue Type: Test > Components: master >Reporter: Nick Dimiduk >Assignee: Yu Li > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4 > > Attachments: HBASE-13590.branch-1.1.patch, > HBASE-13590.branch-1.1.patch, HBASE-13590.branch-1.patch, > HBASE-13590.branch-1.v2.patch, testEnableTableHandler_branch-1.1.log.zip, > testEnableTableHandler_branch-1.log.zip > > > Looking at our [build > history|https://builds.apache.org/job/HBase-1.1/buildTimeTrend], it seems > this test is flakey. See builds 429, 431, 439. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15181) A simple implementation of date based tiered compaction
[ https://issues.apache.org/jira/browse/HBASE-15181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Clara Xiong updated HBASE-15181: Attachment: HBASE-15181-v1.patch > A simple implementation of date based tiered compaction > --- > > Key: HBASE-15181 > URL: https://issues.apache.org/jira/browse/HBASE-15181 > Project: HBase > Issue Type: New Feature > Components: Compaction >Reporter: Clara Xiong >Assignee: Clara Xiong > Fix For: 2.0.0 > > Attachments: HBASE-15181-v1.patch > > > This is a simple implementation of date-based tiered compaction similar to > Cassandra's for the following benefits: > 1. Improve date-range-based scan by structuring store files in date-based > tiered layout. > 2. Reduce compaction overhead. > 3. Improve TTL efficiency. > Perfect fit for the use cases that: > 1. has mostly date-based date write and scan and a focus on the most recent > data. > 2. never or rarely deletes data. > Out-of-order writes are handled gracefully so the data will still get to the > right store file for time-range-scan and re-compacton with existing store > file in the same time window is handled by ExploringCompactionPolicy. > Time range overlapping among store files is tolerated and the performance > impact is minimized. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13590) TestEnableTableHandler.testEnableTableWithNoRegionServers is flakey
[ https://issues.apache.org/jira/browse/HBASE-13590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120838#comment-15120838 ] Nick Dimiduk commented on HBASE-13590: -- Coming back to this one. [~carp84] You think the other failures you mention above are unrelated? I say we take it and let it stew. > TestEnableTableHandler.testEnableTableWithNoRegionServers is flakey > --- > > Key: HBASE-13590 > URL: https://issues.apache.org/jira/browse/HBASE-13590 > Project: HBase > Issue Type: Test > Components: master >Reporter: Nick Dimiduk >Assignee: Yu Li > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4 > > Attachments: HBASE-13590.branch-1.1.patch, > HBASE-13590.branch-1.patch, HBASE-13590.branch-1.v2.patch, > testEnableTableHandler_branch-1.1.log.zip, > testEnableTableHandler_branch-1.log.zip > > > Looking at our [build > history|https://builds.apache.org/job/HBase-1.1/buildTimeTrend], it seems > this test is flakey. See builds 429, 431, 439. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14841) Allow Dictionary to work with BytebufferedCells
[ https://issues.apache.org/jira/browse/HBASE-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-14841: --- Hadoop Flags: Reviewed Status: Patch Available (was: Open) > Allow Dictionary to work with BytebufferedCells > --- > > Key: HBASE-14841 > URL: https://issues.apache.org/jira/browse/HBASE-14841 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: HBASE-14841.patch, HBASE-14841_1.patch, > HBASE-14841_2.patch, HBASE-14841_3.patch, HBASE-14841_4.patch, > HBASE-14841_5.patch, HBASE-14841_6.patch, HBASE-14841_7.patch > > > This is part of HBASE-14832 where we need to ensure that while BBCells are > getting compacted the TagCompression part should be working with BBCells. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15173) Execute mergeRegions RPC call as the request user
[ https://issues.apache.org/jira/browse/HBASE-15173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120695#comment-15120695 ] Anoop Sam John commented on HBASE-15173: +1 > Execute mergeRegions RPC call as the request user > - > > Key: HBASE-15173 > URL: https://issues.apache.org/jira/browse/HBASE-15173 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15173.v1.patch, HBASE-15173.v2.patch, > HBASE-15173.v2.patch, HBASE-15173.v3.patch, HBASE-15173.v3.patch, > HBASE-15173.v3.patch, HBASE-15173.v3.patch > > > This is follow up to HBASE-15132 > Master currently sends mergeRegions RPC to region server under user 'hbase'. > This issue is to execute mergeRegions RPC call as the request user > See tail of HBASE-15132 for related discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
[
https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120736#comment-15120736
]
Ted Yu commented on HBASE-15180:
{code}
48public Cell readCell(boolean withTags) throws IOException {
49 if (intBuf == null) {
50// Lazy init. In real flow, we will use the readCell(int,
boolean) API only
51intBuf = new byte[Bytes.SIZEOF_INT];
{code}
There are two places where readCell() is implemented in the same manner. Can
code duplication be reduced ?
{code}
56protected static class CellReadablePBIS extends PBIS implements
CellReadable {
{code}
What does PB mean above ?
> Reduce garbage created while reading Cells from Codec Decoder
> -
>
> Key: HBASE-15180
> URL: https://issues.apache.org/jira/browse/HBASE-15180
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15180.patch
>
>
> In KeyValueDecoder#parseCell (Default Codec decoder) we use
> KeyValueUtil#iscreate to read cells from the InputStream. Here we 1st create
> a byte[] of length 4 and read the cell length and then an array of Cell's
> length and read in cell bytes into it and create a KV.
> Actually in server we read the reqs into a byte[] and CellScanner is created
> on top of a ByteArrayInputStream on top of this. By default in write path, we
> have MSLAB usage ON. So while adding Cells to memstore, we will copy the Cell
> bytes to MSLAB memory chunks (default 2 MB size) and recreate Cells over that
> bytes. So there is no issue if we create Cells over the RPC read byte[]
> directly here in Decoder. No need for 2 byte[] creation and copy for every
> Cell in request.
> My plan is to make a Cell aware ByteArrayInputStream which can read Cells
> directly from it.
> Same Codec path is used in client side also. There better we can avoid this
> direct Cell create and continue to do the copy to smaller byte[]s path. Plan
> to introduce some thing like a CodecContext associated with every Codec
> instance which can say the server/client context.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14842) PrefixTree write path should work with BytebufferedCells (during compaction)
[ https://issues.apache.org/jira/browse/HBASE-14842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-14842: --- Parent Issue: HBASE-15179 (was: HBASE-11425) > PrefixTree write path should work with BytebufferedCells (during compaction) > > > Key: HBASE-14842 > URL: https://issues.apache.org/jira/browse/HBASE-14842 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: ramkrishna.s.vasudevan > > This is again related to HBASE-14832 where any Prefix Tree cell during > compaction should work with BBCells. For now changing the value/tag part > alone is enough since only the value/tag is going to be offheap and all > others are going to be onheap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14841) Allow Dictionary to work with BytebufferedCells
[ https://issues.apache.org/jira/browse/HBASE-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-14841: --- Attachment: HBASE-14841_6.patch To avoid findbugs, as said in the previous comment created the container as of type Object and the BBNode.equals(), just checks for type Node and we do the proper type cast. > Allow Dictionary to work with BytebufferedCells > --- > > Key: HBASE-14841 > URL: https://issues.apache.org/jira/browse/HBASE-14841 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: HBASE-14841.patch, HBASE-14841_1.patch, > HBASE-14841_2.patch, HBASE-14841_3.patch, HBASE-14841_4.patch, > HBASE-14841_5.patch, HBASE-14841_6.patch > > > This is part of HBASE-14832 where we need to ensure that while BBCells are > getting compacted the TagCompression part should be working with BBCells. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15140) Fix ResultBoundedCompletionService deadlock
[
https://issues.apache.org/jira/browse/HBASE-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120811#comment-15120811
]
Nick Dimiduk commented on HBASE-15140:
--
Sorry, I missed this for 1.1.3.
[~eclark] can you think of any issues with backporting to 1.1? Cherry-picks
cleanly from branch-1.2.
> Fix ResultBoundedCompletionService deadlock
>
>
> Key: HBASE-15140
> URL: https://issues.apache.org/jira/browse/HBASE-15140
> Project: HBase
> Issue Type: Bug
> Components: Client, hbase
>Affects Versions: 1.1.2
>Reporter: Alok Singh
> Fix For: 1.1.4
>
>
> See: https://issues.apache.org/jira/browse/HBASE-14812
> We ran into this deadlock issue on the hbase 1.1.2 build
> {code}
> phoenix-1-thread-1340 id=3183 state=WAITING
> - waiting on <0x1ad981d3> (a
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> - locked <0x1ad981d3> (a
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
> at
> org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:67)
> at
> org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:88)
> at
> org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:79)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:105)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Locked synchronizers: count = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@55145434
> phoenix-1-thread-1341 id=3184 state=WAITING
> - waiting on <0x22e46b2c> (a
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> - locked <0x22e46b2c> (a
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
> at
> org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:67)
> at
> org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:88)
> at
> org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:79)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:105)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
>
[jira] [Updated] (HBASE-15140) Fix ResultBoundedCompletionService deadlock
[
https://issues.apache.org/jira/browse/HBASE-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-15140:
-
Attachment: 14812-cherrypick-branch-1.1.patch
> Fix ResultBoundedCompletionService deadlock
>
>
> Key: HBASE-15140
> URL: https://issues.apache.org/jira/browse/HBASE-15140
> Project: HBase
> Issue Type: Bug
> Components: Client, hbase
>Affects Versions: 1.1.2
>Reporter: Alok Singh
> Fix For: 1.1.4
>
> Attachments: 14812-cherrypick-branch-1.1.patch
>
>
> See: https://issues.apache.org/jira/browse/HBASE-14812
> We ran into this deadlock issue on the hbase 1.1.2 build
> {code}
> phoenix-1-thread-1340 id=3183 state=WAITING
> - waiting on <0x1ad981d3> (a
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> - locked <0x1ad981d3> (a
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
> at
> org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:67)
> at
> org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:88)
> at
> org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:79)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:105)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Locked synchronizers: count = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@55145434
> phoenix-1-thread-1341 id=3184 state=WAITING
> - waiting on <0x22e46b2c> (a
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> - locked <0x22e46b2c> (a
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
> at
> org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:67)
> at
> org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:88)
> at
> org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:79)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:105)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
> at
>
[jira] [Updated] (HBASE-14841) Allow Dictionary to work with BytebufferedCells
[ https://issues.apache.org/jira/browse/HBASE-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-14841: --- Attachment: HBASE-14841_7.patch Updated patch. Based on discussion setContents() will accept Object now and that will be type casted to byte[] and BB. Added assertion for that. > Allow Dictionary to work with BytebufferedCells > --- > > Key: HBASE-14841 > URL: https://issues.apache.org/jira/browse/HBASE-14841 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: HBASE-14841.patch, HBASE-14841_1.patch, > HBASE-14841_2.patch, HBASE-14841_3.patch, HBASE-14841_4.patch, > HBASE-14841_5.patch, HBASE-14841_6.patch, HBASE-14841_7.patch > > > This is part of HBASE-14832 where we need to ensure that while BBCells are > getting compacted the TagCompression part should be working with BBCells. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14841) Allow Dictionary to work with BytebufferedCells
[ https://issues.apache.org/jira/browse/HBASE-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-14841: --- Status: Open (was: Patch Available) > Allow Dictionary to work with BytebufferedCells > --- > > Key: HBASE-14841 > URL: https://issues.apache.org/jira/browse/HBASE-14841 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: HBASE-14841.patch, HBASE-14841_1.patch, > HBASE-14841_2.patch, HBASE-14841_3.patch, HBASE-14841_4.patch, > HBASE-14841_5.patch, HBASE-14841_6.patch > > > This is part of HBASE-14832 where we need to ensure that while BBCells are > getting compacted the TagCompression part should be working with BBCells. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120692#comment-15120692 ] Enis Soztutar commented on HBASE-6721: -- bq. The exact same was requested for memcached block cache, and it was reasonable then. I'm simply asking for this feature to get the same treatment that optional off by default removable features should get Yep, everybody agrees that it should be completely optional and non-core for this. What I am saying is that there is no need to fork out a different module for this. I just checked, nobody asked the memcache thing to be a different module above. bq. This feature should never be used by anyone other than yahoo and we have a duty to our users to make sure that they understand that. That is up for the users to decide. Sophisticated users can make their own decisions. bq. rsgroup as used in the table name is better. Agreed. I've brought this up in the review already. I thought we have addressed the cases, but if we haven't, we should stick with rsgroups rather than groups. Having consistent naming is something we are lacking in hbase (alter table in shell vs modify table in java, etc). > RegionServer Group based Assignment > --- > > Key: HBASE-6721 > URL: https://issues.apache.org/jira/browse/HBASE-6721 > Project: HBase > Issue Type: New Feature >Reporter: Francis Liu >Assignee: Francis Liu > Labels: hbase-6721 > Attachments: 6721-master-webUI.patch, HBASE-6721 > GroupBasedLoadBalancer Sequence Diagram.xml, HBASE-6721-DesigDoc.pdf, > HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, > HBASE-6721_0.98_2.patch, HBASE-6721_10.patch, HBASE-6721_11.patch, > HBASE-6721_12.patch, HBASE-6721_13.patch, HBASE-6721_14.patch, > HBASE-6721_15.patch, HBASE-6721_8.patch, HBASE-6721_9.patch, > HBASE-6721_9.patch, HBASE-6721_94.patch, HBASE-6721_94.patch, > HBASE-6721_94_2.patch, HBASE-6721_94_3.patch, HBASE-6721_94_3.patch, > HBASE-6721_94_4.patch, HBASE-6721_94_5.patch, HBASE-6721_94_6.patch, > HBASE-6721_94_7.patch, HBASE-6721_98_1.patch, HBASE-6721_98_2.patch, > HBASE-6721_hbase-6721_addendum.patch, HBASE-6721_trunk.patch, > HBASE-6721_trunk.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk1.patch, > HBASE-6721_trunk2.patch, balanceCluster Sequence Diagram.svg, > hbase-6721-v15-branch-1.1.patch, hbase-6721-v16.patch, hbase-6721-v17.patch, > hbase-6721-v18.patch, hbase-6721-v19.patch, hbase-6721-v20.patch, > hbase-6721-v21.patch, hbase-6721-v22.patch, hbase-6721-v23.patch, > hbase-6721-v25.patch, immediateAssignments Sequence Diagram.svg, > randomAssignment Sequence Diagram.svg, retainAssignment Sequence Diagram.svg, > roundRobinAssignment Sequence Diagram.svg > > > In multi-tenant deployments of HBase, it is likely that a RegionServer will > be serving out regions from a number of different tables owned by various > client applications. Being able to group a subset of running RegionServers > and assign specific tables to it, provides a client application a level of > isolation and resource allocation. > The proposal essentially is to have an AssignmentManager which is aware of > RegionServer groups and assigns tables to region servers based on groupings. > Load balancing will occur on a per group basis as well. > This is essentially a simplification of the approach taken in HBASE-4120. See > attached document. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14841) Allow Dictionary to work with BytebufferedCells
[ https://issues.apache.org/jira/browse/HBASE-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-14841: --- Parent Issue: HBASE-15179 (was: HBASE-11425) > Allow Dictionary to work with BytebufferedCells > --- > > Key: HBASE-14841 > URL: https://issues.apache.org/jira/browse/HBASE-14841 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: HBASE-14841.patch, HBASE-14841_1.patch, > HBASE-14841_2.patch, HBASE-14841_3.patch, HBASE-14841_4.patch, > HBASE-14841_5.patch, HBASE-14841_6.patch > > > This is part of HBASE-14832 where we need to ensure that while BBCells are > getting compacted the TagCompression part should be working with BBCells. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14841) Allow Dictionary to work with BytebufferedCells
[ https://issues.apache.org/jira/browse/HBASE-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-14841: --- Status: Patch Available (was: Open) > Allow Dictionary to work with BytebufferedCells > --- > > Key: HBASE-14841 > URL: https://issues.apache.org/jira/browse/HBASE-14841 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: HBASE-14841.patch, HBASE-14841_1.patch, > HBASE-14841_2.patch, HBASE-14841_3.patch, HBASE-14841_4.patch, > HBASE-14841_5.patch, HBASE-14841_6.patch > > > This is part of HBASE-14832 where we need to ensure that while BBCells are > getting compacted the TagCompression part should be working with BBCells. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
[ https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120791#comment-15120791 ] Anoop Sam John commented on HBASE-15180: bq.What does PB mean above ? PushbackInputStream. We have PBIS already here. Let me see how can we avoid the duplication > Reduce garbage created while reading Cells from Codec Decoder > - > > Key: HBASE-15180 > URL: https://issues.apache.org/jira/browse/HBASE-15180 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15180.patch > > > In KeyValueDecoder#parseCell (Default Codec decoder) we use > KeyValueUtil#iscreate to read cells from the InputStream. Here we 1st create > a byte[] of length 4 and read the cell length and then an array of Cell's > length and read in cell bytes into it and create a KV. > Actually in server we read the reqs into a byte[] and CellScanner is created > on top of a ByteArrayInputStream on top of this. By default in write path, we > have MSLAB usage ON. So while adding Cells to memstore, we will copy the Cell > bytes to MSLAB memory chunks (default 2 MB size) and recreate Cells over that > bytes. So there is no issue if we create Cells over the RPC read byte[] > directly here in Decoder. No need for 2 byte[] creation and copy for every > Cell in request. > My plan is to make a Cell aware ByteArrayInputStream which can read Cells > directly from it. > Same Codec path is used in client side also. There better we can avoid this > direct Cell create and continue to do the copy to smaller byte[]s path. Plan > to introduce some thing like a CodecContext associated with every Codec > instance which can say the server/client context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15173) Execute mergeRegions RPC call as the request user
[
https://issues.apache.org/jira/browse/HBASE-15173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120835#comment-15120835
]
Hadoop QA commented on HBASE-15173:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 38s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 22s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 8m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 57s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 38s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 14s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 17s
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 131m 44s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 21s
{color} | {color:green} hbase-client in the patch passed with JDK v1.7.0_91.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 123m 41s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 330m 36s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Timed out junit tests |
org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster |
| |
[jira] [Commented] (HBASE-15140) Fix ResultBoundedCompletionService deadlock
[
https://issues.apache.org/jira/browse/HBASE-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120865#comment-15120865
]
Hadoop QA commented on HBASE-15140:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
39s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s
{color} | {color:green} branch-1.1 passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} branch-1.1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 58s
{color} | {color:red} hbase-client in branch-1.1 has 15 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 17s
{color} | {color:red} hbase-client in branch-1.1 failed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 4m
0s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 16s
{color} | {color:red} hbase-client in the patch failed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 30s
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 39s
{color} | {color:green} hbase-client in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 55s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-01-28 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12784833/14812-cherrypick-branch-1.1.patch
|
| JIRA Issue | HBASE-15140 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux
[jira] [Commented] (HBASE-15135) Add metrics for storefile age
[
https://issues.apache.org/jira/browse/HBASE-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120675#comment-15120675
]
Hadoop QA commented on HBASE-15135:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
50s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 50s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 10s
{color} | {color:red} hbase-hadoop2-compat in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 4s
{color} | {color:red} Patch generated 1 new checkstyle issues in hbase-server
(total was 145, now 145). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
23m 6s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 18s
{color} | {color:green} hbase-hadoop2-compat in the patch passed with JDK
v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 15s
{color} | {color:green} hbase-hadoop-compat in the patch passed with JDK
v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 9s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s
{color} | {color:green} hbase-hadoop2-compat in the patch passed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 18s
{color} | {color:green} hbase-hadoop-compat in the patch passed with JDK
v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 12s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense
[jira] [Commented] (HBASE-10877) HBase non-retriable exception list should be expanded
[
https://issues.apache.org/jira/browse/HBASE-10877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120781#comment-15120781
]
Nick Dimiduk commented on HBASE-10877:
--
[~ajinkyakale] An elaboration of this issue, along with a work-around, are
documented on our book. See the end of the mapreduce section:
http://hbase.apache.org/book.html#hbase.mapreduce.classpath
> HBase non-retriable exception list should be expanded
> -
>
> Key: HBASE-10877
> URL: https://issues.apache.org/jira/browse/HBASE-10877
> Project: HBase
> Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Priority: Minor
>
> Example where retries do not make sense:
> {noformat}
> 2014-03-31 20:54:27,765 WARN [InputInitializer [Map 1] #0]
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation:
> Encountered problems when prefetch hbase:meta table:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=35, exceptions:
> Mon Mar 31 20:45:17 UTC 2014,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
> java.lang.IllegalAccessError: class
> com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass
> com.google.protobuf.LiteralByteString
> Mon Mar 31 20:45:17 UTC 2014,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
> java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
> Mon Mar 31 20:45:17 UTC 2014,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
> java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
> Mon Mar 31 20:45:18 UTC 2014,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
> java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
> Mon Mar 31 20:45:20 UTC 2014,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
> java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
> Mon Mar 31 20:45:24 UTC 2014,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
> java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
> Mon Mar 31 20:45:34 UTC 2014,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
> java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
> Mon Mar 31 20:45:45 UTC 2014,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
> java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
> Mon Mar 31 20:45:55 UTC 2014,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
> java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
> Mon Mar 31 20:46:05 UTC 2014,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
> java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
> Mon Mar 31 20:46:25 UTC 2014,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
> java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
> Mon Mar 31 20:46:45 UTC 2014,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
> java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
> Mon Mar 31 20:47:05 UTC 2014,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
> java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
> Mon Mar 31 20:47:25 UTC 2014,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
> java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
> Mon Mar 31 20:47:45 UTC 2014,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
> java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
> Mon Mar 31 20:48:05 UTC 2014,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
> java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
> Mon Mar 31 20:48:25 UTC 2014,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
> java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
> Mon Mar 31 20:48:46 UTC 2014,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
> java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
> Mon Mar 31 20:49:06 UTC 2014,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
> java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
> Mon Mar 31 20:49:26 UTC 2014,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
> java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
> Mon Mar 31 20:49:46 UTC 2014,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
> java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
> Mon Mar 31 20:50:06 UTC 2014,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
> java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
> Mon
[jira] [Updated] (HBASE-15167) Deadlock in TestNamespaceAuditor.testRegionOperations on 1.1
[
https://issues.apache.org/jira/browse/HBASE-15167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-15167:
-
Priority: Critical (was: Major)
> Deadlock in TestNamespaceAuditor.testRegionOperations on 1.1
>
>
> Key: HBASE-15167
> URL: https://issues.apache.org/jira/browse/HBASE-15167
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 1.1.3
>Reporter: Nick Dimiduk
>Priority: Critical
> Fix For: 1.1.4
>
> Attachments: blocked.log
>
>
> This was left as a zombie after one of my test runs this weekend.
> {noformat}
> "WALProcedureStoreSyncThread" daemon prio=10 tid=0x7f3ccc209000
> nid=0x3960 in Object.wait() [0x7f3c6b6b5000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:503)
> at org.apache.hadoop.ipc.Client.call(Client.java:1397)
> - locked <0x0007f2813390> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy23.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy23.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:264)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy24.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy24.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1612)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1488)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1413)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:387)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:383)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:383)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:327)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:766)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:733)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.tryRollWriter(WALProcedureStore.java:668)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.periodicRoll(WALProcedureStore.java:711)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.syncLoop(WALProcedureStore.java:531)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.access$000(WALProcedureStore.java:66)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore$1.run(WALProcedureStore.java:180)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15167) Deadlock in TestNamespaceAuditor.testRegionOperations on 1.1
[
https://issues.apache.org/jira/browse/HBASE-15167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120807#comment-15120807
]
Nick Dimiduk commented on HBASE-15167:
--
Could well be. Commit doesn't cherry-pick cleanly from master. Could I bother
you to try a backport [~chenheng]? Much obliged!
> Deadlock in TestNamespaceAuditor.testRegionOperations on 1.1
>
>
> Key: HBASE-15167
> URL: https://issues.apache.org/jira/browse/HBASE-15167
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 1.1.3
>Reporter: Nick Dimiduk
> Fix For: 1.1.4
>
> Attachments: blocked.log
>
>
> This was left as a zombie after one of my test runs this weekend.
> {noformat}
> "WALProcedureStoreSyncThread" daemon prio=10 tid=0x7f3ccc209000
> nid=0x3960 in Object.wait() [0x7f3c6b6b5000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:503)
> at org.apache.hadoop.ipc.Client.call(Client.java:1397)
> - locked <0x0007f2813390> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy23.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy23.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:264)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy24.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy24.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1612)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1488)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1413)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:387)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:383)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:383)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:327)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:766)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:733)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.tryRollWriter(WALProcedureStore.java:668)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.periodicRoll(WALProcedureStore.java:711)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.syncLoop(WALProcedureStore.java:531)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.access$000(WALProcedureStore.java:66)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore$1.run(WALProcedureStore.java:180)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15167) Deadlock in TestNamespaceAuditor.testRegionOperations on 1.1
[
https://issues.apache.org/jira/browse/HBASE-15167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-15167:
-
Fix Version/s: 1.1.4
> Deadlock in TestNamespaceAuditor.testRegionOperations on 1.1
>
>
> Key: HBASE-15167
> URL: https://issues.apache.org/jira/browse/HBASE-15167
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 1.1.3
>Reporter: Nick Dimiduk
> Fix For: 1.1.4
>
> Attachments: blocked.log
>
>
> This was left as a zombie after one of my test runs this weekend.
> {noformat}
> "WALProcedureStoreSyncThread" daemon prio=10 tid=0x7f3ccc209000
> nid=0x3960 in Object.wait() [0x7f3c6b6b5000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:503)
> at org.apache.hadoop.ipc.Client.call(Client.java:1397)
> - locked <0x0007f2813390> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy23.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy23.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:264)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy24.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy24.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1612)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1488)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1413)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:387)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:383)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:383)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:327)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:766)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:733)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.tryRollWriter(WALProcedureStore.java:668)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.periodicRoll(WALProcedureStore.java:711)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.syncLoop(WALProcedureStore.java:531)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.access$000(WALProcedureStore.java:66)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore$1.run(WALProcedureStore.java:180)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15093) Replication can report incorrect size of log queue for the global source when multiwal is enabled
[ https://issues.apache.org/jira/browse/HBASE-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashu Pachauri updated HBASE-15093: -- Attachment: HBASE-15093-V1.patch Fixed reported checkstyle issues. > Replication can report incorrect size of log queue for the global source when > multiwal is enabled > - > > Key: HBASE-15093 > URL: https://issues.apache.org/jira/browse/HBASE-15093 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.2.0, 1.2.1 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri >Priority: Minor > Attachments: HBASE-15093-V0.patch, HBASE-15093-V1.patch > > > Replication can report incorrect size for the size of log queue for the > global source when multiwal is enabled. This happens because the method > MetricsSource#setSizeofLogQueue performs non-trivial operations in a > multithreaded world, even though it is not synchronized. > We can simply divide MetricsSource#setSizeofLogQueue into > MetricsSource#incrSizeofLogQueue and MetricsSource#decrSizeofLogQueue. Not > sure why we are currently directly setting the size instead of > incrementing/decrementing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15181) A simple implementation of date based tiered compaction
[ https://issues.apache.org/jira/browse/HBASE-15181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Clara Xiong updated HBASE-15181: Status: Patch Available (was: Open) HBASE-15181 > A simple implementation of date based tiered compaction > --- > > Key: HBASE-15181 > URL: https://issues.apache.org/jira/browse/HBASE-15181 > Project: HBase > Issue Type: New Feature > Components: Compaction >Reporter: Clara Xiong >Assignee: Clara Xiong > Fix For: 2.0.0 > > Attachments: HBASE-15181-v1.patch > > > This is a simple implementation of date-based tiered compaction similar to > Cassandra's for the following benefits: > 1. Improve date-range-based scan by structuring store files in date-based > tiered layout. > 2. Reduce compaction overhead. > 3. Improve TTL efficiency. > Perfect fit for the use cases that: > 1. has mostly date-based date write and scan and a focus on the most recent > data. > 2. never or rarely deletes data. > Out-of-order writes are handled gracefully so the data will still get to the > right store file for time-range-scan and re-compacton with existing store > file in the same time window is handled by ExploringCompactionPolicy. > Time range overlapping among store files is tolerated and the performance > impact is minimized. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11368) Multi-column family BulkLoad fails if compactions go on too long
[
https://issues.apache.org/jira/browse/HBASE-11368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120874#comment-15120874
]
Hadoop QA commented on HBASE-11368:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} HBASE-11368 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/latest/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12677543/hbase11368-master.patch
|
| JIRA Issue | HBASE-11368 |
| Powered by | Apache Yetus 0.1.0 http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/331/console |
This message was automatically generated.
> Multi-column family BulkLoad fails if compactions go on too long
>
>
> Key: HBASE-11368
> URL: https://issues.apache.org/jira/browse/HBASE-11368
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: Qiang Tian
> Attachments: hbase-11368-0.98.5.patch, hbase11368-master.patch,
> key_stacktrace_hbase10882.TXT, performance_improvement_verification_98.5.patch
>
>
> Compactions take a read lock. If a multi-column family region, before bulk
> loading, we want to take a write lock on the region. If the compaction takes
> too long, the bulk load fails.
> Various recipes include:
> + Making smaller regions (lame)
> + [~victorunique] suggests major compacting just before bulk loading over in
> HBASE-10882 as a work around.
> Does the compaction need a read lock for that long? Does the bulk load need
> a full write lock when multiple column families? Can we fail more gracefully
> at least?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
[ https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15180: --- Attachment: HBASE-15180.patch > Reduce garbage created while reading Cells from Codec Decoder > - > > Key: HBASE-15180 > URL: https://issues.apache.org/jira/browse/HBASE-15180 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15180.patch > > > In KeyValueDecoder#parseCell (Default Codec decoder) we use > KeyValueUtil#iscreate to read cells from the InputStream. Here we 1st create > a byte[] of length 4 and read the cell length and then an array of Cell's > length and read in cell bytes into it and create a KV. > Actually in server we read the reqs into a byte[] and CellScanner is created > on top of a ByteArrayInputStream on top of this. By default in write path, we > have MSLAB usage ON. So while adding Cells to memstore, we will copy the Cell > bytes to MSLAB memory chunks (default 2 MB size) and recreate Cells over that > bytes. So there is no issue if we create Cells over the RPC read byte[] > directly here in Decoder. No need for 2 byte[] creation and copy for every > Cell in request. > My plan is to make a Cell aware ByteArrayInputStream which can read Cells > directly from it. > Same Codec path is used in client side also. There better we can avoid this > direct Cell create and continue to do the copy to smaller byte[]s path. Plan > to introduce some thing like a CodecContext associated with every Codec > instance which can say the server/client context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15140) Fix ResultBoundedCompletionService deadlock
[
https://issues.apache.org/jira/browse/HBASE-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-15140:
-
Priority: Critical (was: Major)
> Fix ResultBoundedCompletionService deadlock
>
>
> Key: HBASE-15140
> URL: https://issues.apache.org/jira/browse/HBASE-15140
> Project: HBase
> Issue Type: Bug
> Components: Client, hbase
>Affects Versions: 1.1.2
>Reporter: Alok Singh
>Priority: Critical
> Fix For: 1.1.4
>
> Attachments: 14812-cherrypick-branch-1.1.patch
>
>
> See: https://issues.apache.org/jira/browse/HBASE-14812
> We ran into this deadlock issue on the hbase 1.1.2 build
> {code}
> phoenix-1-thread-1340 id=3183 state=WAITING
> - waiting on <0x1ad981d3> (a
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> - locked <0x1ad981d3> (a
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
> at
> org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:67)
> at
> org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:88)
> at
> org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:79)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:105)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Locked synchronizers: count = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@55145434
> phoenix-1-thread-1341 id=3184 state=WAITING
> - waiting on <0x22e46b2c> (a
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> - locked <0x22e46b2c> (a
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
> at
> org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:67)
> at
> org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:88)
> at
> org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:79)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:105)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
> at
>
[jira] [Updated] (HBASE-15140) Fix ResultBoundedCompletionService deadlock
[
https://issues.apache.org/jira/browse/HBASE-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-15140:
-
Status: Patch Available (was: Open)
> Fix ResultBoundedCompletionService deadlock
>
>
> Key: HBASE-15140
> URL: https://issues.apache.org/jira/browse/HBASE-15140
> Project: HBase
> Issue Type: Bug
> Components: Client, hbase
>Affects Versions: 1.1.2
>Reporter: Alok Singh
> Fix For: 1.1.4
>
> Attachments: 14812-cherrypick-branch-1.1.patch
>
>
> See: https://issues.apache.org/jira/browse/HBASE-14812
> We ran into this deadlock issue on the hbase 1.1.2 build
> {code}
> phoenix-1-thread-1340 id=3183 state=WAITING
> - waiting on <0x1ad981d3> (a
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> - locked <0x1ad981d3> (a
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
> at
> org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:67)
> at
> org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:88)
> at
> org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:79)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:105)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Locked synchronizers: count = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@55145434
> phoenix-1-thread-1341 id=3184 state=WAITING
> - waiting on <0x22e46b2c> (a
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> - locked <0x22e46b2c> (a
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
> at
> org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:67)
> at
> org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:88)
> at
> org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:79)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:105)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
> at
>
[jira] [Commented] (HBASE-15171) Avoid counting duplicate kv and generating lots of small hfiles in PutSortReducer
[
https://issues.apache.org/jira/browse/HBASE-15171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120442#comment-15120442
]
Hudson commented on HBASE-15171:
FAILURE: Integrated in HBase-Trunk_matrix #663 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/663/])
HBASE-15171 Avoid counting duplicate kv and generating lots of small (tedyu:
rev 47c41479401ea0aadfa3c3776fe2930bb8e9710d)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/PutSortReducer.java
> Avoid counting duplicate kv and generating lots of small hfiles in
> PutSortReducer
> -
>
> Key: HBASE-15171
> URL: https://issues.apache.org/jira/browse/HBASE-15171
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0, 1.1.2, 0.98.17
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-15171.patch, HBASE-15171.patch, HBASE-15171.patch
>
>
> Once there was one of our online user writing huge number of duplicated kvs
> during bulkload, and we found it generated lots of small hfiles and slows
> down the whole process.
> After debugging, we found in PutSortReducer#reduce, although it already tried
> to handle the pathological case by setting a threshold for single-row size
> and having a TreeMap to avoid writing out duplicated kv, it forgot to exclude
> duplicated kv from the accumulated size. As shown in below code segment:
> {code}
> while (iter.hasNext() && curSize < threshold) {
> Put p = iter.next();
> for (List cells: p.getFamilyCellMap().values()) {
> for (Cell cell: cells) {
> KeyValue kv = KeyValueUtil.ensureKeyValue(cell);
> map.add(kv);
> curSize += kv.heapSize();
> }
> }
> }
> {code}
> We should move the {{curSize += kv.heapSize();}} line out of the outer for
> loop
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14916) Add checkstyle_report.py to other branches
[ https://issues.apache.org/jira/browse/HBASE-14916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-14916: - Resolution: Invalid Assignee: (was: Appy) Status: Resolved (was: Patch Available) Obsolete after yetus integration. > Add checkstyle_report.py to other branches > -- > > Key: HBASE-14916 > URL: https://issues.apache.org/jira/browse/HBASE-14916 > Project: HBase > Issue Type: Bug >Reporter: Appy > Attachments: HBASE-14916-branch-1-v2.patch, > HBASE-14916-branch-1-v3.patch, HBASE-14916-branch-1.patch > > > Given test-patch.sh is always run from master, and that it now uses > checkstyle_report.py, we should pull back the script to other branches too. > Otherwise we see error like: > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/jenkins.build/dev-support/test-patch.sh: > line 662: > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/dev-support/checkstyle_report.py: > No such file or directory > [reference|https://builds.apache.org/job/PreCommit-HBASE-Build/16734//consoleFull] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15181) A simple implementation of date based tiered compaction
[ https://issues.apache.org/jira/browse/HBASE-15181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120571#comment-15120571 ] Devaraj Das commented on HBASE-15181: - dup of https://issues.apache.org/jira/browse/HBASE-14477? > A simple implementation of date based tiered compaction > --- > > Key: HBASE-15181 > URL: https://issues.apache.org/jira/browse/HBASE-15181 > Project: HBase > Issue Type: New Feature > Components: Compaction >Reporter: Clara Xiong >Assignee: Clara Xiong > Fix For: 2.0.0 > > > This is a simple implementation of date-based tiered compaction similar to > Cassandra's for the following benefits: > 1. Improve date-range-based scan by structuring store files in date-based > tiered layout. > 2. Reduce compaction overhead. > 3. Improve TTL efficiency. > Perfect fit for the use cases that: > 1. has mostly date-based date write and scan and a focus on the most recent > data. > 2. never or rarely deletes data. > Out-of-order writes are handled gracefully so the data will still get to the > right store file for time-range-scan and re-compacton with existing store > file in the same time window is handled by ExploringCompactionPolicy. > Time range overlapping among store files is tolerated and the performance > impact is minimized. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15179) Cell/DBB end-to-end on the write-path
Anoop Sam John created HBASE-15179: -- Summary: Cell/DBB end-to-end on the write-path Key: HBASE-15179 URL: https://issues.apache.org/jira/browse/HBASE-15179 Project: HBase Issue Type: Umbrella Components: regionserver Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0 Umbrella jira to make the HBase write path off heap E2E. We have to make sure we have Cells flowing in entire write path. Starting from request received in RPC layer, till the Cells get flushed out as HFiles, we have to keep the Cell data off heap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15177) Reduce garbage created under high load
[ https://issues.apache.org/jira/browse/HBASE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118850#comment-15118850 ] Anoop Sam John commented on HBASE-15177: In the patch there is some changes around setting priority as well. Intended? I did not see the patch in detail. > Reduce garbage created under high load > -- > > Key: HBASE-15177 > URL: https://issues.apache.org/jira/browse/HBASE-15177 > Project: HBase > Issue Type: Improvement >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0 > > Attachments: Screen Shot 2016-01-26 at 10.03.48 PM.png, Screen Shot > 2016-01-26 at 10.03.56 PM.png, Screen Shot 2016-01-26 at 10.06.16 PM.png, > Screen Shot 2016-01-26 at 10.15.15 PM.png, hbase-15177_v0.patch > > > I have been doing some profiling of the garbage being created. The idea was > to follow up on HBASE-14490 and experiment with offheap IPC byte buffers and > byte buffer re-use. However, without changing the IPC byte buffers for now, > there are a couple of (easy) improvements that I've identified from > profiling: > 1. RPCServer.Connection.processRequest() should work with ByteBuffer instead > of byte[] and not-recreate CodedInputStream a few times. > 2. RSRpcServices.getRegion() allocates two byte arrays for region, while only > 1 is needed. > 3. AnnotationReadingPriorityFunction is very expensive in allocations. Mainly > it allocates the regionName byte[] to get the table name. We already set the > priority for most of the operations (multi, get, increment, etc) but we are > only reading the priority in case of multi. We should use the priority from > the client side. > Lets do the simple improvements in this patch, we can get to IPC buffer > re-use in HBASE-14490. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14259) Backport Namespace quota support to 98 branch
[
https://issues.apache.org/jira/browse/HBASE-14259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118848#comment-15118848
]
Jianwei Cui commented on HBASE-14259:
-
Thanks for the patch, [~avandana]. Seems one typo in patch v3?
{code}
+ public void updateQuotaForRegionMerge(HRegionInfo hri) throws IOException {
+if (isInitialized()) { // ===> if (!isInitialized()) {
+ throw new IOException(
+ "Merge operation is being performed even before namespace auditor is
initialized.");
+}
{code}
In TestZKLessNamespaceAuditor, setBoolean("hbase.assignment.usezk", false) will
be applied to TestZKLessNamespaceAuditor#UTIL, not TestNamespaceAuditor#UTIL,
making TestZKLessNamespaceAuditor will also use zookeeper to assign region.
Need to make TestNamespaceAuditor#UTIL protected?
{code}
+@Category(MediumTests.class)
+public class TestZKLessNamespaceAuditor extends TestNamespaceAuditor {
+ private static final HBaseTestingUtility UTIL = new HBaseTestingUtility();
+
+ @BeforeClass
+ public static void before() throws Exception {
+UTIL.getConfiguration().setBoolean("hbase.assignment.usezk", false);
+setupOnce();
+ }
{code}
In AssignmentManager:
{code}
case MERGED:
case READY_TO_MERGE:
case MERGE_PONR:
case MERGED:
+ try {
+regionStateListener.onRegionMerged(hri);
+ } catch (IOException exp) {
+errorMsg = StringUtils.stringifyException(exp);
+ }
{code}
should invoke regionStateListener.onRegionMerged only in MERGED case?
{code}
case MERGE_PONR:
case MERGED:
+ if (code == TransitionCode.MERGED) {
+ try {
+regionStateListener.onRegionMerged(hri);
+ } catch (IOException exp) {
+errorMsg = StringUtils.stringifyException(exp);
+ }
+ }
{code}
> Backport Namespace quota support to 98 branch
> --
>
> Key: HBASE-14259
> URL: https://issues.apache.org/jira/browse/HBASE-14259
> Project: HBase
> Issue Type: Task
>Reporter: Vandana Ayyalasomayajula
>Assignee: Andrew Purtell
> Fix For: 0.98.18
>
> Attachments: HBASE-14259_v1_0.98.patch, HBASE-14259_v2_0.98.patch,
> HBASE-14259_v3_0.98.patch
>
>
> Namespace quota support (HBASE-8410) has been backported to branch-1
> (HBASE-13438). This jira would backport the same to 98 branch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15177) Reduce garbage created under high load
[ https://issues.apache.org/jira/browse/HBASE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118846#comment-15118846 ] Anoop Sam John commented on HBASE-15177: bq. experiment with offheap IPC byte buffers and byte buffer re-use We are doing some PoC in this area. Could not test it for perf and that is why did not raise any Jira. The idea is extension what HBASE-14490. says. We read reqs into off heap ByteBuffers that come from pool. Also there is another improvement am trying to reduce the garbage created by the Codec decoding the cells (from CellScanner) All these are under our project of off heaping the write path (After Read path off heaping which is complete now HBASE-11425) Let me raise some jiras for this as well > Reduce garbage created under high load > -- > > Key: HBASE-15177 > URL: https://issues.apache.org/jira/browse/HBASE-15177 > Project: HBase > Issue Type: Improvement >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0 > > Attachments: Screen Shot 2016-01-26 at 10.03.48 PM.png, Screen Shot > 2016-01-26 at 10.03.56 PM.png, Screen Shot 2016-01-26 at 10.06.16 PM.png, > Screen Shot 2016-01-26 at 10.15.15 PM.png, hbase-15177_v0.patch > > > I have been doing some profiling of the garbage being created. The idea was > to follow up on HBASE-14490 and experiment with offheap IPC byte buffers and > byte buffer re-use. However, without changing the IPC byte buffers for now, > there are a couple of (easy) improvements that I've identified from > profiling: > 1. RPCServer.Connection.processRequest() should work with ByteBuffer instead > of byte[] and not-recreate CodedInputStream a few times. > 2. RSRpcServices.getRegion() allocates two byte arrays for region, while only > 1 is needed. > 3. AnnotationReadingPriorityFunction is very expensive in allocations. Mainly > it allocates the regionName byte[] to get the table name. We already set the > priority for most of the operations (multi, get, increment, etc) but we are > only reading the priority in case of multi. We should use the priority from > the client side. > Lets do the simple improvements in this patch, we can get to IPC buffer > re-use in HBASE-14490. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15145) HBCK and Replication should authenticate to zookepeer using server principal
[ https://issues.apache.org/jira/browse/HBASE-15145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118836#comment-15118836 ] Hudson commented on HBASE-15145: FAILURE: Integrated in HBase-1.1-JDK7 #1647 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1647/]) HBASE-15145 HBCK and Replication should authenticate to zookepeer using (enis: rev aa5dfae303260d3d1f7c39233f67a6038b9fc6d2) * bin/hbase * bin/hbase-config.sh > HBCK and Replication should authenticate to zookepeer using server principal > > > Key: HBASE-15145 > URL: https://issues.apache.org/jira/browse/HBASE-15145 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4 > > Attachments: hbase-15145_v1.patch, hbase-15145_v2.patch > > > In secure clusters, we protect znodes with the server principal in zk. > However, if a user wants to add a replication peer or run HBCK, then she will > get Auth exception. This was not a problem due to an earlier bug. > For replication, the long term fix is HBASE-11392. However, we should still > have a way to launch zkcli with the server principals for manual inspection / > manipulation. > HBCK should always assume the server principals. > Thanks [~Koelli] for reporting this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15163) Add sampling code and metrics for get/scan/multi/mutate count separately
[ https://issues.apache.org/jira/browse/HBASE-15163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119769#comment-15119769 ] Sean Busbey commented on HBASE-15163: - [~stack] is starting on another round of cleaning up flakey tests. please coordinate with him before dismissing test results. > Add sampling code and metrics for get/scan/multi/mutate count separately > > > Key: HBASE-15163 > URL: https://issues.apache.org/jira/browse/HBASE-15163 > Project: HBase > Issue Type: Sub-task >Reporter: Yu Li >Assignee: Yu Li > Attachments: DifferentRequestQPS.png, HBASE-15163.patch, > HBASE-15163_v2.patch, HBASE-15163_v3.patch, HBASE-15163_v3.patch > > > This way we could see QPS of different kinds of requests, to better analyze > what's causing hot spot in system -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15135) Add metrics for storefile age
[
https://issues.apache.org/jira/browse/HBASE-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119647#comment-15119647
]
Mikhail Antonov commented on HBASE-15135:
-
And, sure enough, it fails then
{{hbase-hadoop2-compat in the patch failed.}}
that seems to be the reason for failed tests too.
> Add metrics for storefile age
> -
>
> Key: HBASE-15135
> URL: https://issues.apache.org/jira/browse/HBASE-15135
> Project: HBase
> Issue Type: New Feature
>Reporter: Elliott Clark
>Assignee: Mikhail Antonov
> Attachments: HBASE-15135-v2.patch, HBASE-15135.patch
>
>
> In order to make sure that compactions are fully up to date it would be nice
> to have metrics on:
> * Max storefile age
> * Min storefile age
> * Average storefile age
> * Number of reference files
> If we could have those metrics per region and per regionserver it would be
> awesome.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15135) Add metrics for storefile age
[ https://issues.apache.org/jira/browse/HBASE-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119742#comment-15119742 ] Sean Busbey commented on HBASE-15135: - The build error for module ordering is YETUS-280. you can build with this change in place if you [go to the precommit job|https://builds.apache.org/view/PreCommit%20Builds/job/PreCommit-HBASE-Build/] and use "build with parameters" to check the USE_YETUS_PRERELEASE option. > Add metrics for storefile age > - > > Key: HBASE-15135 > URL: https://issues.apache.org/jira/browse/HBASE-15135 > Project: HBase > Issue Type: New Feature >Reporter: Elliott Clark >Assignee: Mikhail Antonov > Attachments: HBASE-15135-v2.patch, HBASE-15135.patch > > > In order to make sure that compactions are fully up to date it would be nice > to have metrics on: > * Max storefile age > * Min storefile age > * Average storefile age > * Number of reference files > If we could have those metrics per region and per regionserver it would be > awesome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15135) Add metrics for storefile age
[ https://issues.apache.org/jira/browse/HBASE-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119762#comment-15119762 ] Mikhail Antonov commented on HBASE-15135: - Thanks [~busbey]! That puzzled me... Let me try with USE_YETUS_PRERELEASE set on. > Add metrics for storefile age > - > > Key: HBASE-15135 > URL: https://issues.apache.org/jira/browse/HBASE-15135 > Project: HBase > Issue Type: New Feature >Reporter: Elliott Clark >Assignee: Mikhail Antonov > Attachments: HBASE-15135-v2.patch, HBASE-15135.patch > > > In order to make sure that compactions are fully up to date it would be nice > to have metrics on: > * Max storefile age > * Min storefile age > * Average storefile age > * Number of reference files > If we could have those metrics per region and per regionserver it would be > awesome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15021) hadoopqa doing false positives
[
https://issues.apache.org/jira/browse/HBASE-15021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119618#comment-15119618
]
stack commented on HBASE-15021:
---
[~ndimiduk] Probably could have kept this out of 1.1 since it build messing. No
harm in having a test class for builds.
> hadoopqa doing false positives
> --
>
> Key: HBASE-15021
> URL: https://issues.apache.org/jira/browse/HBASE-15021
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: 15021.patch, 15021.thrownpe.patch, 15021.thrownpe.patch,
> 15021.thrownpe.patch, 15021.thrownpe.patch
>
>
> https://builds.apache.org/job/PreCommit-HBASE-Build/16930/consoleText says:
> {color:green}+1 core tests{color}. The patch passed unit tests in .
> ...but here is what happened:
> {code}
> ...
> Results :
> Tests in error:
> org.apache.hadoop.hbase.regionserver.TestRSStatusServlet.testBasic(org.apache.hadoop.hbase.regionserver.TestRSStatusServlet)
> Run 1: TestRSStatusServlet.testBasic:105 � NullPointer
> Run 2: TestRSStatusServlet.testBasic:105 � NullPointer
> Run 3: TestRSStatusServlet.testBasic:105 � NullPointer
> org.apache.hadoop.hbase.regionserver.TestRSStatusServlet.testWithRegions(org.apache.hadoop.hbase.regionserver.TestRSStatusServlet)
> Run 1: TestRSStatusServlet.testWithRegions:119 � NullPointer
> Run 2: TestRSStatusServlet.testWithRegions:119 � NullPointer
> Run 3: TestRSStatusServlet.testWithRegions:119 � NullPointer
> Tests run: 1033, Failures: 0, Errors: 2, Skipped: 21
> ...
> [INFO] Apache HBase - Server . FAILURE
> [17:54.559s]
> ...
> {code}
> Why we reporting pass when it failed?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119628#comment-15119628 ] stack commented on HBASE-15146: --- bq. Not a single unit test repeats between the sets. I'll do a weeding again (unless folks are up for fixing the flakies). I put back the stochastic test but it is flakey. Will purge again. > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 1.2.0 > > Attachments: HBASE-15146-v7.patch, HBASE-15146-v8.patch, > HBASE-15146-v8.patch, HBASE-15146.0.patch, HBASE-15146.1.patch, > HBASE-15146.2.patch, HBASE-15146.3.patch, HBASE-15146.4.patch, > HBASE-15146.5.patch, HBASE-15146.6.patch > > > Blocking on the epoll thread is awful. The new rpc scheduler can have lots of > different queues. Those queues have different capacity limits. Currently the > dispatch method can block trying to add the the blocking queue in any of the > schedulers. > This causes readers to block, tcp acks are delayed, and everything slows down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15135) Add metrics for storefile age
[ https://issues.apache.org/jira/browse/HBASE-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119644#comment-15119644 ] Mikhail Antonov commented on HBASE-15135: - Ok this is kind of weird. The build is passing on my local. When I look at https://builds.apache.org/job/PreCommit-HBASE-Build/307/console I see: Patch maven install cd /testptch/hbase/hbase-hadoop2-compat mvn -Dmaven.repo.local=/home/jenkins/yetus-m2/hbase-master-0 -DHBasePatchProcess -fae clean install -DskipTests=true -Dmaven.javadoc.skip=true > /testptch/patchprocess/patch-mvninstall-hbase-hadoop2-compat.txt 2>&1 Elapsed: 0m 12s cd /testptch/hbase/hbase-hadoop-compat mvn -Dmaven.repo.local=/home/jenkins/yetus-m2/hbase-master-0 -DHBasePatchProcess -fae clean install -DskipTests=true -Dmaven.javadoc.skip=true > /testptch/patchprocess/patch-mvninstall-hbase-hadoop-compat.txt 2>&1 Elapsed: 0m 10s So hbase-hadoop2-compat is now getting built before hbase-hadoop-compat, though hbase-hadoop2-compat pom says it depends on hbase-hadoop-compat? Hmm.. [~busbey] do you have any insight by chance? > Add metrics for storefile age > - > > Key: HBASE-15135 > URL: https://issues.apache.org/jira/browse/HBASE-15135 > Project: HBase > Issue Type: New Feature >Reporter: Elliott Clark >Assignee: Mikhail Antonov > Attachments: HBASE-15135-v2.patch, HBASE-15135.patch > > > In order to make sure that compactions are fully up to date it would be nice > to have metrics on: > * Max storefile age > * Min storefile age > * Average storefile age > * Number of reference files > If we could have those metrics per region and per regionserver it would be > awesome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15135) Add metrics for storefile age
[
https://issues.apache.org/jira/browse/HBASE-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120934#comment-15120934
]
Hadoop QA commented on HBASE-15135:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 45s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 50s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 38s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 15s
{color} | {color:green} hbase-hadoop-compat in the patch passed with JDK
v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 17s
{color} | {color:green} hbase-hadoop2-compat in the patch passed with JDK
v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 88m 43s
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s
{color} | {color:green} hbase-hadoop-compat in the patch passed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 31s
{color} | {color:green} hbase-hadoop2-compat in the patch passed with JDK
[jira] [Updated] (HBASE-15173) Execute mergeRegions RPC call as the request user
[ https://issues.apache.org/jira/browse/HBASE-15173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15173: --- Fix Version/s: 1.3.0 2.0.0 > Execute mergeRegions RPC call as the request user > - > > Key: HBASE-15173 > URL: https://issues.apache.org/jira/browse/HBASE-15173 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15173.v1.patch, HBASE-15173.v2.patch, > HBASE-15173.v2.patch, HBASE-15173.v3.patch, HBASE-15173.v3.patch > > > This is follow up to HBASE-15132 > Master currently sends mergeRegions RPC to region server under user 'hbase'. > This issue is to execute mergeRegions RPC call as the request user > See tail of HBASE-15132 for related discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15171) Avoid counting duplicated kv and generating lots of small hfiles in PutSortReducer
[
https://issues.apache.org/jira/browse/HBASE-15171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-15171:
---
Fix Version/s: 1.3.0
> Avoid counting duplicated kv and generating lots of small hfiles in
> PutSortReducer
> --
>
> Key: HBASE-15171
> URL: https://issues.apache.org/jira/browse/HBASE-15171
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0, 1.1.2, 0.98.17
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-15171.patch, HBASE-15171.patch, HBASE-15171.patch
>
>
> Once there was one of our online user writing huge number of duplicated kvs
> during bulkload, and we found it generated lots of small hfiles and slows
> down the whole process.
> After debugging, we found in PutSortReducer#reduce, although it already tried
> to handle the pathological case by setting a threshold for single-row size
> and having a TreeMap to avoid writing out duplicated kv, it forgot to exclude
> duplicated kv from the accumulated size. As shown in below code segment:
> {code}
> while (iter.hasNext() && curSize < threshold) {
> Put p = iter.next();
> for (List cells: p.getFamilyCellMap().values()) {
> for (Cell cell: cells) {
> KeyValue kv = KeyValueUtil.ensureKeyValue(cell);
> map.add(kv);
> curSize += kv.heapSize();
> }
> }
> }
> {code}
> We should move the {{curSize += kv.heapSize();}} line out of the outer for
> loop
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15171) Avoid counting duplicate kv and generating lots of small hfiles in PutSortReducer
[
https://issues.apache.org/jira/browse/HBASE-15171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119883#comment-15119883
]
Ted Yu commented on HBASE-15171:
Yu:
Mind attaching an addendum that addresses Ram's comment ?
> Avoid counting duplicate kv and generating lots of small hfiles in
> PutSortReducer
> -
>
> Key: HBASE-15171
> URL: https://issues.apache.org/jira/browse/HBASE-15171
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0, 1.1.2, 0.98.17
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-15171.patch, HBASE-15171.patch, HBASE-15171.patch
>
>
> Once there was one of our online user writing huge number of duplicated kvs
> during bulkload, and we found it generated lots of small hfiles and slows
> down the whole process.
> After debugging, we found in PutSortReducer#reduce, although it already tried
> to handle the pathological case by setting a threshold for single-row size
> and having a TreeMap to avoid writing out duplicated kv, it forgot to exclude
> duplicated kv from the accumulated size. As shown in below code segment:
> {code}
> while (iter.hasNext() && curSize < threshold) {
> Put p = iter.next();
> for (List cells: p.getFamilyCellMap().values()) {
> for (Cell cell: cells) {
> KeyValue kv = KeyValueUtil.ensureKeyValue(cell);
> map.add(kv);
> curSize += kv.heapSize();
> }
> }
> }
> {code}
> We should move the {{curSize += kv.heapSize();}} line out of the outer for
> loop
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15173) Execute mergeRegions RPC call as the request user
[ https://issues.apache.org/jira/browse/HBASE-15173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119929#comment-15119929 ] Enis Soztutar commented on HBASE-15173: --- LGTM. > Execute mergeRegions RPC call as the request user > - > > Key: HBASE-15173 > URL: https://issues.apache.org/jira/browse/HBASE-15173 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15173.v1.patch, HBASE-15173.v2.patch, > HBASE-15173.v2.patch, HBASE-15173.v3.patch, HBASE-15173.v3.patch > > > This is follow up to HBASE-15132 > Master currently sends mergeRegions RPC to region server under user 'hbase'. > This issue is to execute mergeRegions RPC call as the request user > See tail of HBASE-15132 for related discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15135) Add metrics for storefile age
[
https://issues.apache.org/jira/browse/HBASE-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119745#comment-15119745
]
Sean Busbey commented on HBASE-15135:
-
and the findbugs complaint says that it's for pre-existing issues on the
changed module, not for the current patch.
the section on the patch says:
{quote}
+1 findbugs3m 6s the patch passed
{quote}
disabling the -1 for pre-existing issues is already filed as HBASE-15151
> Add metrics for storefile age
> -
>
> Key: HBASE-15135
> URL: https://issues.apache.org/jira/browse/HBASE-15135
> Project: HBase
> Issue Type: New Feature
>Reporter: Elliott Clark
>Assignee: Mikhail Antonov
> Attachments: HBASE-15135-v2.patch, HBASE-15135.patch
>
>
> In order to make sure that compactions are fully up to date it would be nice
> to have metrics on:
> * Max storefile age
> * Min storefile age
> * Average storefile age
> * Number of reference files
> If we could have those metrics per region and per regionserver it would be
> awesome.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15171) Avoid counting duplicate kv and generating lots of small hfiles in PutSortReducer
[
https://issues.apache.org/jira/browse/HBASE-15171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119865#comment-15119865
]
ramkrishna.s.vasudevan commented on HBASE-15171:
Instead of iterating again the map, can we just get the return value of
map.add(kv), it it is false don't add the curSize?
add() javadoc says this
{code}
add
public boolean add(E e)
Adds the specified element to this set if it is not already present. More
formally, adds the specified element e to this set if the set contains no
element e2 such that (e==null ? e2==null : e.equals(e2)). If this set already
contains the element, the call leaves the set unchanged and returns false.
{code}
> Avoid counting duplicate kv and generating lots of small hfiles in
> PutSortReducer
> -
>
> Key: HBASE-15171
> URL: https://issues.apache.org/jira/browse/HBASE-15171
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0, 1.1.2, 0.98.17
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-15171.patch, HBASE-15171.patch, HBASE-15171.patch
>
>
> Once there was one of our online user writing huge number of duplicated kvs
> during bulkload, and we found it generated lots of small hfiles and slows
> down the whole process.
> After debugging, we found in PutSortReducer#reduce, although it already tried
> to handle the pathological case by setting a threshold for single-row size
> and having a TreeMap to avoid writing out duplicated kv, it forgot to exclude
> duplicated kv from the accumulated size. As shown in below code segment:
> {code}
> while (iter.hasNext() && curSize < threshold) {
> Put p = iter.next();
> for (List cells: p.getFamilyCellMap().values()) {
> for (Cell cell: cells) {
> KeyValue kv = KeyValueUtil.ensureKeyValue(cell);
> map.add(kv);
> curSize += kv.heapSize();
> }
> }
> }
> {code}
> We should move the {{curSize += kv.heapSize();}} line out of the outer for
> loop
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15171) Avoid counting duplicate kv and generating lots of small hfiles in PutSortReducer
[
https://issues.apache.org/jira/browse/HBASE-15171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-15171:
---
Summary: Avoid counting duplicate kv and generating lots of small hfiles in
PutSortReducer (was: Avoid counting duplicated kv and generating lots of small
hfiles in PutSortReducer)
> Avoid counting duplicate kv and generating lots of small hfiles in
> PutSortReducer
> -
>
> Key: HBASE-15171
> URL: https://issues.apache.org/jira/browse/HBASE-15171
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0, 1.1.2, 0.98.17
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-15171.patch, HBASE-15171.patch, HBASE-15171.patch
>
>
> Once there was one of our online user writing huge number of duplicated kvs
> during bulkload, and we found it generated lots of small hfiles and slows
> down the whole process.
> After debugging, we found in PutSortReducer#reduce, although it already tried
> to handle the pathological case by setting a threshold for single-row size
> and having a TreeMap to avoid writing out duplicated kv, it forgot to exclude
> duplicated kv from the accumulated size. As shown in below code segment:
> {code}
> while (iter.hasNext() && curSize < threshold) {
> Put p = iter.next();
> for (List cells: p.getFamilyCellMap().values()) {
> for (Cell cell: cells) {
> KeyValue kv = KeyValueUtil.ensureKeyValue(cell);
> map.add(kv);
> curSize += kv.heapSize();
> }
> }
> }
> {code}
> We should move the {{curSize += kv.heapSize();}} line out of the outer for
> loop
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15171) Avoid counting duplicated kv and generating lots of small hfiles in PutSortReducer
[
https://issues.apache.org/jira/browse/HBASE-15171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119854#comment-15119854
]
Hadoop QA commented on HBASE-15171:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 49s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
21m 19s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 78m 11s
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 79m 27s
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 199m 36s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-01-27 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12784644/HBASE-15171.patch |
| JIRA Issue | HBASE-15171 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux c7ff12a9c5aa
[jira] [Commented] (HBASE-15171) Avoid counting duplicate kv and generating lots of small hfiles in PutSortReducer
[
https://issues.apache.org/jira/browse/HBASE-15171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120006#comment-15120006
]
Hudson commented on HBASE-15171:
SUCCESS: Integrated in HBase-1.3-IT #464 (See
[https://builds.apache.org/job/HBase-1.3-IT/464/])
HBASE-15171 Avoid counting duplicate kv and generating lots of small (tedyu:
rev 630ad95c923f642d006274b9b1a14397a6713412)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/PutSortReducer.java
> Avoid counting duplicate kv and generating lots of small hfiles in
> PutSortReducer
> -
>
> Key: HBASE-15171
> URL: https://issues.apache.org/jira/browse/HBASE-15171
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0, 1.1.2, 0.98.17
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-15171.patch, HBASE-15171.patch, HBASE-15171.patch
>
>
> Once there was one of our online user writing huge number of duplicated kvs
> during bulkload, and we found it generated lots of small hfiles and slows
> down the whole process.
> After debugging, we found in PutSortReducer#reduce, although it already tried
> to handle the pathological case by setting a threshold for single-row size
> and having a TreeMap to avoid writing out duplicated kv, it forgot to exclude
> duplicated kv from the accumulated size. As shown in below code segment:
> {code}
> while (iter.hasNext() && curSize < threshold) {
> Put p = iter.next();
> for (List cells: p.getFamilyCellMap().values()) {
> for (Cell cell: cells) {
> KeyValue kv = KeyValueUtil.ensureKeyValue(cell);
> map.add(kv);
> curSize += kv.heapSize();
> }
> }
> }
> {code}
> We should move the {{curSize += kv.heapSize();}} line out of the outer for
> loop
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15177) Reduce garbage created under high load
[
https://issues.apache.org/jira/browse/HBASE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120021#comment-15120021
]
Enis Soztutar commented on HBASE-15177:
---
Thanks for checking.
bq. So the reason for going over to BB and BBInputStream is for supporting
offheap BB and BB reuse right?
Yes that was the idea, but the patch does not contain any changes for this. I
have tried with offheap IPC BBs with and without reuse, but the problem is with
PB's CodedInputStream.
This allocates 4K buffer and copy bytes into this buffer when using InputStream
over DBB.
{code}
private CodedInputStream(final InputStream input) {
buffer = new byte[BUFFER_SIZE];
bufferSize = 0;
bufferPos = 0;
totalBytesRetired = 0;
this.input = input;
}
{code}
Because of this extra 4K buffer allocation and extra copying of the bytes from
offheap to on-heap, the benefits of reading into off-heap buffer is not
realized and there is way more garbage created because of this. Also even if
you re-use offheap BBs, CodedInputStream will allocate 4K byte[]'s anyway. I am
testing with gets where the rpc request size is obviously much smaller than 4K.
Maybe we need to re-write CodedInputStream?
bq. But does that directly have an impact on the GC for now?
We are creating 2 CodedInputStreams today for reading each RPC call. This patch
saves on 1.
bq. This change from 'null' to 'controller' may not be really needed for the
close() scanner call?
We are doing a close scanner RPC, and it should also have a priority
associated.
bq. We are doing some PoC in this area. Could not test it for perf and that is
why did not raise any Jira. The idea is extension what HBASE-14490. says. We
read reqs into off heap ByteBuffers that come from pool.
Would be good to have this, but we have to be careful about the CIS as above.
bq. In the patch there is some changes around setting priority as well.
Intended? I did not see the patch in detail.
Yes, this is to save on the unnecessary garbage being created from
AnnotationReadingPriorityFunction. We already set the priority from the client
side for gets and multis. But it was only used on the server side if it is a
multi. This patch fixes that, and also makes sure that scanner open and close
sets the priority as well.
> Reduce garbage created under high load
> --
>
> Key: HBASE-15177
> URL: https://issues.apache.org/jira/browse/HBASE-15177
> Project: HBase
> Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0
>
> Attachments: Screen Shot 2016-01-26 at 10.03.48 PM.png, Screen Shot
> 2016-01-26 at 10.03.56 PM.png, Screen Shot 2016-01-26 at 10.06.16 PM.png,
> Screen Shot 2016-01-26 at 10.15.15 PM.png, hbase-15177_v0.patch
>
>
> I have been doing some profiling of the garbage being created. The idea was
> to follow up on HBASE-14490 and experiment with offheap IPC byte buffers and
> byte buffer re-use. However, without changing the IPC byte buffers for now,
> there are a couple of (easy) improvements that I've identified from
> profiling:
> 1. RPCServer.Connection.processRequest() should work with ByteBuffer instead
> of byte[] and not-recreate CodedInputStream a few times.
> 2. RSRpcServices.getRegion() allocates two byte arrays for region, while only
> 1 is needed.
> 3. AnnotationReadingPriorityFunction is very expensive in allocations. Mainly
> it allocates the regionName byte[] to get the table name. We already set the
> priority for most of the operations (multi, get, increment, etc) but we are
> only reading the priority in case of multi. We should use the priority from
> the client side.
> Lets do the simple improvements in this patch, we can get to IPC buffer
> re-use in HBASE-14490.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15130) Backport 0.98 Scan different TimeRange for each column family
[ https://issues.apache.org/jira/browse/HBASE-15130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119985#comment-15119985 ] churro morales commented on HBASE-15130: [~busbey] ahh okay, I don't have permissions to make configuration parameter changes for the jobs. Would any one of you guys mind kicking off a job on my behalf? > Backport 0.98 Scan different TimeRange for each column family > -- > > Key: HBASE-15130 > URL: https://issues.apache.org/jira/browse/HBASE-15130 > Project: HBase > Issue Type: Bug > Components: Client, regionserver, Scanners >Affects Versions: 0.98.17 >Reporter: churro morales >Assignee: churro morales > Fix For: 0.98.18 > > Attachments: HBASE-15130-0.98.patch, HBASE-15130-0.98.v1.patch > > > branch 98 version backport for HBASE-14355 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15073) Finer grained control over normalization actions for RegionNormalizer
[
https://issues.apache.org/jira/browse/HBASE-15073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119947#comment-15119947
]
Jonathan Hsieh commented on HBASE-15073:
The 1.x releases are supposed to be rock solid, and only ideally only add
features that have gone through some bake time and hardening. This concern was
brought up with a long list of other features on previous versions as well --
for example, mob, distributed log reply, snapshots, and replication.
I'm much for receptive to an argument for on by default in 2.x/master -- there
new defaults and bigger changes are to be expected.
> Finer grained control over normalization actions for RegionNormalizer
> -
>
> Key: HBASE-15073
> URL: https://issues.apache.org/jira/browse/HBASE-15073
> Project: HBase
> Issue Type: Task
> Components: regionserver
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 15073-v1.txt, 15073-v2.txt, 15073-v2.txt, 15073-v3.txt,
> 15073-v4.txt, 15073-v5.txt
>
>
> Currently both region split and merge actions are carried out during
> normalization for underlying table.
> However, for certain use case(s) (see
> https://issues.apache.org/jira/browse/HBASE-13103?focusedCommentId=14366255=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14366255),
> it would be desirable to perform only one type of action.
> There is one boolean flag, keyed by NORMALIZATION_ENABLED_KEY, per table that
> enables normalization.
> To provide finer grained control, we have several options:
> 1. introduce another per table flag to indicate which type(s) of actions are
> allowed ("N" for disabled, "S" for split only, "M" for merge only and "MS"
> for both split and merge)
> 2. introduce another global flag to indicate which type(s) of actions are
> allowed
> 3. modify the meaning of existing flag keyed by NORMALIZATION_ENABLED_KEY so
> that it indicates type(s) of actions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13259) mmap() based BucketCache IOEngine
[ https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119994#comment-15119994 ] Zee Chen commented on HBASE-13259: -- [~ram_krish] Yes please go ahead. > mmap() based BucketCache IOEngine > - > > Key: HBASE-13259 > URL: https://issues.apache.org/jira/browse/HBASE-13259 > Project: HBase > Issue Type: New Feature > Components: BlockCache >Affects Versions: 0.98.10 >Reporter: Zee Chen >Assignee: Zee Chen >Priority: Critical > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, ioread-1.svg, > mmap-0.98-v1.patch, mmap-1.svg, mmap-trunk-v1.patch > > > Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data > from kernel space to user space. This is a good choice when the total working > set size is much bigger than the available RAM and the latency is dominated > by IO access. However, when the entire working set is small enough to fit in > the RAM, using mmap() (and subsequent memcpy()) to move data from kernel > space to user space is faster. I have run some short keyval gets tests and > the results indicate a reduction of 2%-7% of kernel CPU on my system, > depending on the load. On the gets, the latency histograms from mmap() are > identical to those from pread(), but peak throughput is close to 40% higher. > This patch modifies ByteByfferArray to allow it to specify a backing file. > Example for using this feature: set hbase.bucketcache.ioengine to > mmap:/dev/shm/bucketcache.0 in hbase-site.xml. > Attached perf measured CPU usage breakdown in flames graph. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15173) Execute mergeRegions RPC call as the request user
[
https://issues.apache.org/jira/browse/HBASE-15173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120040#comment-15120040
]
Hadoop QA commented on HBASE-15173:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 58s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
21m 30s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 50s
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 13s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 4s
{color} | {color:green} hbase-client in the patch passed with JDK v1.7.0_91.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 123m 55s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 276m 48s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hbase.replication.TestReplicationWithTags |
| JDK v1.7.0_91 Timed out junit tests |
[jira] [Updated] (HBASE-15173) Execute mergeRegions RPC call as the request user
[ https://issues.apache.org/jira/browse/HBASE-15173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15173: --- Attachment: HBASE-15173.v3.patch > Execute mergeRegions RPC call as the request user > - > > Key: HBASE-15173 > URL: https://issues.apache.org/jira/browse/HBASE-15173 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15173.v1.patch, HBASE-15173.v2.patch, > HBASE-15173.v2.patch, HBASE-15173.v3.patch, HBASE-15173.v3.patch, > HBASE-15173.v3.patch > > > This is follow up to HBASE-15132 > Master currently sends mergeRegions RPC to region server under user 'hbase'. > This issue is to execute mergeRegions RPC call as the request user > See tail of HBASE-15132 for related discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15167) Deadlock in TestNamespaceAuditor.testRegionOperations on 1.1
[
https://issues.apache.org/jira/browse/HBASE-15167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120121#comment-15120121
]
Heng Chen commented on HBASE-15167:
---
It has relates with HBASE-14650?
> Deadlock in TestNamespaceAuditor.testRegionOperations on 1.1
>
>
> Key: HBASE-15167
> URL: https://issues.apache.org/jira/browse/HBASE-15167
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 1.1.3
>Reporter: Nick Dimiduk
> Attachments: blocked.log
>
>
> This was left as a zombie after one of my test runs this weekend.
> {noformat}
> "WALProcedureStoreSyncThread" daemon prio=10 tid=0x7f3ccc209000
> nid=0x3960 in Object.wait() [0x7f3c6b6b5000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:503)
> at org.apache.hadoop.ipc.Client.call(Client.java:1397)
> - locked <0x0007f2813390> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy23.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy23.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:264)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy24.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy24.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1612)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1488)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1413)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:387)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:383)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:383)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:327)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:766)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:733)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.tryRollWriter(WALProcedureStore.java:668)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.periodicRoll(WALProcedureStore.java:711)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.syncLoop(WALProcedureStore.java:531)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.access$000(WALProcedureStore.java:66)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore$1.run(WALProcedureStore.java:180)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15173) Execute mergeRegions RPC call as the request user
[
https://issues.apache.org/jira/browse/HBASE-15173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120083#comment-15120083
]
Ted Yu commented on HBASE-15173:
>From Java 8 test log:
{code}
Caused by: java.lang.RuntimeException: Error while running command to get file
permissions : ExitCodeException exitCode=127: /bin/ls: error while loading
shared libraries: libattr.so.1: failed to map segment from shared object:
Permission denied
{code}
> Execute mergeRegions RPC call as the request user
> -
>
> Key: HBASE-15173
> URL: https://issues.apache.org/jira/browse/HBASE-15173
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-15173.v1.patch, HBASE-15173.v2.patch,
> HBASE-15173.v2.patch, HBASE-15173.v3.patch, HBASE-15173.v3.patch,
> HBASE-15173.v3.patch
>
>
> This is follow up to HBASE-15132
> Master currently sends mergeRegions RPC to region server under user 'hbase'.
> This issue is to execute mergeRegions RPC call as the request user
> See tail of HBASE-15132 for related discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15178) metrics on web UI sometimes flakey
[ https://issues.apache.org/jira/browse/HBASE-15178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120105#comment-15120105 ] Heng Chen commented on HBASE-15178: --- duplicate with HBASE-12961 > metrics on web UI sometimes flakey > -- > > Key: HBASE-15178 > URL: https://issues.apache.org/jira/browse/HBASE-15178 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.6 >Reporter: Heng Chen > Attachments: 70DD704D-7E05-49BA-AC84-0646671F67AA.png > > > see attachement -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-15178) metrics on web UI sometimes flakey
[ https://issues.apache.org/jira/browse/HBASE-15178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen resolved HBASE-15178. --- Resolution: Duplicate > metrics on web UI sometimes flakey > -- > > Key: HBASE-15178 > URL: https://issues.apache.org/jira/browse/HBASE-15178 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.6 >Reporter: Heng Chen > Attachments: 70DD704D-7E05-49BA-AC84-0646671F67AA.png > > > see attachement -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15169) Backport HBASE-14362 'TestWALProcedureStoreOnHDFS is super duper flaky' to branch-1.1
[ https://issues.apache.org/jira/browse/HBASE-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120119#comment-15120119 ] Heng Chen commented on HBASE-15169: --- TestWALProcedureStoreOnHDFS still not run. org.apache.hadoop.hbase.namespace.TestNamespaceAuditor should be fixed in HBASE-15167 > Backport HBASE-14362 'TestWALProcedureStoreOnHDFS is super duper flaky' to > branch-1.1 > - > > Key: HBASE-15169 > URL: https://issues.apache.org/jira/browse/HBASE-15169 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Nick Dimiduk >Assignee: Heng Chen >Priority: Critical > Fix For: 1.1.4 > > Attachments: HBASE-15169-branch-1.1.patch > > > This one fails consistently for me and there's a fix hanging out upstream. > Let's bring it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15181) A simple implementation of date based tiered compaction
Clara Xiong created HBASE-15181: --- Summary: A simple implementation of date based tiered compaction Key: HBASE-15181 URL: https://issues.apache.org/jira/browse/HBASE-15181 Project: HBase Issue Type: New Feature Components: Compaction Reporter: Clara Xiong Assignee: Clara Xiong Fix For: 2.0.0 This is a simple implementation of date-based tiered compaction similar to Cassandra's for the following benefits: 1. Improve date-range-based scan by structuring store files in date-based tiered layout. 2. Reduce compaction overhead. 3. Improve TTL efficiency. Perfect fit for the use cases that: 1. has mostly date-based date write and scan and a focus on the most recent data. 2. never or rarely deletes data. Out-of-order writes are handled gracefully so the data will still get to the right store file for time-range-scan and re-compacton with existing store file in the same time window is handled by ExploringCompactionPolicy. Time range overlapping among store files is tolerated and the performance impact is minimized. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15169) Backport HBASE-14362 'TestWALProcedureStoreOnHDFS is super duper flaky' to branch-1.1
[ https://issues.apache.org/jira/browse/HBASE-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-15169: -- Attachment: HBASE-15169-branch-1.1.patch retry > Backport HBASE-14362 'TestWALProcedureStoreOnHDFS is super duper flaky' to > branch-1.1 > - > > Key: HBASE-15169 > URL: https://issues.apache.org/jira/browse/HBASE-15169 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Nick Dimiduk >Assignee: Heng Chen >Priority: Critical > Fix For: 1.1.4 > > Attachments: HBASE-15169-branch-1.1.patch, > HBASE-15169-branch-1.1.patch > > > This one fails consistently for me and there's a fix hanging out upstream. > Let's bring it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15135) Add metrics for storefile age
[
https://issues.apache.org/jira/browse/HBASE-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120211#comment-15120211
]
Hadoop QA commented on HBASE-15135:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 23s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 25s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 33s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 13s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s
{color} | {color:green} hbase-hadoop-compat in the patch passed with JDK
v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s
{color} | {color:green} hbase-hadoop2-compat in the patch passed with JDK
v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 125m 39s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s
{color} | {color:green} hbase-hadoop-compat in the patch passed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} | {color:green} hbase-hadoop2-compat in the patch passed with JDK
[jira] [Created] (HBASE-15182) unify normalizer switch
Heng Chen created HBASE-15182: - Summary: unify normalizer switch Key: HBASE-15182 URL: https://issues.apache.org/jira/browse/HBASE-15182 Project: HBase Issue Type: Sub-task Reporter: Heng Chen After HBASE-15128, we will have an uniform way to do switch. Let's unify normalizer into it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
