[jira] [Commented] (HBASE-16635) RpcClient under heavy load leaks some netty bytebuf
[
https://issues.apache.org/jira/browse/HBASE-16635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492474#comment-15492474
]
Duo Zhang commented on HBASE-16635:
---
Can you add the -Dio.netty.leakDetection.level=advanced options and run again?
This will output some stacktrace infos which could help us find out where the
ByteBufs are touched.
Thanks.
> RpcClient under heavy load leaks some netty bytebuf
> ---
>
> Key: HBASE-16635
> URL: https://issues.apache.org/jira/browse/HBASE-16635
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Fix For: 2.0.0
>
>
> Yet to analyse the actual root cause.
> But the case is that when we run a PE tool with 50 threads under heavy load
> when the writes are clogged I think we have some netty Bytebuf leak. Not sure
> if it is a serious issue but we get this log
> {code}
> 2016-09-14 19:37:09,767 ERROR [Default-IPC-NioEventLoopGroup-1-16]
> util.ResourceLeakDetector: LEAK: ByteBuf.release() was not called before it's
> garbage-collected. Enable advanced leak reporting to find out where the leak
> occurred. To enable advanced leak reporting, specify the JVM option
> '-Dio.netty.leakDetection.level=advanced' or call
> ResourceLeakDetector.setLevel() See
> http://netty.io/wiki/reference-counted-objects.html for more information.
> {code}
> So reading the given link it is because of some ByteBuf that was not released
> properly by the client and hence it gets GCed automatically. Netty provides
> tips and tricks to find the root cause. Will get back here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16598) Clean up zookeeper useMulti in HBase code
[
https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492457#comment-15492457
]
Hadoop QA commented on HBASE-16598:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 41s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 10m
5s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 31s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 18s
{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 14s
{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 14s {color} |
{color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 14s {color}
| {color:red} hbase-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 9m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 36s
{color} | {color:red} The patch causes 20 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 13s
{color} | {color:red} The patch causes 20 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 49s
{color} | {color:red} The patch causes 20 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 26s
{color} | {color:red} The patch causes 20 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 2s
{color} | {color:red} The patch causes 20 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 39s
{color} | {color:red} The patch causes 20 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 16s
{color} | {color:red} The patch causes 20 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 52s
{color} | {color:red} The patch causes 20 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 31s
{color} | {color:red} The patch causes 20 errors with Hadoop v2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 14s
{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 47s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 16s {color}
|
[jira] [Updated] (HBASE-16598) Clean up zookeeper useMulti in HBase code
[ https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-16598: - Attachment: HBASE-16598-v3.patch > Clean up zookeeper useMulti in HBase code > - > > Key: HBASE-16598 > URL: https://issues.apache.org/jira/browse/HBASE-16598 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Jerry He > Attachments: HBASE-16598-v2.patch, HBASE-16598-v3.patch, > HBASE-16598.patch > > > We have zookeeper useMulti default true since HBase 1.0.0 > And in our Doc, "ZooKeeper 3.4.x is required as of HBase 1.0.0" > The benefit of useMulti is obvious. > Let's clean up the code to remove useMulti false case so that we don't have > to continue with the hassle of maintaining two version of the code (e.g. in > replication) . No go back to pre 3.4.x Zookeeper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16634) Speedup TestExportSnapshot
[
https://issues.apache.org/jira/browse/HBASE-16634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492395#comment-15492395
]
Hadoop QA commented on HBASE-16634:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 37s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 80m 7s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 116m 28s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDefaultVisLabelService
|
| |
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithCustomVisLabService
|
| |
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12828574/HBASE-16634-v0.patch |
| JIRA Issue | HBASE-16634 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux a03b4b963be4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 8ef6c76 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3556/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3556/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3556/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3556/console |
| Powered by | Apache Yetus 0.3.0
[jira] [Updated] (HBASE-16598) Clean up zookeeper useMulti in HBase code
[ https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-16598: - Attachment: HBASE-16598-v2.patch Thanks, Matteo. Removing HConstants.ZOOKEEPER_USEMULTI was the last touch in the my iterations. My bad. Attached v2. > Clean up zookeeper useMulti in HBase code > - > > Key: HBASE-16598 > URL: https://issues.apache.org/jira/browse/HBASE-16598 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Jerry He > Attachments: HBASE-16598-v2.patch, HBASE-16598.patch > > > We have zookeeper useMulti default true since HBase 1.0.0 > And in our Doc, "ZooKeeper 3.4.x is required as of HBase 1.0.0" > The benefit of useMulti is obvious. > Let's clean up the code to remove useMulti false case so that we don't have > to continue with the hassle of maintaining two version of the code (e.g. in > replication) . No go back to pre 3.4.x Zookeeper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16635) RpcClient under heavy load leaks some netty bytebuf
[
https://issues.apache.org/jira/browse/HBASE-16635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-16635:
---
Summary: RpcClient under heavy load leaks some netty bytebuf (was:
RpcClient under heave load leaks some netty bytebuf)
> RpcClient under heavy load leaks some netty bytebuf
> ---
>
> Key: HBASE-16635
> URL: https://issues.apache.org/jira/browse/HBASE-16635
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Fix For: 2.0.0
>
>
> Yet to analyse the actual root cause.
> But the case is that when we run a PE tool with 50 threads under heavy load
> when the writes are clogged I think we have some netty Bytebuf leak. Not sure
> if it is a serious issue but we get this log
> {code}
> 2016-09-14 19:37:09,767 ERROR [Default-IPC-NioEventLoopGroup-1-16]
> util.ResourceLeakDetector: LEAK: ByteBuf.release() was not called before it's
> garbage-collected. Enable advanced leak reporting to find out where the leak
> occurred. To enable advanced leak reporting, specify the JVM option
> '-Dio.netty.leakDetection.level=advanced' or call
> ResourceLeakDetector.setLevel() See
> http://netty.io/wiki/reference-counted-objects.html for more information.
> {code}
> So reading the given link it is because of some ByteBuf that was not released
> properly by the client and hence it gets GCed automatically. Netty provides
> tips and tricks to find the root cause. Will get back here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16635) RpcClient under heave load leaks some netty bytebuf
ramkrishna.s.vasudevan created HBASE-16635:
--
Summary: RpcClient under heave load leaks some netty bytebuf
Key: HBASE-16635
URL: https://issues.apache.org/jira/browse/HBASE-16635
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
Fix For: 2.0.0
Yet to analyse the actual root cause.
But the case is that when we run a PE tool with 50 threads under heavy load
when the writes are clogged I think we have some netty Bytebuf leak. Not sure
if it is a serious issue but we get this log
{code}
2016-09-14 19:37:09,767 ERROR [Default-IPC-NioEventLoopGroup-1-16]
util.ResourceLeakDetector: LEAK: ByteBuf.release() was not called before it's
garbage-collected. Enable advanced leak reporting to find out where the leak
occurred. To enable advanced leak reporting, specify the JVM option
'-Dio.netty.leakDetection.level=advanced' or call
ResourceLeakDetector.setLevel() See
http://netty.io/wiki/reference-counted-objects.html for more information.
{code}
So reading the given link it is because of some ByteBuf that was not released
properly by the client and hence it gets GCed automatically. Netty provides
tips and tricks to find the root cause. Will get back here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16620) Fix backup command-line tool usability issues
[ https://issues.apache.org/jira/browse/HBASE-16620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492357#comment-15492357 ] Ted Yu commented on HBASE-16620: How about giving IGNORE a more descriptive name so that other people understands its purpose. > Fix backup command-line tool usability issues > - > > Key: HBASE-16620 > URL: https://issues.apache.org/jira/browse/HBASE-16620 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Attachments: 16620-v2.patch, 16620-v4.patch, 16620.addendum, > HBASE-16620-v1.patch, HBASE-16620-v3.patch, HBASE-16620.add.patch > > > We need to address issues found by [~saint@gmail.com] > https://issues.apache.org/jira/browse/HBASE-7912?focusedCommentId=15484865=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15484865 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16620) Fix backup command-line tool usability issues
[ https://issues.apache.org/jira/browse/HBASE-16620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492316#comment-15492316 ] Vladimir Rodionov commented on HBASE-16620: --- Check the code. execute is void and can throw only IOException We need to report successful finish (no exception), command failure (exception), command failure due to format error (IGNORE) > Fix backup command-line tool usability issues > - > > Key: HBASE-16620 > URL: https://issues.apache.org/jira/browse/HBASE-16620 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Attachments: 16620-v2.patch, 16620-v4.patch, 16620.addendum, > HBASE-16620-v1.patch, HBASE-16620-v3.patch, HBASE-16620.add.patch > > > We need to address issues found by [~saint@gmail.com] > https://issues.apache.org/jira/browse/HBASE-7912?focusedCommentId=15484865=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15484865 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16598) Clean up zookeeper useMulti in HBase code
[
https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492286#comment-15492286
]
Matteo Bertozzi commented on HBASE-16598:
-
this does not seem to compile, since we removed ZOOKEEPER_USEMULTI from
HConstants.java
{code}
public static void multiOrSequential(ZooKeeperWatcher zkw, List
ops,
boolean runSequentialOnMultiFailure) throws KeeperException {
+if (zkw.getConfiguration().get(HConstants.ZOOKEEPER_USEMULTI) != null) {
+ LOG.warn(HConstants.ZOOKEEPER_USEMULTI + " is deprecated. Default to
true always.");
+}
{code}
but other than that, patch looks great
> Clean up zookeeper useMulti in HBase code
> -
>
> Key: HBASE-16598
> URL: https://issues.apache.org/jira/browse/HBASE-16598
> Project: HBase
> Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
> Attachments: HBASE-16598.patch
>
>
> We have zookeeper useMulti default true since HBase 1.0.0
> And in our Doc, "ZooKeeper 3.4.x is required as of HBase 1.0.0"
> The benefit of useMulti is obvious.
> Let's clean up the code to remove useMulti false case so that we don't have
> to continue with the hassle of maintaining two version of the code (e.g. in
> replication) . No go back to pre 3.4.x Zookeeper.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16165) Decrease RpcServer.callQueueSize before writeResponse causes OOM
[
https://issues.apache.org/jira/browse/HBASE-16165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492260#comment-15492260
]
Hadoop QA commented on HBASE-16165:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
15s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} branch-1.1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} branch-1.1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 52s
{color} | {color:red} hbase-server in branch-1.1 has 80 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 35s
{color} | {color:red} hbase-server in branch-1.1 failed with JDK v1.8.0_101.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 57s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 26s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_101.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 96m 27s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 131m 43s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-09-15 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12828570/HBASE-16165-branch-1.1.patch
|
| JIRA Issue | HBASE-16165 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux e2bd05fb664f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
|
[jira] [Commented] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess
[
https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492177#comment-15492177
]
Hadoop QA commented on HBASE-16631:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 15s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 54s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 51s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 135m 33s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.TestZooKeeper |
| | org.apache.hadoop.hbase.master.TestDistributedLogSplitting |
| |
org.apache.hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController
|
| | org.apache.hadoop.hbase.TestIOFencing |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12828568/HBASE-16631.v1.patch |
| JIRA Issue | HBASE-16631 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux c666960d7036 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 8ef6c76 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3554/artifact/patchprocess/patch-unit-hbase-server.txt
|
|
[jira] [Updated] (HBASE-16598) Clean up zookeeper useMulti in HBase code
[ https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-16598: - Attachment: HBASE-16598.patch > Clean up zookeeper useMulti in HBase code > - > > Key: HBASE-16598 > URL: https://issues.apache.org/jira/browse/HBASE-16598 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Jerry He > Attachments: HBASE-16598.patch > > > We have zookeeper useMulti default true since HBase 1.0.0 > And in our Doc, "ZooKeeper 3.4.x is required as of HBase 1.0.0" > The benefit of useMulti is obvious. > Let's clean up the code to remove useMulti false case so that we don't have > to continue with the hassle of maintaining two version of the code (e.g. in > replication) . No go back to pre 3.4.x Zookeeper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16598) Clean up zookeeper useMulti in HBase code
[ https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-16598: - Status: Patch Available (was: Open) > Clean up zookeeper useMulti in HBase code > - > > Key: HBASE-16598 > URL: https://issues.apache.org/jira/browse/HBASE-16598 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Jerry He > Attachments: HBASE-16598.patch > > > We have zookeeper useMulti default true since HBase 1.0.0 > And in our Doc, "ZooKeeper 3.4.x is required as of HBase 1.0.0" > The benefit of useMulti is obvious. > Let's clean up the code to remove useMulti false case so that we don't have > to continue with the hassle of maintaining two version of the code (e.g. in > replication) . No go back to pre 3.4.x Zookeeper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-16599) Expand the use of dynamic jar dir
[ https://issues.apache.org/jira/browse/HBASE-16599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He reassigned HBASE-16599: Assignee: Jerry He > Expand the use of dynamic jar dir > - > > Key: HBASE-16599 > URL: https://issues.apache.org/jira/browse/HBASE-16599 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Jerry He > > Dynamic jar dir is currently used for Custom Filer and Comparator only. > Let's expand its use so that custom extensions (custom split policy, custom > compaction policy, etc) can be more easily deployed. > Useful on managed Cloud deployments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16634) Speedup TestExportSnapshot
[ https://issues.apache.org/jira/browse/HBASE-16634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16634: Status: Patch Available (was: Open) > Speedup TestExportSnapshot > -- > > Key: HBASE-16634 > URL: https://issues.apache.org/jira/browse/HBASE-16634 > Project: HBase > Issue Type: Test > Components: snapshots, test >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16634-v0.patch > > > TestExportSnapshot is a long and heavy test since has to take snapshot, > export (via MR), restore them and so on. this in general make the test flaky > due to slowness. > let's try to speed it up a bit by reducing the number of regions of the table > we are testing on 9 regions at the moment we can reduce it at least to 2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16634) Speedup TestExportSnapshot
[ https://issues.apache.org/jira/browse/HBASE-16634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16634: Attachment: HBASE-16634-v0.patch > Speedup TestExportSnapshot > -- > > Key: HBASE-16634 > URL: https://issues.apache.org/jira/browse/HBASE-16634 > Project: HBase > Issue Type: Test > Components: snapshots, test >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16634-v0.patch > > > TestExportSnapshot is a long and heavy test since has to take snapshot, > export (via MR), restore them and so on. this in general make the test flaky > due to slowness. > let's try to speed it up a bit by reducing the number of regions of the table > we are testing on 9 regions at the moment we can reduce it at least to 2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-16634) Speedup TestExportSnapshot
[
https://issues.apache.org/jira/browse/HBASE-16634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-16634:
Comment: was deleted
(was: | (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 23s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 48s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 139m 31s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegion |
| | hadoop.hbase.client.TestCloneSnapshotFromClientWithRegionReplicas |
| | hadoop.hbase.client.TestCloneSnapshotFromClient |
| | hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient |
| | org.apache.hadoop.hbase.client.TestMetaWithReplicas |
| | org.apache.hadoop.hbase.client.TestEnableTable |
| | org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12828550/HBASE-16634-v0.patch |
| JIRA Issue | HBASE-16634 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 9f3d7b6a2273 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 8ef6c76 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3553/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3553/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Commented] (HBASE-16634) Speedup TestExportSnapshot
[
https://issues.apache.org/jira/browse/HBASE-16634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492094#comment-15492094
]
Hadoop QA commented on HBASE-16634:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 23s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 48s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 139m 31s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegion |
| | hadoop.hbase.client.TestCloneSnapshotFromClientWithRegionReplicas |
| | hadoop.hbase.client.TestCloneSnapshotFromClient |
| | hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient |
| | org.apache.hadoop.hbase.client.TestMetaWithReplicas |
| | org.apache.hadoop.hbase.client.TestEnableTable |
| | org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12828550/HBASE-16634-v0.patch |
| JIRA Issue | HBASE-16634 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 9f3d7b6a2273 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 8ef6c76 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3553/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3553/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Commented] (HBASE-16620) Fix backup command-line tool usability issues
[
https://issues.apache.org/jira/browse/HBASE-16620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492092#comment-15492092
]
Ted Yu commented on HBASE-16620:
{code}
-System.exit(-1);
+throw new IOException(IGNORE);
{code}
What's the benefit of throwing IOE ?
Can you fill in better exception message ?
Thanks
> Fix backup command-line tool usability issues
> -
>
> Key: HBASE-16620
> URL: https://issues.apache.org/jira/browse/HBASE-16620
> Project: HBase
> Issue Type: Bug
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Labels: backup
> Attachments: 16620-v2.patch, 16620-v4.patch, 16620.addendum,
> HBASE-16620-v1.patch, HBASE-16620-v3.patch, HBASE-16620.add.patch
>
>
> We need to address issues found by [~saint@gmail.com]
> https://issues.apache.org/jira/browse/HBASE-7912?focusedCommentId=15484865=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15484865
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492077#comment-15492077 ] stack commented on HBASE-16631: --- OK. +1 I have no plan. Ijust thinking about what our asynchronous client API will be and that it and here in AP align. Later > Extract AsyncRequestFuture related code from AsyncProcess > - > > Key: HBASE-16631 > URL: https://issues.apache.org/jira/browse/HBASE-16631 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen >Assignee: Heng Chen > Attachments: HBASE-16631.v1.patch, HBASE-16631.v1.patch, > HBASE-16631.wip.patch > > > Now, AsyncProcess class is too large (over 2000+ lines), and there are so > many sub classes in it. > AsyncRequestFutureImpl is the biggest subclass in AP, we could extract it > out from AP to reduce the AP size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16613) Return the unused ByteBuffer to BoundedByteBufferPool when no cell is retrieved from the CellScanner
[
https://issues.apache.org/jira/browse/HBASE-16613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492049#comment-15492049
]
Hudson commented on HBASE-16613:
FAILURE: Integrated in Jenkins build HBase-1.2-JDK7 #27 (See
[https://builds.apache.org/job/HBase-1.2-JDK7/27/])
HBASE-16613 Return the unused ByteBuffer to BoundedByteBufferPool when (tedyu:
rev b7888df614ca23b3fd030ef2733f2a044f9280fa)
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java
> Return the unused ByteBuffer to BoundedByteBufferPool when no cell is
> retrieved from the CellScanner
>
>
> Key: HBASE-16613
> URL: https://issues.apache.org/jira/browse/HBASE-16613
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 1.3.0, 1.4.0, 1.2.4
>
> Attachments: HBASE-16613.branch-1.v0.patch,
> HBASE-16613.branch-1.v1.patch, HBASE-16613.branch-1.v1.patch
>
>
> The critical code is shown below:
> {code:title=IPCUtil.java|borderStyle=solid}
> // We should put the ByteBuffer into pool before return null
> public ByteBuffer buildCellBlock(final Codec codec, final CompressionCodec
> compressor,
> final CellScanner cellScanner, final BoundedByteBufferPool pool) {
> ...
> if (pool != null) {
> ByteBuffer bb = pool.getBuffer();
> bufferSize = bb.capacity();
> baos = new ByteBufferOutputStream(bb);
> }
> ...
> int count = 0;
> while (cellScanner.advance()) {
> encoder.write(cellScanner.current());
> count++;
> }
> encoder.flush();
> if (count == 0) return null;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16381) Shell deleteall command should support row key prefixes
[ https://issues.apache.org/jira/browse/HBASE-16381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492032#comment-15492032 ] Jerry He commented on HBASE-16381: -- Looks good. I like that option. Good work. > Shell deleteall command should support row key prefixes > --- > > Key: HBASE-16381 > URL: https://issues.apache.org/jira/browse/HBASE-16381 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Andrew Purtell >Assignee: Yi Liang >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16381-V1.patch, HBASE-16381-V2.patch, > HBASE-16381-V3.patch, HBASE-16381-V4.patch, HBASE-16381-V5.patch, > HBASE-16381-V6.patch > > > The shell's deleteall command should support deleting a row range using a row > key prefix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16613) Return the unused ByteBuffer to BoundedByteBufferPool when no cell is retrieved from the CellScanner
[
https://issues.apache.org/jira/browse/HBASE-16613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492011#comment-15492011
]
Hudson commented on HBASE-16613:
SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #16 (See
[https://builds.apache.org/job/HBase-1.3-JDK8/16/])
HBASE-16613 Return the unused ByteBuffer to BoundedByteBufferPool when (tedyu:
rev 55d3dc415c304cabe6ed140380ec69ce08abab63)
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java
> Return the unused ByteBuffer to BoundedByteBufferPool when no cell is
> retrieved from the CellScanner
>
>
> Key: HBASE-16613
> URL: https://issues.apache.org/jira/browse/HBASE-16613
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 1.3.0, 1.4.0, 1.2.4
>
> Attachments: HBASE-16613.branch-1.v0.patch,
> HBASE-16613.branch-1.v1.patch, HBASE-16613.branch-1.v1.patch
>
>
> The critical code is shown below:
> {code:title=IPCUtil.java|borderStyle=solid}
> // We should put the ByteBuffer into pool before return null
> public ByteBuffer buildCellBlock(final Codec codec, final CompressionCodec
> compressor,
> final CellScanner cellScanner, final BoundedByteBufferPool pool) {
> ...
> if (pool != null) {
> ByteBuffer bb = pool.getBuffer();
> bufferSize = bb.capacity();
> baos = new ByteBufferOutputStream(bb);
> }
> ...
> int count = 0;
> while (cellScanner.advance()) {
> encoder.write(cellScanner.current());
> count++;
> }
> encoder.flush();
> if (count == 0) return null;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16613) Return the unused ByteBuffer to BoundedByteBufferPool when no cell is retrieved from the CellScanner
[
https://issues.apache.org/jira/browse/HBASE-16613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492009#comment-15492009
]
Hudson commented on HBASE-16613:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #23 (See
[https://builds.apache.org/job/HBase-1.2-JDK8/23/])
HBASE-16613 Return the unused ByteBuffer to BoundedByteBufferPool when (tedyu:
rev b7888df614ca23b3fd030ef2733f2a044f9280fa)
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java
> Return the unused ByteBuffer to BoundedByteBufferPool when no cell is
> retrieved from the CellScanner
>
>
> Key: HBASE-16613
> URL: https://issues.apache.org/jira/browse/HBASE-16613
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 1.3.0, 1.4.0, 1.2.4
>
> Attachments: HBASE-16613.branch-1.v0.patch,
> HBASE-16613.branch-1.v1.patch, HBASE-16613.branch-1.v1.patch
>
>
> The critical code is shown below:
> {code:title=IPCUtil.java|borderStyle=solid}
> // We should put the ByteBuffer into pool before return null
> public ByteBuffer buildCellBlock(final Codec codec, final CompressionCodec
> compressor,
> final CellScanner cellScanner, final BoundedByteBufferPool pool) {
> ...
> if (pool != null) {
> ByteBuffer bb = pool.getBuffer();
> bufferSize = bb.capacity();
> baos = new ByteBufferOutputStream(bb);
> }
> ...
> int count = 0;
> while (cellScanner.advance()) {
> encoder.write(cellScanner.current());
> count++;
> }
> encoder.flush();
> if (count == 0) return null;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16620) Fix backup command-line tool usability issues
[ https://issues.apache.org/jira/browse/HBASE-16620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491998#comment-15491998 ] Vladimir Rodionov commented on HBASE-16620: --- [~tedyu], addendum patch > Fix backup command-line tool usability issues > - > > Key: HBASE-16620 > URL: https://issues.apache.org/jira/browse/HBASE-16620 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Attachments: 16620-v2.patch, 16620-v4.patch, 16620.addendum, > HBASE-16620-v1.patch, HBASE-16620-v3.patch, HBASE-16620.add.patch > > > We need to address issues found by [~saint@gmail.com] > https://issues.apache.org/jira/browse/HBASE-7912?focusedCommentId=15484865=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15484865 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16620) Fix backup command-line tool usability issues
[ https://issues.apache.org/jira/browse/HBASE-16620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16620: -- Attachment: HBASE-16620.add.patch Addendum patch. Adds more tests and removes all System.exit in BackupCommands > Fix backup command-line tool usability issues > - > > Key: HBASE-16620 > URL: https://issues.apache.org/jira/browse/HBASE-16620 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Attachments: 16620-v2.patch, 16620-v4.patch, 16620.addendum, > HBASE-16620-v1.patch, HBASE-16620-v3.patch, HBASE-16620.add.patch > > > We need to address issues found by [~saint@gmail.com] > https://issues.apache.org/jira/browse/HBASE-7912?focusedCommentId=15484865=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15484865 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-16620) Fix backup command-line tool usability issues
[ https://issues.apache.org/jira/browse/HBASE-16620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov reopened HBASE-16620: --- > Fix backup command-line tool usability issues > - > > Key: HBASE-16620 > URL: https://issues.apache.org/jira/browse/HBASE-16620 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Attachments: 16620-v2.patch, 16620-v4.patch, 16620.addendum, > HBASE-16620-v1.patch, HBASE-16620-v3.patch, HBASE-16620.add.patch > > > We need to address issues found by [~saint@gmail.com] > https://issues.apache.org/jira/browse/HBASE-7912?focusedCommentId=15484865=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15484865 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16165) Decrease RpcServer.callQueueSize before writeResponse causes OOM
[
https://issues.apache.org/jira/browse/HBASE-16165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491994#comment-15491994
]
Guanghao Zhang commented on HBASE-16165:
HBASE-16165.patch can be used for master, branch-1.4, branch-1.3 and
branch-1.2. HBASE-16165-branch-1.1.patch can be used for branch-1.1 and 0.98.
> Decrease RpcServer.callQueueSize before writeResponse causes OOM
>
>
> Key: HBASE-16165
> URL: https://issues.apache.org/jira/browse/HBASE-16165
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC, rpc
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.6, 1.2.3, 0.98.22
>Reporter: Duo Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Labels: trivial
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 0.98.23, 1.2.4
>
> Attachments: HBASE-16165-branch-1.1.patch, HBASE-16165.patch
>
>
> In RpcServer, we use {{callQueueSizeInBytes}} to avoid queuing too many calls
> which causes OOM. But in {{CallRunner.run}}, we decrease it before send the
> response back. And even after calling {{sendResponseIfReady}}, the call
> object could stay in our heap for a long time if we can not write out the
> response(That's why we need a Responder thread...). This makes it possible
> that the actual size of all call object in heap is larger than
> {{maxQueueSizeInBytes}} and causes OOM.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16165) Decrease RpcServer.callQueueSize before writeResponse causes OOM
[
https://issues.apache.org/jira/browse/HBASE-16165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-16165:
---
Attachment: HBASE-16165-branch-1.1.patch
> Decrease RpcServer.callQueueSize before writeResponse causes OOM
>
>
> Key: HBASE-16165
> URL: https://issues.apache.org/jira/browse/HBASE-16165
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC, rpc
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.6, 1.2.3, 0.98.22
>Reporter: Duo Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Labels: trivial
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 0.98.23, 1.2.4
>
> Attachments: HBASE-16165-branch-1.1.patch, HBASE-16165.patch
>
>
> In RpcServer, we use {{callQueueSizeInBytes}} to avoid queuing too many calls
> which causes OOM. But in {{CallRunner.run}}, we decrease it before send the
> response back. And even after calling {{sendResponseIfReady}}, the call
> object could stay in our heap for a long time if we can not write out the
> response(That's why we need a Responder thread...). This makes it possible
> that the actual size of all call object in heap is larger than
> {{maxQueueSizeInBytes}} and causes OOM.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-16631: -- Attachment: HBASE-16631.v1.patch retry > Extract AsyncRequestFuture related code from AsyncProcess > - > > Key: HBASE-16631 > URL: https://issues.apache.org/jira/browse/HBASE-16631 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen >Assignee: Heng Chen > Attachments: HBASE-16631.v1.patch, HBASE-16631.v1.patch, > HBASE-16631.wip.patch > > > Now, AsyncProcess class is too large (over 2000+ lines), and there are so > many sub classes in it. > AsyncRequestFutureImpl is the biggest subclass in AP, we could extract it > out from AP to reduce the AP size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess
[
https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491921#comment-15491921
]
Heng Chen edited comment on HBASE-16631 at 9/15/16 12:43 AM:
-
{quote}
We'll probably want to change our Future to subclass CompletableFuture but that
can come later.
{quote}
Any issue about it? I also have some plan to refactor AP, for example, unify
submit and submitAll, not sure what is your plan.
was (Author: chenheng):
{quote}
We'll probably want to change our Future to subclass CompletableFuture but that
can come later.
{quote}
Any issue about it?
> Extract AsyncRequestFuture related code from AsyncProcess
> -
>
> Key: HBASE-16631
> URL: https://issues.apache.org/jira/browse/HBASE-16631
> Project: HBase
> Issue Type: Sub-task
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-16631.v1.patch, HBASE-16631.wip.patch
>
>
> Now, AsyncProcess class is too large (over 2000+ lines), and there are so
> many sub classes in it.
> AsyncRequestFutureImpl is the biggest subclass in AP, we could extract it
> out from AP to reduce the AP size.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess
[
https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491921#comment-15491921
]
Heng Chen commented on HBASE-16631:
---
{quote}
We'll probably want to change our Future to subclass CompletableFuture but that
can come later.
{quote}
Any issue about it?
> Extract AsyncRequestFuture related code from AsyncProcess
> -
>
> Key: HBASE-16631
> URL: https://issues.apache.org/jira/browse/HBASE-16631
> Project: HBase
> Issue Type: Sub-task
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-16631.v1.patch, HBASE-16631.wip.patch
>
>
> Now, AsyncProcess class is too large (over 2000+ lines), and there are so
> many sub classes in it.
> AsyncRequestFutureImpl is the biggest subclass in AP, we could extract it
> out from AP to reduce the AP size.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16381) Shell deleteall command should support row key prefixes
[ https://issues.apache.org/jira/browse/HBASE-16381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491904#comment-15491904 ] Heng Chen commented on HBASE-16381: --- LGTM. +1 Will commit it if no other concerns. > Shell deleteall command should support row key prefixes > --- > > Key: HBASE-16381 > URL: https://issues.apache.org/jira/browse/HBASE-16381 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Andrew Purtell >Assignee: Yi Liang >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16381-V1.patch, HBASE-16381-V2.patch, > HBASE-16381-V3.patch, HBASE-16381-V4.patch, HBASE-16381-V5.patch, > HBASE-16381-V6.patch > > > The shell's deleteall command should support deleting a row range using a row > key prefix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491896#comment-15491896 ] Heng Chen commented on HBASE-16631: --- There is no logic changed, just split the AP and a little changes about function scope. > Extract AsyncRequestFuture related code from AsyncProcess > - > > Key: HBASE-16631 > URL: https://issues.apache.org/jira/browse/HBASE-16631 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen >Assignee: Heng Chen > Attachments: HBASE-16631.v1.patch, HBASE-16631.wip.patch > > > Now, AsyncProcess class is too large (over 2000+ lines), and there are so > many sub classes in it. > AsyncRequestFutureImpl is the biggest subclass in AP, we could extract it > out from AP to reduce the AP size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16634) Speedup TestExportSnapshot
[ https://issues.apache.org/jira/browse/HBASE-16634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16634: Status: Open (was: Patch Available) > Speedup TestExportSnapshot > -- > > Key: HBASE-16634 > URL: https://issues.apache.org/jira/browse/HBASE-16634 > Project: HBase > Issue Type: Test > Components: snapshots, test >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > > TestExportSnapshot is a long and heavy test since has to take snapshot, > export (via MR), restore them and so on. this in general make the test flaky > due to slowness. > let's try to speed it up a bit by reducing the number of regions of the table > we are testing on 9 regions at the moment we can reduce it at least to 2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16634) Speedup TestExportSnapshot
[ https://issues.apache.org/jira/browse/HBASE-16634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16634: Attachment: (was: HBASE-16634-v0.patch) > Speedup TestExportSnapshot > -- > > Key: HBASE-16634 > URL: https://issues.apache.org/jira/browse/HBASE-16634 > Project: HBase > Issue Type: Test > Components: snapshots, test >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > > TestExportSnapshot is a long and heavy test since has to take snapshot, > export (via MR), restore them and so on. this in general make the test flaky > due to slowness. > let's try to speed it up a bit by reducing the number of regions of the table > we are testing on 9 regions at the moment we can reduce it at least to 2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16632) Optimize HBase RPC Encryption Performance
[
https://issues.apache.org/jira/browse/HBASE-16632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Latham resolved HBASE-16632.
-
Resolution: Duplicate
> Optimize HBase RPC Encryption Performance
> -
>
> Key: HBASE-16632
> URL: https://issues.apache.org/jira/browse/HBASE-16632
> Project: HBase
> Issue Type: Bug
> Components: security
>Reporter: Robert Yokota
>
> Similar to HADOOP-10768, when HBase RPC encryption is enabled by setting
> {{hbase.rpc.protection}} to {{privacy}}, it does not use AES-NI acceleration
> by default.
> We have implemented a workaround for HBase to use accelerated encryption that
> can be shown to improve performance, see
> https://rayokota.wordpress.com/2016/09/13/adventures-in-hardening-hbase/. I
> will attach a patch of the changes for our workaround, which borrows heavily
> from the patch for HADOOP-10768, but I'll leave it to the HBase security
> experts as to what is the best approach.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16381) Shell deleteall command should support row key prefixes
[
https://issues.apache.org/jira/browse/HBASE-16381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491855#comment-15491855
]
Hadoop QA commented on HBASE-16381:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 0s
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 0s
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 8s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 56s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 13s
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 36s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12828542/HBASE-16381-V6.patch |
| JIRA Issue | HBASE-16381 |
| Optional Tests | asflicense javac javadoc unit rubocop ruby_lint |
| uname | Linux a51f816f7f60 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 8ef6c76 |
| Default Java | 1.8.0_101 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3552/testReport/ |
| modules | C: hbase-shell U: hbase-shell |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3552/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Shell deleteall command should support row key prefixes
> ---
>
> Key: HBASE-16381
> URL: https://issues.apache.org/jira/browse/HBASE-16381
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Reporter: Andrew Purtell
>Assignee: Yi Liang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16381-V1.patch, HBASE-16381-V2.patch,
> HBASE-16381-V3.patch, HBASE-16381-V4.patch, HBASE-16381-V5.patch,
> HBASE-16381-V6.patch
>
>
> The shell's deleteall command should support deleting a row range using a row
> key prefix.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16613) Return the unused ByteBuffer to BoundedByteBufferPool when no cell is retrieved from the CellScanner
[
https://issues.apache.org/jira/browse/HBASE-16613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491821#comment-15491821
]
Hudson commented on HBASE-16613:
FAILURE: Integrated in Jenkins build HBase-1.3-JDK7 #16 (See
[https://builds.apache.org/job/HBase-1.3-JDK7/16/])
HBASE-16613 Return the unused ByteBuffer to BoundedByteBufferPool when (tedyu:
rev 55d3dc415c304cabe6ed140380ec69ce08abab63)
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java
> Return the unused ByteBuffer to BoundedByteBufferPool when no cell is
> retrieved from the CellScanner
>
>
> Key: HBASE-16613
> URL: https://issues.apache.org/jira/browse/HBASE-16613
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 1.3.0, 1.4.0, 1.2.4
>
> Attachments: HBASE-16613.branch-1.v0.patch,
> HBASE-16613.branch-1.v1.patch, HBASE-16613.branch-1.v1.patch
>
>
> The critical code is shown below:
> {code:title=IPCUtil.java|borderStyle=solid}
> // We should put the ByteBuffer into pool before return null
> public ByteBuffer buildCellBlock(final Codec codec, final CompressionCodec
> compressor,
> final CellScanner cellScanner, final BoundedByteBufferPool pool) {
> ...
> if (pool != null) {
> ByteBuffer bb = pool.getBuffer();
> bufferSize = bb.capacity();
> baos = new ByteBufferOutputStream(bb);
> }
> ...
> int count = 0;
> while (cellScanner.advance()) {
> encoder.write(cellScanner.current());
> count++;
> }
> encoder.flush();
> if (count == 0) return null;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16634) Speedup TestExportSnapshot
[ https://issues.apache.org/jira/browse/HBASE-16634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16634: Status: Patch Available (was: Open) > Speedup TestExportSnapshot > -- > > Key: HBASE-16634 > URL: https://issues.apache.org/jira/browse/HBASE-16634 > Project: HBase > Issue Type: Test > Components: snapshots, test >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16634-v0.patch > > > TestExportSnapshot is a long and heavy test since has to take snapshot, > export (via MR), restore them and so on. this in general make the test flaky > due to slowness. > let's try to speed it up a bit by reducing the number of regions of the table > we are testing on 9 regions at the moment we can reduce it at least to 2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16634) Speedup TestExportSnapshot
[ https://issues.apache.org/jira/browse/HBASE-16634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16634: Attachment: HBASE-16634-v0.patch > Speedup TestExportSnapshot > -- > > Key: HBASE-16634 > URL: https://issues.apache.org/jira/browse/HBASE-16634 > Project: HBase > Issue Type: Test > Components: snapshots, test >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16634-v0.patch > > > TestExportSnapshot is a long and heavy test since has to take snapshot, > export (via MR), restore them and so on. this in general make the test flaky > due to slowness. > let's try to speed it up a bit by reducing the number of regions of the table > we are testing on 9 regions at the moment we can reduce it at least to 2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16634) Speedup TestExportSnapshot
Matteo Bertozzi created HBASE-16634: --- Summary: Speedup TestExportSnapshot Key: HBASE-16634 URL: https://issues.apache.org/jira/browse/HBASE-16634 Project: HBase Issue Type: Test Components: snapshots, test Affects Versions: 2.0.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 2.0.0, 1.4.0 Attachments: HBASE-16634-v0.patch TestExportSnapshot is a long and heavy test since has to take snapshot, export (via MR), restore them and so on. this in general make the test flaky due to slowness. let's try to speed it up a bit by reducing the number of regions of the table we are testing on 9 regions at the moment we can reduce it at least to 2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491782#comment-15491782 ] Ted Yu commented on HBASE-14123: None of the failed tests were backup / restore related. > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, > 14123-master.v21.txt, 14123-master.v3.txt, 14123-master.v5.txt, > 14123-master.v6.txt, 14123-master.v7.txt, 14123-master.v8.txt, > 14123-master.v9.txt, 14123-v14.txt, HBASE-14123-for-7912-v1.patch, > HBASE-14123-for-7912-v6.patch, HBASE-14123-v1.patch, HBASE-14123-v10.patch, > HBASE-14123-v11.patch, HBASE-14123-v12.patch, HBASE-14123-v13.patch, > HBASE-14123-v15.patch, HBASE-14123-v16.patch, HBASE-14123-v2.patch, > HBASE-14123-v3.patch, HBASE-14123-v4.patch, HBASE-14123-v5.patch, > HBASE-14123-v6.patch, HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16381) Shell deleteall command should support row key prefixes
[
https://issues.apache.org/jira/browse/HBASE-16381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liang updated HBASE-16381:
-
Attachment: HBASE-16381-V6.patch
Carry Chen Heng's comments
{code}
hbase> deleteall 't1', {ROWPREFIXFILTER => 'prefix', CACHE => 100}
{code}
CACHE is optional, and is used to specify how many deletes batched to be sent
to server at one time, default is 100
> Shell deleteall command should support row key prefixes
> ---
>
> Key: HBASE-16381
> URL: https://issues.apache.org/jira/browse/HBASE-16381
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Reporter: Andrew Purtell
>Assignee: Yi Liang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16381-V1.patch, HBASE-16381-V2.patch,
> HBASE-16381-V3.patch, HBASE-16381-V4.patch, HBASE-16381-V5.patch,
> HBASE-16381-V6.patch
>
>
> The shell's deleteall command should support deleting a row range using a row
> key prefix.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491749#comment-15491749
]
Hadoop QA commented on HBASE-14123:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 5s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 47 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 5s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 14m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
58s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 32s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 46s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 14m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
5s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 590 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 18s
{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 25s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 55s
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 17s
{color} | {color:red} hbase-client generated 4 new + 14 unchanged - 0 fixed =
18 total (was 14) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 38s
{color} | {color:red} root generated 4 new + 20 unchanged - 0 fixed = 24 total
(was 20) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} |
[jira] [Commented] (HBASE-16633) Optimize HBase RPC Encryption Performance
[
https://issues.apache.org/jira/browse/HBASE-16633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491706#comment-15491706
]
Robert Yokota commented on HBASE-16633:
---
I've attached a patch. Most of the code is from HADOOP-10768. The patch does
not account for {{AsyncHBaseSaslRpcClientHandler}}
> Optimize HBase RPC Encryption Performance
> -
>
> Key: HBASE-16633
> URL: https://issues.apache.org/jira/browse/HBASE-16633
> Project: HBase
> Issue Type: Bug
> Components: security
>Reporter: Robert Yokota
> Attachments: HBASE-16633.master.001.patch
>
>
> Similar to HADOOP-10768, when HBase RPC encryption is enabled by setting
> {{hbase.rpc.protection}} to {{privacy}}, it does not use AES-NI acceleration
> by default.
> We have implemented a workaround for HBase to use accelerated encryption that
> can be shown to improve performance, see
> https://rayokota.wordpress.com/2016/09/13/adventures-in-hardening-hbase/. I
> will attach a patch of the changes for our workaround, which borrows heavily
> from the patch for HADOOP-10768, but I'll leave it to the HBase security
> experts as to what is the best approach.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16633) Optimize HBase RPC Encryption Performance
[
https://issues.apache.org/jira/browse/HBASE-16633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Yokota updated HBASE-16633:
--
Attachment: HBASE-16633.master.001.patch
> Optimize HBase RPC Encryption Performance
> -
>
> Key: HBASE-16633
> URL: https://issues.apache.org/jira/browse/HBASE-16633
> Project: HBase
> Issue Type: Bug
> Components: security
>Reporter: Robert Yokota
> Attachments: HBASE-16633.master.001.patch
>
>
> Similar to HADOOP-10768, when HBase RPC encryption is enabled by setting
> {{hbase.rpc.protection}} to {{privacy}}, it does not use AES-NI acceleration
> by default.
> We have implemented a workaround for HBase to use accelerated encryption that
> can be shown to improve performance, see
> https://rayokota.wordpress.com/2016/09/13/adventures-in-hardening-hbase/. I
> will attach a patch of the changes for our workaround, which borrows heavily
> from the patch for HADOOP-10768, but I'll leave it to the HBase security
> experts as to what is the best approach.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16633) Optimize HBase RPC Encryption Performance
Robert Yokota created HBASE-16633:
-
Summary: Optimize HBase RPC Encryption Performance
Key: HBASE-16633
URL: https://issues.apache.org/jira/browse/HBASE-16633
Project: HBase
Issue Type: Bug
Components: security
Reporter: Robert Yokota
Similar to HADOOP-10768, when HBase RPC encryption is enabled by setting
{{hbase.rpc.protection}} to {{privacy}}, it does not use AES-NI acceleration by
default.
We have implemented a workaround for HBase to use accelerated encryption that
can be shown to improve performance, see
https://rayokota.wordpress.com/2016/09/13/adventures-in-hardening-hbase/. I
will attach a patch of the changes for our workaround, which borrows heavily
from the patch for HADOOP-10768, but I'll leave it to the HBase security
experts as to what is the best approach.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16632) Optimize HBase RPC Encryption Performance
Robert Yokota created HBASE-16632:
-
Summary: Optimize HBase RPC Encryption Performance
Key: HBASE-16632
URL: https://issues.apache.org/jira/browse/HBASE-16632
Project: HBase
Issue Type: Bug
Components: security
Reporter: Robert Yokota
Similar to HADOOP-10768, when HBase RPC encryption is enabled by setting
{{hbase.rpc.protection}} to {{privacy}}, it does not use AES-NI acceleration by
default.
We have implemented a workaround for HBase to use accelerated encryption that
can be shown to improve performance, see
https://rayokota.wordpress.com/2016/09/13/adventures-in-hardening-hbase/. I
will attach a patch of the changes for our workaround, which borrows heavily
from the patch for HADOOP-10768, but I'll leave it to the HBase security
experts as to what is the best approach.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16388) Prevent client threads being blocked by only one slow region server
[ https://issues.apache.org/jira/browse/HBASE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491629#comment-15491629 ] Hudson commented on HBASE-16388: FAILURE: Integrated in Jenkins build HBase-1.4 #416 (See [https://builds.apache.org/job/HBase-1.4/416/]) HBASE-16388 Prevent client threads being blocked by only one slow region (stack: rev 069d1f73fa7e1a2b4c21ba95dea867d077a51068) * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/ServerTooBusyException.java * (edit) hbase-common/src/main/resources/hbase-default.xml * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AbstractRpcClient.java > Prevent client threads being blocked by only one slow region server > --- > > Key: HBASE-16388 > URL: https://issues.apache.org/jira/browse/HBASE-16388 > Project: HBase > Issue Type: New Feature >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16388-branch-1-v1.patch, > HBASE-16388-branch-1-v2.patch, HBASE-16388-v1.patch, HBASE-16388-v2.patch, > HBASE-16388-v2.patch, HBASE-16388-v2.patch, HBASE-16388-v2.patch, > HBASE-16388-v3.patch > > > It is a general use case for HBase's users that they have several > threads/handlers in their service, and each handler has its own Table/HTable > instance. Generally users think each handler is independent and won't > interact each other. > However, in an extreme case, if a region server is very slow, every requests > to this RS will timeout, handlers of users' service may be occupied by the > long-waiting requests even requests belong to other RS will also be timeout. > For example: > If we have 100 handlers in a client service(timeout is 1000ms) and HBase has > 10 region servers whose average response time is 50ms. If no region server is > slow, we can handle 2000 requests per second. > Now this service's QPS is 1000. If there is one region server very slow and > all requests to it will be timeout. Users hope that only 10% requests failed, > and 90% requests' response time is still 50ms, because only 10% requests are > located to the slow RS. However, each second we have 100 long-waiting > requests which exactly occupies all 100 handles. So all handlers is blocked, > the availability of this service is almost zero. > To prevent this case, we can limit the max concurrent requests to one RS in > process-level. Requests exceeding the limit will throws > ServerBusyException(extends DoNotRetryIOE) immediately to users. In the above > case, if we set this limit to 20, only 20 handlers will be occupied and other > 80 handlers can still handle requests to other RS. The availability of this > service is 90% as expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14328) WebUI error 500 table doesn't exist
[ https://issues.apache.org/jira/browse/HBASE-14328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Liptak updated HBASE-14328: - Status: Open (was: Patch Available) I'm not working this at this time. Thanks > WebUI error 500 table doesn't exist > --- > > Key: HBASE-14328 > URL: https://issues.apache.org/jira/browse/HBASE-14328 > Project: HBase > Issue Type: Bug > Components: Interface >Affects Versions: 1.1.0.1 >Reporter: Jean-Marc Spaggiari >Priority: Minor > Labels: beginner > Attachments: HBASE-14328.1.patch > > > When trying to load a non-existing table using the WebUI we get an ugly error > 500. Instead, we should display a nice "The table you are trying to reach > doesn't exit" message... > http://server.com:60010/table.jsp?name=TestTable > HTTP ERROR 500 > Problem accessing /table.jsp. Reason: > TestTable > Caused by: > org.apache.hadoop.hbase.TableNotFoundException: TestTable > at > org.apache.hadoop.hbase.client.HTable.getTableDescriptor(HTable.java:651) > at > org.apache.hadoop.hbase.generated.master.table_jsp._jspService(table_jsp.java:74) > at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:98) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > at > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) > at > org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:113) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1351) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.hbase.http.NoCacheFilter.doFilter(NoCacheFilter.java:49) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.hbase.http.NoCacheFilter.doFilter(NoCacheFilter.java:49) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) > at > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > at > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767) > at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) > at > org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) > at > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > at org.mortbay.jetty.Server.handle(Server.java:326) > at > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > at > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > at > org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) > at > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > Powered by Jetty:// -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14328) WebUI error 500 table doesn't exist
[ https://issues.apache.org/jira/browse/HBASE-14328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Liptak updated HBASE-14328: - Assignee: (was: Gabor Liptak) > WebUI error 500 table doesn't exist > --- > > Key: HBASE-14328 > URL: https://issues.apache.org/jira/browse/HBASE-14328 > Project: HBase > Issue Type: Bug > Components: Interface >Affects Versions: 1.1.0.1 >Reporter: Jean-Marc Spaggiari >Priority: Minor > Labels: beginner > Attachments: HBASE-14328.1.patch > > > When trying to load a non-existing table using the WebUI we get an ugly error > 500. Instead, we should display a nice "The table you are trying to reach > doesn't exit" message... > http://server.com:60010/table.jsp?name=TestTable > HTTP ERROR 500 > Problem accessing /table.jsp. Reason: > TestTable > Caused by: > org.apache.hadoop.hbase.TableNotFoundException: TestTable > at > org.apache.hadoop.hbase.client.HTable.getTableDescriptor(HTable.java:651) > at > org.apache.hadoop.hbase.generated.master.table_jsp._jspService(table_jsp.java:74) > at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:98) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > at > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) > at > org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:113) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1351) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.hbase.http.NoCacheFilter.doFilter(NoCacheFilter.java:49) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.hbase.http.NoCacheFilter.doFilter(NoCacheFilter.java:49) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) > at > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > at > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767) > at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) > at > org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) > at > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > at org.mortbay.jetty.Server.handle(Server.java:326) > at > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > at > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > at > org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) > at > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > Powered by Jetty:// -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-13879) Add hbase.hstore.compactionThreshold to HConstants
[ https://issues.apache.org/jira/browse/HBASE-13879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Liptak resolved HBASE-13879. -- Resolution: Won't Fix > Add hbase.hstore.compactionThreshold to HConstants > -- > > Key: HBASE-13879 > URL: https://issues.apache.org/jira/browse/HBASE-13879 > Project: HBase > Issue Type: Improvement >Reporter: Gabor Liptak >Assignee: Gabor Liptak >Priority: Minor > Attachments: HBASE-13879.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16388) Prevent client threads being blocked by only one slow region server
[ https://issues.apache.org/jira/browse/HBASE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491289#comment-15491289 ] Hudson commented on HBASE-16388: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1601 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1601/]) HBASE-16388 Prevent client threads being blocked by only one slow region (stack: rev 8ef6c76344127f2f4d2f9536d87fa6fc7b5c7132) * (edit) hbase-common/src/main/resources/hbase-default.xml * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/ServerTooBusyException.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AbstractRpcClient.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java > Prevent client threads being blocked by only one slow region server > --- > > Key: HBASE-16388 > URL: https://issues.apache.org/jira/browse/HBASE-16388 > Project: HBase > Issue Type: New Feature >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16388-branch-1-v1.patch, > HBASE-16388-branch-1-v2.patch, HBASE-16388-v1.patch, HBASE-16388-v2.patch, > HBASE-16388-v2.patch, HBASE-16388-v2.patch, HBASE-16388-v2.patch, > HBASE-16388-v3.patch > > > It is a general use case for HBase's users that they have several > threads/handlers in their service, and each handler has its own Table/HTable > instance. Generally users think each handler is independent and won't > interact each other. > However, in an extreme case, if a region server is very slow, every requests > to this RS will timeout, handlers of users' service may be occupied by the > long-waiting requests even requests belong to other RS will also be timeout. > For example: > If we have 100 handlers in a client service(timeout is 1000ms) and HBase has > 10 region servers whose average response time is 50ms. If no region server is > slow, we can handle 2000 requests per second. > Now this service's QPS is 1000. If there is one region server very slow and > all requests to it will be timeout. Users hope that only 10% requests failed, > and 90% requests' response time is still 50ms, because only 10% requests are > located to the slow RS. However, each second we have 100 long-waiting > requests which exactly occupies all 100 handles. So all handlers is blocked, > the availability of this service is almost zero. > To prevent this case, we can limit the max concurrent requests to one RS in > process-level. Requests exceeding the limit will throws > ServerBusyException(extends DoNotRetryIOE) immediately to users. In the above > case, if we set this limit to 20, only 20 handlers will be occupied and other > 80 handlers can still handle requests to other RS. The availability of this > service is 90% as expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16388) Prevent client threads being blocked by only one slow region server
[ https://issues.apache.org/jira/browse/HBASE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16388: -- Release Note: Add a new configuration, hbase.client.perserver.requests.threshold, to limit the max number of concurrent request to one region server. If the user still create new request after reaching the limit, client will throw ServerTooBusyException and do not send the request to the server. This is a client side feature and can prevent client's threads being blocked by one slow region server resulting in the availability of client is much lower than the availability of region servers. For completeness, here extract on new config from hbase-default.xml: Property: hbase.client.perserver.requests.threshold Default: 2147483647 Description: The max number of concurrent pending requests for one server in all client threads (process level). Exceeding requests will be thrown ServerTooBusyException immediately to prevent user's threads being occupied and blocked by only one slow region server. If you use a fix number of threads to access HBase in a synchronous way, set this to a suitable value which is related to the number of threads will help you. See https://issues.apache.org/jira/browse/HBASE-16388 for details. was:Add a new configuration, hbase.client.perserver.requests.threshold, to limit the max number of concurrent request to one region server. If the user still create new request after reaching the limit, client will throw ServerBusyException and do not send the request to the server. This is a client side feature and can prevent client's threads being blocked by one slow region server resulting in the availability of client is much lower than the availability of region servers. Added the hbase-default.xml extract to the release note (after [~jerryhe]'s questions above) > Prevent client threads being blocked by only one slow region server > --- > > Key: HBASE-16388 > URL: https://issues.apache.org/jira/browse/HBASE-16388 > Project: HBase > Issue Type: New Feature >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16388-branch-1-v1.patch, > HBASE-16388-branch-1-v2.patch, HBASE-16388-v1.patch, HBASE-16388-v2.patch, > HBASE-16388-v2.patch, HBASE-16388-v2.patch, HBASE-16388-v2.patch, > HBASE-16388-v3.patch > > > It is a general use case for HBase's users that they have several > threads/handlers in their service, and each handler has its own Table/HTable > instance. Generally users think each handler is independent and won't > interact each other. > However, in an extreme case, if a region server is very slow, every requests > to this RS will timeout, handlers of users' service may be occupied by the > long-waiting requests even requests belong to other RS will also be timeout. > For example: > If we have 100 handlers in a client service(timeout is 1000ms) and HBase has > 10 region servers whose average response time is 50ms. If no region server is > slow, we can handle 2000 requests per second. > Now this service's QPS is 1000. If there is one region server very slow and > all requests to it will be timeout. Users hope that only 10% requests failed, > and 90% requests' response time is still 50ms, because only 10% requests are > located to the slow RS. However, each second we have 100 long-waiting > requests which exactly occupies all 100 handles. So all handlers is blocked, > the availability of this service is almost zero. > To prevent this case, we can limit the max concurrent requests to one RS in > process-level. Requests exceeding the limit will throws > ServerBusyException(extends DoNotRetryIOE) immediately to users. In the above > case, if we set this limit to 20, only 20 handlers will be occupied and other > 80 handlers can still handle requests to other RS. The availability of this > service is 90% as expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16388) Prevent client threads being blocked by only one slow region server
[ https://issues.apache.org/jira/browse/HBASE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491232#comment-15491232 ] Phil Yang commented on HBASE-16388: --- Thanks for reviews and committing. I'm not familiar with the release strategy. I didn't make a patch for branch-1.3 because I thought we will not add new feature to 1.3. If we need this feature in 1.3 I can make a patch for 1.3 tomorrow(UTC+8) > Prevent client threads being blocked by only one slow region server > --- > > Key: HBASE-16388 > URL: https://issues.apache.org/jira/browse/HBASE-16388 > Project: HBase > Issue Type: New Feature >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16388-branch-1-v1.patch, > HBASE-16388-branch-1-v2.patch, HBASE-16388-v1.patch, HBASE-16388-v2.patch, > HBASE-16388-v2.patch, HBASE-16388-v2.patch, HBASE-16388-v2.patch, > HBASE-16388-v3.patch > > > It is a general use case for HBase's users that they have several > threads/handlers in their service, and each handler has its own Table/HTable > instance. Generally users think each handler is independent and won't > interact each other. > However, in an extreme case, if a region server is very slow, every requests > to this RS will timeout, handlers of users' service may be occupied by the > long-waiting requests even requests belong to other RS will also be timeout. > For example: > If we have 100 handlers in a client service(timeout is 1000ms) and HBase has > 10 region servers whose average response time is 50ms. If no region server is > slow, we can handle 2000 requests per second. > Now this service's QPS is 1000. If there is one region server very slow and > all requests to it will be timeout. Users hope that only 10% requests failed, > and 90% requests' response time is still 50ms, because only 10% requests are > located to the slow RS. However, each second we have 100 long-waiting > requests which exactly occupies all 100 handles. So all handlers is blocked, > the availability of this service is almost zero. > To prevent this case, we can limit the max concurrent requests to one RS in > process-level. Requests exceeding the limit will throws > ServerBusyException(extends DoNotRetryIOE) immediately to users. In the above > case, if we set this limit to 20, only 20 handlers will be occupied and other > 80 handlers can still handle requests to other RS. The availability of this > service is 90% as expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16388) Prevent client threads being blocked by only one slow region server
[ https://issues.apache.org/jira/browse/HBASE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491225#comment-15491225 ] Phil Yang commented on HBASE-16388: --- As said in hbase-default, the limit is process-level, even user uses multiple HConnections, because the counter cache is a static object. > Prevent client threads being blocked by only one slow region server > --- > > Key: HBASE-16388 > URL: https://issues.apache.org/jira/browse/HBASE-16388 > Project: HBase > Issue Type: New Feature >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16388-branch-1-v1.patch, > HBASE-16388-branch-1-v2.patch, HBASE-16388-v1.patch, HBASE-16388-v2.patch, > HBASE-16388-v2.patch, HBASE-16388-v2.patch, HBASE-16388-v2.patch, > HBASE-16388-v3.patch > > > It is a general use case for HBase's users that they have several > threads/handlers in their service, and each handler has its own Table/HTable > instance. Generally users think each handler is independent and won't > interact each other. > However, in an extreme case, if a region server is very slow, every requests > to this RS will timeout, handlers of users' service may be occupied by the > long-waiting requests even requests belong to other RS will also be timeout. > For example: > If we have 100 handlers in a client service(timeout is 1000ms) and HBase has > 10 region servers whose average response time is 50ms. If no region server is > slow, we can handle 2000 requests per second. > Now this service's QPS is 1000. If there is one region server very slow and > all requests to it will be timeout. Users hope that only 10% requests failed, > and 90% requests' response time is still 50ms, because only 10% requests are > located to the slow RS. However, each second we have 100 long-waiting > requests which exactly occupies all 100 handles. So all handlers is blocked, > the availability of this service is almost zero. > To prevent this case, we can limit the max concurrent requests to one RS in > process-level. Requests exceeding the limit will throws > ServerBusyException(extends DoNotRetryIOE) immediately to users. In the above > case, if we set this limit to 20, only 20 handlers will be occupied and other > 80 handlers can still handle requests to other RS. The availability of this > service is 90% as expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16604) Scanner retries on IOException can cause the scans to miss data
[
https://issues.apache.org/jira/browse/HBASE-16604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-16604:
--
Attachment: hbase-16604_v3.patch
One more UT fix.
> Scanner retries on IOException can cause the scans to miss data
>
>
> Key: HBASE-16604
> URL: https://issues.apache.org/jira/browse/HBASE-16604
> Project: HBase
> Issue Type: Bug
> Components: regionserver, Scanners
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: hbase-16604_v1.patch, hbase-16604_v2.patch,
> hbase-16604_v3.patch
>
>
> Debugging an ITBLL failure, where the Verify did not "see" all the data in
> the cluster, I've noticed that if we end up getting a generic IOException
> from the HFileReader level, we may end up missing the rest of the data in the
> region. I was able to manually test this, and this stack trace helps to
> understand what is going on:
> {code}
> 2016-09-09 16:27:15,633 INFO [hconnection-0x71ad3d8a-shared--pool21-t9]
> client.ScannerCallable(376): Open scanner=1 for
> scan={"loadColumnFamiliesOnDemand":null,"startRow":"","stopRow":"","batch":-1,"cacheBlocks":true,"totalColumns":1,"maxResultSize":2097152,"families":{"testFamily":["testFamily"]},"caching":100,"maxVersions":1,"timeRange":[0,9223372036854775807]}
> on region
> region=testScanThrowsException,,1473463632707.b2adfb618e5d0fe225c1dc40c0eabfee.,
> hostname=hw10676,51833,1473463626529, seqNum=2
> 2016-09-09 16:27:15,634 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2196): scan request:scanner_id: 1 number_of_rows:
> 100 close_scanner: false next_call_seq: 0 client_handles_partials: true
> client_handles_heartbeats: true renew: false
> 2016-09-09 16:27:15,635 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2510): Rolling back next call seqId
> 2016-09-09 16:27:15,635 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2565): Throwing new
> ServiceExceptionjava.io.IOException: Could not reseek
> StoreFileScanner[HFileScanner for reader
> reader=hdfs://localhost:51795/user/enis/test-data/d6fb1c70-93c1-4099-acb7-5723fc05a737/data/default/testScanThrowsException/b2adfb618e5d0fe225c1dc40c0eabfee/testFamily/5a213cc23b714e5e8e1a140ebbe72f2c,
> compression=none, cacheConf=blockCache=LruBlockCache{blockCount=0,
> currentSize=1567264, freeSize=1525578848, maxSize=1527146112,
> heapSize=1567264, minSize=1450788736, minFactor=0.95, multiSize=725394368,
> multiFactor=0.5, singleSize=362697184, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false, firstKey=aaa/testFamily:testFamily/1473463633859/Put,
> lastKey=zzz/testFamily:testFamily/1473463634271/Put, avgKeyLen=35,
> avgValueLen=3, entries=17576, length=866998,
> cur=/testFamily:/OLDEST_TIMESTAMP/Minimum/vlen=0/seqid=0] to key
> /testFamily:testFamily/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
> 2016-09-09 16:27:15,635 DEBUG
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833] ipc.CallRunner(110):
> B.fifo.QRpcServer.handler=5,queue=0,port=51833: callId: 26 service:
> ClientService methodName: Scan size: 26 connection: 192.168.42.75:51903
> java.io.IOException: Could not reseek StoreFileScanner[HFileScanner for
> reader
> reader=hdfs://localhost:51795/user/enis/test-data/d6fb1c70-93c1-4099-acb7-5723fc05a737/data/default/testScanThrowsException/b2adfb618e5d0fe225c1dc40c0eabfee/testFamily/5a213cc23b714e5e8e1a140ebbe72f2c,
> compression=none, cacheConf=blockCache=LruBlockCache{blockCount=0,
> currentSize=1567264, freeSize=1525578848, maxSize=1527146112,
> heapSize=1567264, minSize=1450788736, minFactor=0.95, multiSize=725394368,
> multiFactor=0.5, singleSize=362697184, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false, firstKey=aaa/testFamily:testFamily/1473463633859/Put,
> lastKey=zzz/testFamily:testFamily/1473463634271/Put, avgKeyLen=35,
> avgValueLen=3, entries=17576, length=866998,
> cur=/testFamily:/OLDEST_TIMESTAMP/Minimum/vlen=0/seqid=0] to key
> /testFamily:testFamily/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:224)
> at
> org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:55)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:312)
> at
>
[jira] [Updated] (HBASE-11368) Multi-column family BulkLoad fails if compactions go on too long
[ https://issues.apache.org/jira/browse/HBASE-11368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-11368: - Description: Compactions take a read lock. If a multi-column family region, before bulk loading, we want to take a write lock on the region. If the compaction takes too long, the bulk load fails. Various recipes include: + Making smaller regions (lame) + [~victorunique] suggests major compacting just before bulk loading over in HBASE-10882 as a work around. Does the compaction need a read lock for that long? Does the bulk load need a full write lock when multiple column families? Can we fail more gracefully at least? Note: Fixed in subtask HBASE-14575 was: Compactions take a read lock. If a multi-column family region, before bulk loading, we want to take a write lock on the region. If the compaction takes too long, the bulk load fails. Various recipes include: + Making smaller regions (lame) + [~victorunique] suggests major compacting just before bulk loading over in HBASE-10882 as a work around. Does the compaction need a read lock for that long? Does the bulk load need a full write lock when multiple column families? Can we fail more gracefully at least? > Multi-column family BulkLoad fails if compactions go on too long > > > Key: HBASE-11368 > URL: https://issues.apache.org/jira/browse/HBASE-11368 > Project: HBase > Issue Type: Bug >Reporter: stack > Attachments: hbase-11368-0.98.5.patch, hbase11368-master.patch, > key_stacktrace_hbase10882.TXT, performance_improvement_verification_98.5.patch > > > Compactions take a read lock. If a multi-column family region, before bulk > loading, we want to take a write lock on the region. If the compaction takes > too long, the bulk load fails. > Various recipes include: > + Making smaller regions (lame) > + [~victorunique] suggests major compacting just before bulk loading over in > HBASE-10882 as a work around. > Does the compaction need a read lock for that long? Does the bulk load need > a full write lock when multiple column families? Can we fail more gracefully > at least? > Note: Fixed in subtask HBASE-14575 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14575) Relax region read lock for compactions
[ https://issues.apache.org/jira/browse/HBASE-14575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-14575: - Description: Per [~devaraj]'s idea on parent issue, let's see if we can reduce the scope of critical section under which compactions hold the region read lock. Here is summary from parent issue: Another idea is we can reduce the scope of when the read lock is held during compaction. In theory the compactor only needs a region read lock while deciding what files to compact and at the time of committing the compaction. We're protected from the case of region close events because compactions are checking (every 10k bytes written) if the store has been closed in order to abort in such a case. was: Per [~devaraj]'s idea on parent issue, let's see if we can reduce the scope of critical section under which compactions hold the region read lock. Here is summary from parent issue: Another idea is we can reduce the scope of when the read lock is held during compaction. In theory the compactor only needs a region read lock while deciding what files to compact and at the time of committing the compaction. We're protected from the case of region close events because compactions are checking (every 10k bytes written) if the store has been closed in order to abort in such a case. Note: Fixed in subtask HBASE-14575 > Relax region read lock for compactions > -- > > Key: HBASE-14575 > URL: https://issues.apache.org/jira/browse/HBASE-14575 > Project: HBase > Issue Type: Sub-task > Components: Compaction, regionserver >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 2.0.0, 1.3.0 > > Attachments: 14575-test-case.patch, 14575-v1.patch, 14575-v2.patch, > 14575-v3.patch, 14575-v4.patch, 14575-v5.patch, 14575-v6.txt, 14575-v6.txt, > 14575-v6.txt, 14575-v7.txt, 14575.v00.patch > > > Per [~devaraj]'s idea on parent issue, let's see if we can reduce the scope > of critical section under which compactions hold the region read lock. > Here is summary from parent issue: > Another idea is we can reduce the scope of when the read lock is held during > compaction. In theory the compactor only needs a region read lock while > deciding what files to compact and at the time of committing the compaction. > We're protected from the case of region close events because compactions are > checking (every 10k bytes written) if the store has been closed in order to > abort in such a case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14575) Relax region read lock for compactions
[ https://issues.apache.org/jira/browse/HBASE-14575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-14575: - Description: Per [~devaraj]'s idea on parent issue, let's see if we can reduce the scope of critical section under which compactions hold the region read lock. Here is summary from parent issue: Another idea is we can reduce the scope of when the read lock is held during compaction. In theory the compactor only needs a region read lock while deciding what files to compact and at the time of committing the compaction. We're protected from the case of region close events because compactions are checking (every 10k bytes written) if the store has been closed in order to abort in such a case. Note: Fixed in subtask HBASE-14575 was: Per [~devaraj]'s idea on parent issue, let's see if we can reduce the scope of critical section under which compactions hold the region read lock. Here is summary from parent issue: Another idea is we can reduce the scope of when the read lock is held during compaction. In theory the compactor only needs a region read lock while deciding what files to compact and at the time of committing the compaction. We're protected from the case of region close events because compactions are checking (every 10k bytes written) if the store has been closed in order to abort in such a case. > Relax region read lock for compactions > -- > > Key: HBASE-14575 > URL: https://issues.apache.org/jira/browse/HBASE-14575 > Project: HBase > Issue Type: Sub-task > Components: Compaction, regionserver >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 2.0.0, 1.3.0 > > Attachments: 14575-test-case.patch, 14575-v1.patch, 14575-v2.patch, > 14575-v3.patch, 14575-v4.patch, 14575-v5.patch, 14575-v6.txt, 14575-v6.txt, > 14575-v6.txt, 14575-v7.txt, 14575.v00.patch > > > Per [~devaraj]'s idea on parent issue, let's see if we can reduce the scope > of critical section under which compactions hold the region read lock. > Here is summary from parent issue: > Another idea is we can reduce the scope of when the read lock is held during > compaction. In theory the compactor only needs a region read lock while > deciding what files to compact and at the time of committing the compaction. > We're protected from the case of region close events because compactions are > checking (every 10k bytes written) if the store has been closed in order to > abort in such a case. > Note: Fixed in subtask HBASE-14575 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11368) Multi-column family BulkLoad fails if compactions go on too long
[ https://issues.apache.org/jira/browse/HBASE-11368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-11368: - Resolution: Fixed Assignee: (was: Qiang Tian) Status: Resolved (was: Patch Available) > Multi-column family BulkLoad fails if compactions go on too long > > > Key: HBASE-11368 > URL: https://issues.apache.org/jira/browse/HBASE-11368 > Project: HBase > Issue Type: Bug >Reporter: stack > Attachments: hbase-11368-0.98.5.patch, hbase11368-master.patch, > key_stacktrace_hbase10882.TXT, performance_improvement_verification_98.5.patch > > > Compactions take a read lock. If a multi-column family region, before bulk > loading, we want to take a write lock on the region. If the compaction takes > too long, the bulk load fails. > Various recipes include: > + Making smaller regions (lame) > + [~victorunique] suggests major compacting just before bulk loading over in > HBASE-10882 as a work around. > Does the compaction need a read lock for that long? Does the bulk load need > a full write lock when multiple column families? Can we fail more gracefully > at least? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15448) HBase Backup Phase 3: Restore optimization 2
[
https://issues.apache.org/jira/browse/HBASE-15448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491181#comment-15491181
]
Hadoop QA commented on HBASE-15448:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} HBASE-15448 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12828508/HBASE-15448-v4.patch |
| JIRA Issue | HBASE-15448 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3550/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> HBase Backup Phase 3: Restore optimization 2
>
>
> Key: HBASE-15448
> URL: https://issues.apache.org/jira/browse/HBASE-15448
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-15448-v1.patch, HBASE-15448-v2.patch,
> HBASE-15448-v4.patch
>
>
> JIRA opened to continue work on restore optimization.
> This will focus on the following
> # During incremental backup image restore - restoring full backup into region
> boundaries of the most recent incremental backup image.
> # Combining multiple tables into single M/R job
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15448) HBase Backup Phase 3: Restore optimization 2
[ https://issues.apache.org/jira/browse/HBASE-15448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-15448: -- Attachment: HBASE-15448-v4.patch > HBase Backup Phase 3: Restore optimization 2 > > > Key: HBASE-15448 > URL: https://issues.apache.org/jira/browse/HBASE-15448 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: HBASE-15448-v1.patch, HBASE-15448-v2.patch, > HBASE-15448-v4.patch > > > JIRA opened to continue work on restore optimization. > This will focus on the following > # During incremental backup image restore - restoring full backup into region > boundaries of the most recent incremental backup image. > # Combining multiple tables into single M/R job -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16388) Prevent client threads being blocked by only one slow region server
[ https://issues.apache.org/jira/browse/HBASE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491167#comment-15491167 ] Jerry He commented on HBASE-16388: -- Good feature! Just need the release notes and the hbase-default.xml to be more clear. The threshold limit is per Connection/HConnection. This is to compare to a situation where multi-threaded client opens multiple HConnections, > Prevent client threads being blocked by only one slow region server > --- > > Key: HBASE-16388 > URL: https://issues.apache.org/jira/browse/HBASE-16388 > Project: HBase > Issue Type: New Feature >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16388-branch-1-v1.patch, > HBASE-16388-branch-1-v2.patch, HBASE-16388-v1.patch, HBASE-16388-v2.patch, > HBASE-16388-v2.patch, HBASE-16388-v2.patch, HBASE-16388-v2.patch, > HBASE-16388-v3.patch > > > It is a general use case for HBase's users that they have several > threads/handlers in their service, and each handler has its own Table/HTable > instance. Generally users think each handler is independent and won't > interact each other. > However, in an extreme case, if a region server is very slow, every requests > to this RS will timeout, handlers of users' service may be occupied by the > long-waiting requests even requests belong to other RS will also be timeout. > For example: > If we have 100 handlers in a client service(timeout is 1000ms) and HBase has > 10 region servers whose average response time is 50ms. If no region server is > slow, we can handle 2000 requests per second. > Now this service's QPS is 1000. If there is one region server very slow and > all requests to it will be timeout. Users hope that only 10% requests failed, > and 90% requests' response time is still 50ms, because only 10% requests are > located to the slow RS. However, each second we have 100 long-waiting > requests which exactly occupies all 100 handles. So all handlers is blocked, > the availability of this service is almost zero. > To prevent this case, we can limit the max concurrent requests to one RS in > process-level. Requests exceeding the limit will throws > ServerBusyException(extends DoNotRetryIOE) immediately to users. In the above > case, if we set this limit to 20, only 20 handlers will be occupied and other > 80 handlers can still handle requests to other RS. The availability of this > service is 90% as expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16620) Fix backup command-line tool usability issues
[ https://issues.apache.org/jira/browse/HBASE-16620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491158#comment-15491158 ] Vladimir Rodionov commented on HBASE-16620: --- I added additional unit test for command helps, will add more tests today as an addendum to this patch. > Fix backup command-line tool usability issues > - > > Key: HBASE-16620 > URL: https://issues.apache.org/jira/browse/HBASE-16620 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Attachments: 16620-v2.patch, 16620-v4.patch, 16620.addendum, > HBASE-16620-v1.patch, HBASE-16620-v3.patch > > > We need to address issues found by [~saint@gmail.com] > https://issues.apache.org/jira/browse/HBASE-7912?focusedCommentId=15484865=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15484865 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16620) Fix backup command-line tool usability issues
[ https://issues.apache.org/jira/browse/HBASE-16620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491155#comment-15491155 ] Vladimir Rodionov commented on HBASE-16620: --- The command-line tool now: * Correctly print usage for every command ( -h, -help) * Does not throw exception in case of a wrong command format (only when command fails) > Fix backup command-line tool usability issues > - > > Key: HBASE-16620 > URL: https://issues.apache.org/jira/browse/HBASE-16620 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Attachments: 16620-v2.patch, 16620-v4.patch, 16620.addendum, > HBASE-16620-v1.patch, HBASE-16620-v3.patch > > > We need to address issues found by [~saint@gmail.com] > https://issues.apache.org/jira/browse/HBASE-7912?focusedCommentId=15484865=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15484865 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16620) Fix backup command-line tool usability issues
[
https://issues.apache.org/jira/browse/HBASE-16620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491108#comment-15491108
]
Ted Yu commented on HBASE-16620:
The following comments from HBASE-7912 have been addressed.
bq. Convert is unexplained as is silent
convert / silent have been dropped since they're obsolete.
bq. -w workers are threads or mapreduce tasks?
{code}
+ + " -w number of parallel workers (MapReduce tasks).\n"
{code}
bq. I'd guess it is the scheme that is disallowed but the complaint is that
'first' does not exist.
backup set 'first' should be first created using "hbase backup set" command.
{code}
kalashnikov:hbase.git.commit2 stack$ ./bin/hbase backup set
2016-09-12 10:02:27,178 ERROR [main] util.AbstractHBaseTool: Error running
command-line tool
{code}
{code}
kalashnikov:hbase.git.commit2 stack$ ./bin/hbase backup set add first
2016-09-12 10:04:07,068 ERROR [main] util.AbstractHBaseTool: Error running
command-line tool
java.lang.RuntimeException: Wrong args
{code}
If argument is empty, usage would be printed - for each (sub)command.
[~vrodionov]:
Can you add what I missed ?
[~stack]:
You're welcome to try from the tip of HBASE-7912 branch.
> Fix backup command-line tool usability issues
> -
>
> Key: HBASE-16620
> URL: https://issues.apache.org/jira/browse/HBASE-16620
> Project: HBase
> Issue Type: Bug
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Labels: backup
> Attachments: 16620-v2.patch, 16620-v4.patch, 16620.addendum,
> HBASE-16620-v1.patch, HBASE-16620-v3.patch
>
>
> We need to address issues found by [~saint@gmail.com]
> https://issues.apache.org/jira/browse/HBASE-7912?focusedCommentId=15484865=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15484865
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14123: --- Attachment: 14123-master.v21.txt Patch v21 is up to commit 7650f605905d25c7267259bf89cd236680c049f3 > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, > 14123-master.v21.txt, 14123-master.v3.txt, 14123-master.v5.txt, > 14123-master.v6.txt, 14123-master.v7.txt, 14123-master.v8.txt, > 14123-master.v9.txt, 14123-v14.txt, HBASE-14123-for-7912-v1.patch, > HBASE-14123-for-7912-v6.patch, HBASE-14123-v1.patch, HBASE-14123-v10.patch, > HBASE-14123-v11.patch, HBASE-14123-v12.patch, HBASE-14123-v13.patch, > HBASE-14123-v15.patch, HBASE-14123-v16.patch, HBASE-14123-v2.patch, > HBASE-14123-v3.patch, HBASE-14123-v4.patch, HBASE-14123-v5.patch, > HBASE-14123-v6.patch, HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16620) Fix backup command-line tool usability issues
[ https://issues.apache.org/jira/browse/HBASE-16620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16620: --- Attachment: 16620.addendum Addendum adds test category to TestBackupCommandLineTool > Fix backup command-line tool usability issues > - > > Key: HBASE-16620 > URL: https://issues.apache.org/jira/browse/HBASE-16620 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Attachments: 16620-v2.patch, 16620-v4.patch, 16620.addendum, > HBASE-16620-v1.patch, HBASE-16620-v3.patch > > > We need to address issues found by [~saint@gmail.com] > https://issues.apache.org/jira/browse/HBASE-7912?focusedCommentId=15484865=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15484865 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15490996#comment-15490996
]
Hadoop QA commented on HBASE-14123:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 5s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 47 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 59s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 14m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
55s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 31s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 45s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 14m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
4s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s
{color} | {color:red} The patch has 590 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 19s
{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 30s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 59s
{color} | {color:red} hbase-client generated 4 new + 0 unchanged - 0 fixed = 4
total (was 0) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 10s
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 17s
{color} | {color:red} hbase-client generated 4 new + 14 unchanged - 0 fixed =
18 total (was 14) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 55s
{color} | {color:red} root generated 4 new + 20 unchanged - 0 fixed = 24 total
(was 20) {color} |
|
[jira] [Commented] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess
[
https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15490978#comment-15490978
]
Hadoop QA commented on HBASE-16631:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 50s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 51s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 48s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 126m 11s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegion |
| Timed out junit tests |
org.apache.hadoop.hbase.security.access.TestAccessController2 |
| | org.apache.hadoop.hbase.mob.TestExpiredMobFileCleaner |
| | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
| | org.apache.hadoop.hbase.security.access.TestCellACLWithMultipleVersions |
| | org.apache.hadoop.hbase.io.encoding.TestEncodedSeekers |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12828476/HBASE-16631.v1.patch |
| JIRA Issue | HBASE-16631 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 394f37f9f964 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4c6a98b |
| Default Java |
[jira] [Commented] (HBASE-16613) Return the unused ByteBuffer to BoundedByteBufferPool when no cell is retrieved from the CellScanner
[
https://issues.apache.org/jira/browse/HBASE-16613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15490979#comment-15490979
]
stack commented on HBASE-16613:
---
Agree
> Return the unused ByteBuffer to BoundedByteBufferPool when no cell is
> retrieved from the CellScanner
>
>
> Key: HBASE-16613
> URL: https://issues.apache.org/jira/browse/HBASE-16613
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 1.3.0, 1.4.0, 1.2.4
>
> Attachments: HBASE-16613.branch-1.v0.patch,
> HBASE-16613.branch-1.v1.patch, HBASE-16613.branch-1.v1.patch
>
>
> The critical code is shown below:
> {code:title=IPCUtil.java|borderStyle=solid}
> // We should put the ByteBuffer into pool before return null
> public ByteBuffer buildCellBlock(final Codec codec, final CompressionCodec
> compressor,
> final CellScanner cellScanner, final BoundedByteBufferPool pool) {
> ...
> if (pool != null) {
> ByteBuffer bb = pool.getBuffer();
> bufferSize = bb.capacity();
> baos = new ByteBufferOutputStream(bb);
> }
> ...
> int count = 0;
> while (cellScanner.advance()) {
> encoder.write(cellScanner.current());
> count++;
> }
> encoder.flush();
> if (count == 0) return null;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16165) Decrease RpcServer.callQueueSize before writeResponse causes OOM
[
https://issues.apache.org/jira/browse/HBASE-16165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15490880#comment-15490880
]
stack commented on HBASE-16165:
---
Go for it [~Apache9] Bug fix.
> Decrease RpcServer.callQueueSize before writeResponse causes OOM
>
>
> Key: HBASE-16165
> URL: https://issues.apache.org/jira/browse/HBASE-16165
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC, rpc
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.6, 1.2.3, 0.98.22
>Reporter: Duo Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Labels: trivial
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 0.98.23, 1.2.4
>
> Attachments: HBASE-16165.patch
>
>
> In RpcServer, we use {{callQueueSizeInBytes}} to avoid queuing too many calls
> which causes OOM. But in {{CallRunner.run}}, we decrease it before send the
> response back. And even after calling {{sendResponseIfReady}}, the call
> object could stay in our heap for a long time if we can not write out the
> response(That's why we need a Responder thread...). This makes it possible
> that the actual size of all call object in heap is larger than
> {{maxQueueSizeInBytes}} and causes OOM.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16620) Fix backup command-line tool usability issues
[ https://issues.apache.org/jira/browse/HBASE-16620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15490872#comment-15490872 ] stack commented on HBASE-16620: --- What of the issues were addressed by this patch? There is no description. Thanks. Is the branch ready for another usage by dumb 'user'? > Fix backup command-line tool usability issues > - > > Key: HBASE-16620 > URL: https://issues.apache.org/jira/browse/HBASE-16620 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Attachments: 16620-v2.patch, 16620-v4.patch, HBASE-16620-v1.patch, > HBASE-16620-v3.patch > > > We need to address issues found by [~saint@gmail.com] > https://issues.apache.org/jira/browse/HBASE-7912?focusedCommentId=15484865=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15484865 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16388) Prevent client threads being blocked by only one slow region server
[ https://issues.apache.org/jira/browse/HBASE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16388: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.4.0 2.0.0 Status: Resolved (was: Patch Available) Pushed to branch-1 and master. Thanks for the nice patch [~yangzhe1991] I tried to backport to 1.3 but it wouldn't go back. > Prevent client threads being blocked by only one slow region server > --- > > Key: HBASE-16388 > URL: https://issues.apache.org/jira/browse/HBASE-16388 > Project: HBase > Issue Type: New Feature >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16388-branch-1-v1.patch, > HBASE-16388-branch-1-v2.patch, HBASE-16388-v1.patch, HBASE-16388-v2.patch, > HBASE-16388-v2.patch, HBASE-16388-v2.patch, HBASE-16388-v2.patch, > HBASE-16388-v3.patch > > > It is a general use case for HBase's users that they have several > threads/handlers in their service, and each handler has its own Table/HTable > instance. Generally users think each handler is independent and won't > interact each other. > However, in an extreme case, if a region server is very slow, every requests > to this RS will timeout, handlers of users' service may be occupied by the > long-waiting requests even requests belong to other RS will also be timeout. > For example: > If we have 100 handlers in a client service(timeout is 1000ms) and HBase has > 10 region servers whose average response time is 50ms. If no region server is > slow, we can handle 2000 requests per second. > Now this service's QPS is 1000. If there is one region server very slow and > all requests to it will be timeout. Users hope that only 10% requests failed, > and 90% requests' response time is still 50ms, because only 10% requests are > located to the slow RS. However, each second we have 100 long-waiting > requests which exactly occupies all 100 handles. So all handlers is blocked, > the availability of this service is almost zero. > To prevent this case, we can limit the max concurrent requests to one RS in > process-level. Requests exceeding the limit will throws > ServerBusyException(extends DoNotRetryIOE) immediately to users. In the above > case, if we set this limit to 20, only 20 handlers will be occupied and other > 80 handlers can still handle requests to other RS. The availability of this > service is 90% as expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16388) Prevent client threads being blocked by only one slow region server
[ https://issues.apache.org/jira/browse/HBASE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15490850#comment-15490850 ] stack commented on HBASE-16388: --- Findbugs issue is unrelated: CodeWarning RV Return value of java.util.concurrent.CountDownLatch.await(long, TimeUnit) ignored in org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper() Looks like attempt at fixing this elsewhere did not work. I tried the tests locally and they passed. Need to dig in on these tests on why they are failing. They seem unrelated to this patch. Going to commit. > Prevent client threads being blocked by only one slow region server > --- > > Key: HBASE-16388 > URL: https://issues.apache.org/jira/browse/HBASE-16388 > Project: HBase > Issue Type: New Feature >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-16388-branch-1-v1.patch, > HBASE-16388-branch-1-v2.patch, HBASE-16388-v1.patch, HBASE-16388-v2.patch, > HBASE-16388-v2.patch, HBASE-16388-v2.patch, HBASE-16388-v2.patch, > HBASE-16388-v3.patch > > > It is a general use case for HBase's users that they have several > threads/handlers in their service, and each handler has its own Table/HTable > instance. Generally users think each handler is independent and won't > interact each other. > However, in an extreme case, if a region server is very slow, every requests > to this RS will timeout, handlers of users' service may be occupied by the > long-waiting requests even requests belong to other RS will also be timeout. > For example: > If we have 100 handlers in a client service(timeout is 1000ms) and HBase has > 10 region servers whose average response time is 50ms. If no region server is > slow, we can handle 2000 requests per second. > Now this service's QPS is 1000. If there is one region server very slow and > all requests to it will be timeout. Users hope that only 10% requests failed, > and 90% requests' response time is still 50ms, because only 10% requests are > located to the slow RS. However, each second we have 100 long-waiting > requests which exactly occupies all 100 handles. So all handlers is blocked, > the availability of this service is almost zero. > To prevent this case, we can limit the max concurrent requests to one RS in > process-level. Requests exceeding the limit will throws > ServerBusyException(extends DoNotRetryIOE) immediately to users. In the above > case, if we set this limit to 20, only 20 handlers will be occupied and other > 80 handlers can still handle requests to other RS. The availability of this > service is 90% as expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15490827#comment-15490827 ] stack commented on HBASE-16631: --- Is this a straight extraction or did you make other changes? If so, patch LGTM. We'll probably want to change our Future to subclass CompletableFuture but that can come later. I like that you keep the implementations package private. > Extract AsyncRequestFuture related code from AsyncProcess > - > > Key: HBASE-16631 > URL: https://issues.apache.org/jira/browse/HBASE-16631 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen >Assignee: Heng Chen > Attachments: HBASE-16631.v1.patch, HBASE-16631.wip.patch > > > Now, AsyncProcess class is too large (over 2000+ lines), and there are so > many sub classes in it. > AsyncRequestFutureImpl is the biggest subclass in AP, we could extract it > out from AP to reduce the AP size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[
https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15490786#comment-15490786
]
ramkrishna.s.vasudevan commented on HBASE-16608:
Went thro the code.
{code}
try {
Action nextStep = policy();
switch (nextStep){
case FLATTEN: // Youngest Segment in the pipeline is with SkipList
index, make it flat
compactingMemStore.flattenOneSegment(versionedList.getVersion());
case NOP: // intentionally falling through
return;
case MERGE:
case COMPACT:
break;
default: throw new RuntimeException("Unknown action " + action); //
sanity check
}
// Create one segment representing all segments in the compaction
pipeline,
// either by compaction or by merge
if (!isInterrupted.get()) {
result = createSubstitution();
}
{code}
So if the policy returns FLATTEN we do flatten and also do the substitution if
the 'action' is already MERGE?
I ran the test - there is no full GC issue now because we keep on doing merge
but since the flush is not flushing all the segments there are lot of blocking
writes and that totally slows down the client and after some point of time
almost no mutation reach the server. There are also other issues but may not be
realted to this patch.
Thanks.
> Introducing the ability to merge ImmutableSegments without copy-compaction or
> SQM usage
> ---
>
> Key: HBASE-16608
> URL: https://issues.apache.org/jira/browse/HBASE-16608
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
> Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15490724#comment-15490724 ] Heng Chen commented on HBASE-16631: --- wdyt? [~stack] > Extract AsyncRequestFuture related code from AsyncProcess > - > > Key: HBASE-16631 > URL: https://issues.apache.org/jira/browse/HBASE-16631 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen >Assignee: Heng Chen > Attachments: HBASE-16631.v1.patch, HBASE-16631.wip.patch > > > Now, AsyncProcess class is too large (over 2000+ lines), and there are so > many sub classes in it. > AsyncRequestFutureImpl is the biggest subclass in AP, we could extract it > out from AP to reduce the AP size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15490690#comment-15490690 ] Ted Yu commented on HBASE-14123: [~stack]: Patch v20 has been uploaded to review board. Please review the code. Thanks > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, > 14123-master.v3.txt, 14123-master.v5.txt, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-16631: -- Status: Patch Available (was: Open) > Extract AsyncRequestFuture related code from AsyncProcess > - > > Key: HBASE-16631 > URL: https://issues.apache.org/jira/browse/HBASE-16631 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen >Assignee: Heng Chen > Attachments: HBASE-16631.v1.patch, HBASE-16631.wip.patch > > > Now, AsyncProcess class is too large (over 2000+ lines), and there are so > many sub classes in it. > AsyncRequestFutureImpl is the biggest subclass in AP, we could extract it > out from AP to reduce the AP size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14123: --- Attachment: 14123-master.v20.txt Patch v20 sync's up to commit e5d77d095c4f726c2cbaef25be36ff547c07126d > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, > 14123-master.v3.txt, 14123-master.v5.txt, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15490683#comment-15490683 ] Heng Chen commented on HBASE-16631: --- With this patch, AsyncProcess has 966 lines, and AsyncRequestFutureImpl has 1290 lines. > Extract AsyncRequestFuture related code from AsyncProcess > - > > Key: HBASE-16631 > URL: https://issues.apache.org/jira/browse/HBASE-16631 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen >Assignee: Heng Chen > Attachments: HBASE-16631.v1.patch, HBASE-16631.wip.patch > > > Now, AsyncProcess class is too large (over 2000+ lines), and there are so > many sub classes in it. > AsyncRequestFutureImpl is the biggest subclass in AP, we could extract it > out from AP to reduce the AP size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-16631: -- Attachment: HBASE-16631.v1.patch > Extract AsyncRequestFuture related code from AsyncProcess > - > > Key: HBASE-16631 > URL: https://issues.apache.org/jira/browse/HBASE-16631 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen >Assignee: Heng Chen > Attachments: HBASE-16631.v1.patch, HBASE-16631.wip.patch > > > Now, AsyncProcess class is too large (over 2000+ lines), and there are so > many sub classes in it. > AsyncRequestFutureImpl is the biggest subclass in AP, we could extract it > out from AP to reduce the AP size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16620) Fix backup command-line tool usability issues
[ https://issues.apache.org/jira/browse/HBASE-16620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16620: --- Labels: backup (was: ) Hadoop Flags: Reviewed > Fix backup command-line tool usability issues > - > > Key: HBASE-16620 > URL: https://issues.apache.org/jira/browse/HBASE-16620 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Attachments: 16620-v2.patch, 16620-v4.patch, HBASE-16620-v1.patch, > HBASE-16620-v3.patch > > > We need to address issues found by [~saint@gmail.com] > https://issues.apache.org/jira/browse/HBASE-7912?focusedCommentId=15484865=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15484865 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16620) Fix backup command-line tool usability issues
[ https://issues.apache.org/jira/browse/HBASE-16620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-16620. Resolution: Fixed Thanks for the patch, Vlad. > Fix backup command-line tool usability issues > - > > Key: HBASE-16620 > URL: https://issues.apache.org/jira/browse/HBASE-16620 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Attachments: 16620-v2.patch, 16620-v4.patch, HBASE-16620-v1.patch, > HBASE-16620-v3.patch > > > We need to address issues found by [~saint@gmail.com] > https://issues.apache.org/jira/browse/HBASE-7912?focusedCommentId=15484865=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15484865 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16620) Fix backup command-line tool usability issues
[ https://issues.apache.org/jira/browse/HBASE-16620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16620: --- Summary: Fix backup command-line tool usability issues (was: Command-line tool usability issues) > Fix backup command-line tool usability issues > - > > Key: HBASE-16620 > URL: https://issues.apache.org/jira/browse/HBASE-16620 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 16620-v2.patch, 16620-v4.patch, HBASE-16620-v1.patch, > HBASE-16620-v3.patch > > > We need to address issues found by [~saint@gmail.com] > https://issues.apache.org/jira/browse/HBASE-7912?focusedCommentId=15484865=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15484865 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-16631: -- Summary: Extract AsyncRequestFuture related code from AsyncProcess (was: Extract AsyncRequestFuture relates code from AsyncProcess) > Extract AsyncRequestFuture related code from AsyncProcess > - > > Key: HBASE-16631 > URL: https://issues.apache.org/jira/browse/HBASE-16631 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen >Assignee: Heng Chen > Attachments: HBASE-16631.wip.patch > > > Now, AsyncProcess class is too large (over 2000+ lines), and there are so > many sub classes in it. > AsyncRequestFutureImpl is the biggest subclass in AP, we could extract it > out from AP to reduce the AP size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16620) Command-line tool usability issues
[ https://issues.apache.org/jira/browse/HBASE-16620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16620: --- Attachment: 16620-v4.patch Patch v4 removes white spaces. Also changes test names. > Command-line tool usability issues > -- > > Key: HBASE-16620 > URL: https://issues.apache.org/jira/browse/HBASE-16620 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 16620-v2.patch, 16620-v4.patch, HBASE-16620-v1.patch, > HBASE-16620-v3.patch > > > We need to address issues found by [~saint@gmail.com] > https://issues.apache.org/jira/browse/HBASE-7912?focusedCommentId=15484865=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15484865 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16165) Decrease RpcServer.callQueueSize before writeResponse causes OOM
[
https://issues.apache.org/jira/browse/HBASE-16165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-16165:
--
Affects Version/s: 1.4.0
1.3.0
2.0.0
1.1.6
1.2.3
0.98.22
Labels: trivial (was: )
Fix Version/s: 1.2.4
0.98.23
1.1.7
1.4.0
1.3.0
2.0.0
Component/s: rpc
IPC/RPC
> Decrease RpcServer.callQueueSize before writeResponse causes OOM
>
>
> Key: HBASE-16165
> URL: https://issues.apache.org/jira/browse/HBASE-16165
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC, rpc
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.6, 1.2.3, 0.98.22
>Reporter: Duo Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Labels: trivial
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 0.98.23, 1.2.4
>
> Attachments: HBASE-16165.patch
>
>
> In RpcServer, we use {{callQueueSizeInBytes}} to avoid queuing too many calls
> which causes OOM. But in {{CallRunner.run}}, we decrease it before send the
> response back. And even after calling {{sendResponseIfReady}}, the call
> object could stay in our heap for a long time if we can not write out the
> response(That's why we need a Responder thread...). This makes it possible
> that the actual size of all call object in heap is larger than
> {{maxQueueSizeInBytes}} and causes OOM.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16165) Decrease RpcServer.callQueueSize before writeResponse causes OOM
[
https://issues.apache.org/jira/browse/HBASE-16165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15490375#comment-15490375
]
Duo Zhang commented on HBASE-16165:
---
This should be commited to all active branches.
[~mantonov] Is it safe to be commited to branch-1.3? A trivial change. Thanks.
> Decrease RpcServer.callQueueSize before writeResponse causes OOM
>
>
> Key: HBASE-16165
> URL: https://issues.apache.org/jira/browse/HBASE-16165
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC, rpc
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.6, 1.2.3, 0.98.22
>Reporter: Duo Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Labels: trivial
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 0.98.23, 1.2.4
>
> Attachments: HBASE-16165.patch
>
>
> In RpcServer, we use {{callQueueSizeInBytes}} to avoid queuing too many calls
> which causes OOM. But in {{CallRunner.run}}, we decrease it before send the
> response back. And even after calling {{sendResponseIfReady}}, the call
> object could stay in our heap for a long time if we can not write out the
> response(That's why we need a Responder thread...). This makes it possible
> that the actual size of all call object in heap is larger than
> {{maxQueueSizeInBytes}} and causes OOM.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16630) Fragmentation in long running Bucket Cache
[ https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15490281#comment-15490281 ] Anoop Sam John commented on HBASE-16630: Ya that assert was just added as we always passed HBB at that time. Now we can add functions to write a ByteBuff directly onto BBArray.. > Fragmentation in long running Bucket Cache > -- > > Key: HBASE-16630 > URL: https://issues.apache.org/jira/browse/HBASE-16630 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3 >Reporter: deepankar >Assignee: deepankar > Attachments: HBASE-16630.patch > > > As we are running bucket cache for a long time in our system, we are > observing cases where some nodes after some time does not fully utilize the > bucket cache, in some cases it is even worse in the sense they get stuck at a > value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables > are configured in-memory for simplicity sake). > We took a heap dump and analyzed what is happening and saw that is classic > case of fragmentation, current implementation of BucketCache (mainly > BucketAllocator) relies on the logic that fullyFreeBuckets are available for > switching/adjusting cache usage between different bucketSizes . But once a > compaction / bulkload happens and the blocks are evicted from a bucket size , > these are usually evicted from random places of the buckets of a bucketSize > and thus locking the number of buckets associated with a bucketSize and in > the worst case of the fragmentation we have seen some bucketSizes with > occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to > share with the other bucketSize. > Currently the existing eviction logic helps in the cases where cache used is > more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also > done, the eviction (freeSpace function) will not evict anything and the cache > utilization will be stuck at that value without any allocations for other > required sizes. > The fix for this we came up with is simple that we do deFragmentation ( > compaction) of the bucketSize and thus increasing the occupancy ratio and > also freeing up the buckets to be fullyFree, this logic itself is not > complicated as the bucketAllocator takes care of packing the blocks in the > buckets, we need evict and re-allocate the blocks for all the BucketSizes > that dont fit the criteria. > I am attaching an initial patch just to give an idea of what we are thinking > and I'll improve it based on the comments from the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16165) Decrease RpcServer.callQueueSize before writeResponse causes OOM
[
https://issues.apache.org/jira/browse/HBASE-16165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15490235#comment-15490235
]
binlijin commented on HBASE-16165:
--
+1.
> Decrease RpcServer.callQueueSize before writeResponse causes OOM
>
>
> Key: HBASE-16165
> URL: https://issues.apache.org/jira/browse/HBASE-16165
> Project: HBase
> Issue Type: Bug
>Reporter: Duo Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-16165.patch
>
>
> In RpcServer, we use {{callQueueSizeInBytes}} to avoid queuing too many calls
> which causes OOM. But in {{CallRunner.run}}, we decrease it before send the
> response back. And even after calling {{sendResponseIfReady}}, the call
> object could stay in our heap for a long time if we can not write out the
> response(That's why we need a Responder thread...). This makes it possible
> that the actual size of all call object in heap is larger than
> {{maxQueueSizeInBytes}} and causes OOM.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16165) Decrease RpcServer.callQueueSize before writeResponse causes OOM
[
https://issues.apache.org/jira/browse/HBASE-16165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15490195#comment-15490195
]
Hadoop QA commented on HBASE-16165:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 11s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 14s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 114m 5s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestFromClientSide |
| | org.apache.hadoop.hbase.client.TestScannerTimeout |
| |
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
| | org.apache.hadoop.hbase.client.TestMobSnapshotCloneIndependence |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12828420/HBASE-16165.patch |
| JIRA Issue | HBASE-16165 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux e96e8bf3e301 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4c6a98b |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3546/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3546/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3546/testReport/ |
|
[jira] [Commented] (HBASE-16165) Decrease RpcServer.callQueueSize before writeResponse causes OOM
[
https://issues.apache.org/jira/browse/HBASE-16165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15489945#comment-15489945
]
Duo Zhang commented on HBASE-16165:
---
+1.
> Decrease RpcServer.callQueueSize before writeResponse causes OOM
>
>
> Key: HBASE-16165
> URL: https://issues.apache.org/jira/browse/HBASE-16165
> Project: HBase
> Issue Type: Bug
>Reporter: Duo Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-16165.patch
>
>
> In RpcServer, we use {{callQueueSizeInBytes}} to avoid queuing too many calls
> which causes OOM. But in {{CallRunner.run}}, we decrease it before send the
> response back. And even after calling {{sendResponseIfReady}}, the call
> object could stay in our heap for a long time if we can not write out the
> response(That's why we need a Responder thread...). This makes it possible
> that the actual size of all call object in heap is larger than
> {{maxQueueSizeInBytes}} and causes OOM.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16165) Decrease RpcServer.callQueueSize before writeResponse causes OOM
[
https://issues.apache.org/jira/browse/HBASE-16165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15489936#comment-15489936
]
Guanghao Zhang commented on HBASE-16165:
We observed an OOM case in our production cluster. Table A in source cluster
has 500+ regions but it only has 1 region in slave cluster. Then the mr job
write a lot data in source cluster. It replicate to slave cluster and all data
write to one regionserver. Then the regionserver crashed by OOM. One fix is to
decrease RpcServer.callQueueSize when the responder wirte out the response
really. Another fix is nullify the param early. Upload a little fix for this
and set the param null when send response.
> Decrease RpcServer.callQueueSize before writeResponse causes OOM
>
>
> Key: HBASE-16165
> URL: https://issues.apache.org/jira/browse/HBASE-16165
> Project: HBase
> Issue Type: Bug
>Reporter: Duo Zhang
>Priority: Minor
> Attachments: HBASE-16165.patch
>
>
> In RpcServer, we use {{callQueueSizeInBytes}} to avoid queuing too many calls
> which causes OOM. But in {{CallRunner.run}}, we decrease it before send the
> response back. And even after calling {{sendResponseIfReady}}, the call
> object could stay in our heap for a long time if we can not write out the
> response(That's why we need a Responder thread...). This makes it possible
> that the actual size of all call object in heap is larger than
> {{maxQueueSizeInBytes}} and causes OOM.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16165) Decrease RpcServer.callQueueSize before writeResponse causes OOM
[
https://issues.apache.org/jira/browse/HBASE-16165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-16165:
---
Assignee: Guanghao Zhang
Status: Patch Available (was: Open)
> Decrease RpcServer.callQueueSize before writeResponse causes OOM
>
>
> Key: HBASE-16165
> URL: https://issues.apache.org/jira/browse/HBASE-16165
> Project: HBase
> Issue Type: Bug
>Reporter: Duo Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-16165.patch
>
>
> In RpcServer, we use {{callQueueSizeInBytes}} to avoid queuing too many calls
> which causes OOM. But in {{CallRunner.run}}, we decrease it before send the
> response back. And even after calling {{sendResponseIfReady}}, the call
> object could stay in our heap for a long time if we can not write out the
> response(That's why we need a Responder thread...). This makes it possible
> that the actual size of all call object in heap is larger than
> {{maxQueueSizeInBytes}} and causes OOM.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16165) Decrease RpcServer.callQueueSize before writeResponse causes OOM
[
https://issues.apache.org/jira/browse/HBASE-16165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-16165:
---
Attachment: HBASE-16165.patch
Upload a little fix. Set the param null when send response.
> Decrease RpcServer.callQueueSize before writeResponse causes OOM
>
>
> Key: HBASE-16165
> URL: https://issues.apache.org/jira/browse/HBASE-16165
> Project: HBase
> Issue Type: Bug
>Reporter: Duo Zhang
>Priority: Minor
> Attachments: HBASE-16165.patch
>
>
> In RpcServer, we use {{callQueueSizeInBytes}} to avoid queuing too many calls
> which causes OOM. But in {{CallRunner.run}}, we decrease it before send the
> response back. And even after calling {{sendResponseIfReady}}, the call
> object could stay in our heap for a long time if we can not write out the
> response(That's why we need a Responder thread...). This makes it possible
> that the actual size of all call object in heap is larger than
> {{maxQueueSizeInBytes}} and causes OOM.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
