[jira] [Commented] (HBASE-13170) Allow block cache to be external
[ https://issues.apache.org/jira/browse/HBASE-13170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351470#comment-14351470 ] Elliott Clark commented on HBASE-13170: --- Yes SSD's allow you to persist block cache between restarts. However that's not actually that helpful for some of my services. The current bucket cache does not allow two or more regionservers to share the same cache.Moving single regions for rolling restart can roll them to a full block cache. I can re-deploy at any time and the rolling restart only adds the latency of close/open region. It doesn't add the latency of a cold block cache. Read replicas can warm and share each other's cache. No need for double the memory just because there is a read replica. Additionally memcached is faster than the cheap ssd's and memory will last for many many more write cycles. No need to worry about wear leveling, or SSD garbage collection, erasure pages, etc. Finally memcache allows scaling the read caching of a HBase cluster independent of expensive storage nodes. > Allow block cache to be external > > > Key: HBASE-13170 > URL: https://issues.apache.org/jira/browse/HBASE-13170 > Project: HBase > Issue Type: New Feature > Components: io >Affects Versions: 2.0.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 1.1.0 > > > Allow an external service to provide the block cache. This has the nice > property of allowing failover/upgrades to happen without causing a fully cold > cache. > Additionally this allows read replicas to share some of the same memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13136) TestSplitLogManager.testGetPreviousRecoveryMode is flakey
[
https://issues.apache.org/jira/browse/HBASE-13136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangduo updated HBASE-13136:
-
Affects Version/s: 1.0.0
0.98.11
Fix Version/s: 0.98.12
1.1.0
1.0.1
2.0.0
0.98.11 is already RC so set to 0.98.12
> TestSplitLogManager.testGetPreviousRecoveryMode is flakey
> -
>
> Key: HBASE-13136
> URL: https://issues.apache.org/jira/browse/HBASE-13136
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.0.0, 2.0.0, 1.1.0, 0.98.11
>Reporter: zhangduo
>Assignee: zhangduo
> Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.12
>
> Attachments: HBASE-13136-0.98.patch, HBASE-13136.patch
>
>
> Add test code to run it 100 times then we can make it fail always.
> {code:title=TestSplitLogManager.java}
> @Test
> public void test() throws Exception {
> for (int i = 0; i < 100; i++) {
> setup();
> testGetPreviousRecoveryMode();
> teardown();
> }
> }
> {code}
> Add then add some ugly debug logs(Yeah I usually debug in this way...)
> {code:title=ZKSplitLogManagerCoordination.java}
> @Override
> public void setRecoveryMode(boolean isForInitialization) throws IOException
> {
> synchronized(this) {
> if (this.isDrainingDone) {
> // when there is no outstanding splitlogtask after master start up,
> we already have up to
> // date recovery mode
> return;
> }
> }
> if (this.watcher == null) {
> // when watcher is null(testing code) and recovery mode can only be
> LOG_SPLITTING
> synchronized(this) {
> this.isDrainingDone = true;
> this.recoveryMode = RecoveryMode.LOG_SPLITTING;
> }
> return;
> }
> boolean hasSplitLogTask = false;
> boolean hasRecoveringRegions = false;
> RecoveryMode previousRecoveryMode = RecoveryMode.UNKNOWN;
> RecoveryMode recoveryModeInConfig =
> (isDistributedLogReplay(conf)) ? RecoveryMode.LOG_REPLAY :
> RecoveryMode.LOG_SPLITTING;
> // Firstly check if there are outstanding recovering regions
> try {
> List regions = ZKUtil.listChildrenNoWatch(watcher,
> watcher.recoveringRegionsZNode);
> LOG.debug("===" + regions);
> if (regions != null && !regions.isEmpty()) {
> hasRecoveringRegions = true;
> previousRecoveryMode = RecoveryMode.LOG_REPLAY;
> }
> if (previousRecoveryMode == RecoveryMode.UNKNOWN) {
> // Secondly check if there are outstanding split log task
> List tasks = ZKUtil.listChildrenNoWatch(watcher,
> watcher.splitLogZNode);
> LOG.debug("===" + tasks);
> if (tasks != null && !tasks.isEmpty()) {
> hasSplitLogTask = true;
> if (isForInitialization) {
> // during initialization, try to get recovery mode from
> splitlogtask
> int listSize = tasks.size();
> for (int i = 0; i < listSize; i++) {
> String task = tasks.get(i);
> try {
> byte[] data =
> ZKUtil.getData(this.watcher,
> ZKUtil.joinZNode(watcher.splitLogZNode, task));
> if (data == null) continue;
> SplitLogTask slt = SplitLogTask.parseFrom(data);
> previousRecoveryMode = slt.getMode();
> if (previousRecoveryMode == RecoveryMode.UNKNOWN) {
> // created by old code base where we don't set recovery
> mode in splitlogtask
> // we can safely set to LOG_SPLITTING because we're in
> master initialization code
> // before SSH is enabled & there is no outstanding
> recovering regions
> previousRecoveryMode = RecoveryMode.LOG_SPLITTING;
> }
> break;
> } catch (DeserializationException e) {
> LOG.warn("Failed parse data for znode " + task, e);
> } catch (InterruptedException e) {
> throw new InterruptedIOException();
> }
> }
> }
> }
> }
> } catch (KeeperException e) {
> throw new IOException(e);
> }
> synchronized (this) {
> if (this.isDrainingDone) {
> return;
> }
> if (!hasSplitLogTask && !hasRecoveringRegions) {
> this.isDrainingDone = true;
> LOG.debug("set to " + recoveryModeInConfig);
> this.recoveryMode = recoveryModeInConfig;
> return;
> } else if (!isForInitialization) {
> // splitlogtask hasn't drained yet, keep existing recovery mode
> return;
> }
> if (previousRecoveryMode != RecoveryMode.UNKNOWN) {
> LOG.debug("set to " + previousRecover
[jira] [Commented] (HBASE-13133) NPE when running TestSplitLogManager
[
https://issues.apache.org/jira/browse/HBASE-13133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351467#comment-14351467
]
zhangduo commented on HBASE-13133:
--
1.0 also has this problem.
{noformat}
2015-03-07 15:37:00,907 ERROR [main-EventThread]
zookeeper.ClientCnxn$EventThread(613): Caught unexpected throwable
java.lang.NullPointerException
at
org.apache.hadoop.hbase.coordination.ZKSplitLogManagerCoordination.findOrCreateOrphanTask(ZKSplitLogManagerCoordination.java:547)
at
org.apache.hadoop.hbase.coordination.ZKSplitLogManagerCoordination.handleUnassignedTask(ZKSplitLogManagerCoordination.java:177)
at
org.apache.hadoop.hbase.coordination.ZKSplitLogManagerCoordination.getDataSetWatchSuccess(ZKSplitLogManagerCoordination.java:466)
at
org.apache.hadoop.hbase.coordination.ZKSplitLogManagerCoordination.access$6(ZKSplitLogManagerCoordination.java:449)
at
org.apache.hadoop.hbase.coordination.ZKSplitLogManagerCoordination$GetDataAsyncCallback.processResult(ZKSplitLogManagerCoordination.java:1037)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:561)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
{noformat}
> NPE when running TestSplitLogManager
>
>
> Key: HBASE-13133
> URL: https://issues.apache.org/jira/browse/HBASE-13133
> Project: HBase
> Issue Type: Bug
>Reporter: zhangduo
>Assignee: Andrey Stepachev
> Fix For: 2.0.0, 1.1.0
>
> Attachments: HBASE-13133.patch
>
>
> https://builds.apache.org/job/HBase-TRUNK/6187/testReport/junit/org.apache.hadoop.hbase.master/TestSplitLogManager/testOrphanTaskAcquisition/
> {noformat}
> 2015-03-01 01:34:58,902 INFO [Thread-23] master.TestSplitLogManager(298):
> TestOrphanTaskAcquisition
> 2015-03-01 01:34:58,904 DEBUG [Thread-23]
> coordination.ZKSplitLogManagerCoordination(870): Distributed log replay=true
> 2015-03-01 01:34:58,907 INFO [Thread-23]
> coordination.ZKSplitLogManagerCoordination(594): found orphan task
> orphan%2Ftest%2Fslash
> 2015-03-01 01:34:58,913 INFO [Thread-23]
> coordination.ZKSplitLogManagerCoordination(598): Found 1 orphan tasks and 0
> rescan nodes
> 2015-03-01 01:34:58,913 ERROR [main-EventThread]
> zookeeper.ClientCnxn$EventThread(613): Caught unexpected throwable
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.coordination.ZKSplitLogManagerCoordination.findOrCreateOrphanTask(ZKSplitLogManagerCoordination.java:546)
> at
> org.apache.hadoop.hbase.coordination.ZKSplitLogManagerCoordination.heartbeat(ZKSplitLogManagerCoordination.java:556)
> at
> org.apache.hadoop.hbase.coordination.ZKSplitLogManagerCoordination.getDataSetWatchSuccess(ZKSplitLogManagerCoordination.java:467)
> at
> org.apache.hadoop.hbase.coordination.ZKSplitLogManagerCoordination.access$700(ZKSplitLogManagerCoordination.java:74)
> at
> org.apache.hadoop.hbase.coordination.ZKSplitLogManagerCoordination$GetDataAsyncCallback.processResult(ZKSplitLogManagerCoordination.java:1020)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:561)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> {noformat}
> I got this NPE almost every time when running TestSplitLogManager locally. I
> am not sure whether it is the root cause of test failing, but seems related.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13136) TestSplitLogManager.testGetPreviousRecoveryMode is flakey
[
https://issues.apache.org/jira/browse/HBASE-13136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangduo updated HBASE-13136:
-
Attachment: HBASE-13136-0.98.patch
Patch for 0.98.
1.0 can use the same patch with master.
> TestSplitLogManager.testGetPreviousRecoveryMode is flakey
> -
>
> Key: HBASE-13136
> URL: https://issues.apache.org/jira/browse/HBASE-13136
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.1.0
>Reporter: zhangduo
>Assignee: zhangduo
> Attachments: HBASE-13136-0.98.patch, HBASE-13136.patch
>
>
> Add test code to run it 100 times then we can make it fail always.
> {code:title=TestSplitLogManager.java}
> @Test
> public void test() throws Exception {
> for (int i = 0; i < 100; i++) {
> setup();
> testGetPreviousRecoveryMode();
> teardown();
> }
> }
> {code}
> Add then add some ugly debug logs(Yeah I usually debug in this way...)
> {code:title=ZKSplitLogManagerCoordination.java}
> @Override
> public void setRecoveryMode(boolean isForInitialization) throws IOException
> {
> synchronized(this) {
> if (this.isDrainingDone) {
> // when there is no outstanding splitlogtask after master start up,
> we already have up to
> // date recovery mode
> return;
> }
> }
> if (this.watcher == null) {
> // when watcher is null(testing code) and recovery mode can only be
> LOG_SPLITTING
> synchronized(this) {
> this.isDrainingDone = true;
> this.recoveryMode = RecoveryMode.LOG_SPLITTING;
> }
> return;
> }
> boolean hasSplitLogTask = false;
> boolean hasRecoveringRegions = false;
> RecoveryMode previousRecoveryMode = RecoveryMode.UNKNOWN;
> RecoveryMode recoveryModeInConfig =
> (isDistributedLogReplay(conf)) ? RecoveryMode.LOG_REPLAY :
> RecoveryMode.LOG_SPLITTING;
> // Firstly check if there are outstanding recovering regions
> try {
> List regions = ZKUtil.listChildrenNoWatch(watcher,
> watcher.recoveringRegionsZNode);
> LOG.debug("===" + regions);
> if (regions != null && !regions.isEmpty()) {
> hasRecoveringRegions = true;
> previousRecoveryMode = RecoveryMode.LOG_REPLAY;
> }
> if (previousRecoveryMode == RecoveryMode.UNKNOWN) {
> // Secondly check if there are outstanding split log task
> List tasks = ZKUtil.listChildrenNoWatch(watcher,
> watcher.splitLogZNode);
> LOG.debug("===" + tasks);
> if (tasks != null && !tasks.isEmpty()) {
> hasSplitLogTask = true;
> if (isForInitialization) {
> // during initialization, try to get recovery mode from
> splitlogtask
> int listSize = tasks.size();
> for (int i = 0; i < listSize; i++) {
> String task = tasks.get(i);
> try {
> byte[] data =
> ZKUtil.getData(this.watcher,
> ZKUtil.joinZNode(watcher.splitLogZNode, task));
> if (data == null) continue;
> SplitLogTask slt = SplitLogTask.parseFrom(data);
> previousRecoveryMode = slt.getMode();
> if (previousRecoveryMode == RecoveryMode.UNKNOWN) {
> // created by old code base where we don't set recovery
> mode in splitlogtask
> // we can safely set to LOG_SPLITTING because we're in
> master initialization code
> // before SSH is enabled & there is no outstanding
> recovering regions
> previousRecoveryMode = RecoveryMode.LOG_SPLITTING;
> }
> break;
> } catch (DeserializationException e) {
> LOG.warn("Failed parse data for znode " + task, e);
> } catch (InterruptedException e) {
> throw new InterruptedIOException();
> }
> }
> }
> }
> }
> } catch (KeeperException e) {
> throw new IOException(e);
> }
> synchronized (this) {
> if (this.isDrainingDone) {
> return;
> }
> if (!hasSplitLogTask && !hasRecoveringRegions) {
> this.isDrainingDone = true;
> LOG.debug("set to " + recoveryModeInConfig);
> this.recoveryMode = recoveryModeInConfig;
> return;
> } else if (!isForInitialization) {
> // splitlogtask hasn't drained yet, keep existing recovery mode
> return;
> }
> if (previousRecoveryMode != RecoveryMode.UNKNOWN) {
> LOG.debug("set to " + previousRecoveryMode);
> this.isDrainingDone = (previousRecoveryMode == recoveryModeInConfig);
> this.recoveryMode = previousRecoveryMode;
> } else {
> LOG.debug("set to " +
[jira] [Commented] (HBASE-13170) Allow block cache to be external
[ https://issues.apache.org/jira/browse/HBASE-13170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351443#comment-14351443 ] Vladimir Rodionov commented on HBASE-13170: --- Yes, bucket cache with ioengine = file. > Allow block cache to be external > > > Key: HBASE-13170 > URL: https://issues.apache.org/jira/browse/HBASE-13170 > Project: HBase > Issue Type: New Feature > Components: io >Affects Versions: 2.0.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 1.1.0 > > > Allow an external service to provide the block cache. This has the nice > property of allowing failover/upgrades to happen without causing a fully cold > cache. > Additionally this allows read replicas to share some of the same memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12990) MetaScanner should be replaced by MetaTableAccessor
[
https://issues.apache.org/jira/browse/HBASE-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351442#comment-14351442
]
Hadoop QA commented on HBASE-12990:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703218/HBASE-12990.v7.patch
against master branch at commit 95fc53f1056238e0b5f65df7f3015f2aec3d0a20.
ATTACHMENT ID: 12703218
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 35 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13131//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13131//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13131//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13131//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13131//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13131//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13131//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13131//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13131//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13131//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13131//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13131//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13131//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13131//console
This message is automatically generated.
> MetaScanner should be replaced by MetaTableAccessor
> ---
>
> Key: HBASE-12990
> URL: https://issues.apache.org/jira/browse/HBASE-12990
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Affects Versions: 2.0.0
>Reporter: Andrey Stepachev
>Assignee: Andrey Stepachev
> Attachments: HBASE-12990.patch, HBASE-12990.v2.patch,
> HBASE-12990.v3.patch, HBASE-12990.v4.patch, HBASE-12990.v5.patch,
> HBASE-12990.v5.patch, HBASE-12990.v5.patch, HBASE-12990.v6.patch,
> HBASE-12990.v7.patch, HBASE-12990.v7.patch, HBASE-12990.v7.patch
>
>
> MetaScanner and MetaTableAccessor do similar things, but seems they tend to
> diverge. Let's have only one thing to enquiry META.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13168) Backport HBASE-12590 "A solution for data skew in HBase-Mapreduce Job"
[
https://issues.apache.org/jira/browse/HBASE-13168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351439#comment-14351439
]
Hadoop QA commented on HBASE-13168:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703219/13168-branch-1.patch
against branch-1 branch at commit 95fc53f1056238e0b5f65df7f3015f2aec3d0a20.
ATTACHMENT ID: 12703219
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
3813 checkstyle errors (more than the master's current 3811 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13132//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13132//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13132//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13132//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13132//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13132//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13132//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13132//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13132//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13132//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13132//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13132//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13132//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13132//console
This message is automatically generated.
> Backport HBASE-12590 "A solution for data skew in HBase-Mapreduce Job"
> --
>
> Key: HBASE-13168
> URL: https://issues.apache.org/jira/browse/HBASE-13168
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
>Reporter: Nick Dimiduk
>Assignee: Ted Yu
> Fix For: 1.1.0
>
> Attachments: 13168-branch-1.patch
>
>
> We should bring this back into active release branches. Seems like it
> shouldn't break any API compatibility, but should be checked more closely.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13170) Allow block cache to be external
[ https://issues.apache.org/jira/browse/HBASE-13170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351435#comment-14351435 ] Nick Dimiduk commented on HBASE-13170: -- Actually, any BucketCache when used with the save to file feature. But only the data from the bucketcache, iirc. > Allow block cache to be external > > > Key: HBASE-13170 > URL: https://issues.apache.org/jira/browse/HBASE-13170 > Project: HBase > Issue Type: New Feature > Components: io >Affects Versions: 2.0.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 1.1.0 > > > Allow an external service to provide the block cache. This has the nice > property of allowing failover/upgrades to happen without causing a fully cold > cache. > Additionally this allows read replicas to share some of the same memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13170) Allow block cache to be external
[ https://issues.apache.org/jira/browse/HBASE-13170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351426#comment-14351426 ] Vladimir Rodionov commented on HBASE-13170: --- SSD-based block cache gives you all you are asking for, [~eclark]. > Allow block cache to be external > > > Key: HBASE-13170 > URL: https://issues.apache.org/jira/browse/HBASE-13170 > Project: HBase > Issue Type: New Feature > Components: io >Affects Versions: 2.0.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 1.1.0 > > > Allow an external service to provide the block cache. This has the nice > property of allowing failover/upgrades to happen without causing a fully cold > cache. > Additionally this allows read replicas to share some of the same memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351425#comment-14351425 ] ramkrishna.s.vasudevan commented on HBASE-13142: Nice. Will try to port this to use this in our internal branch also. > [PERF] Reuse the IPCUtil#buildCellBlock buffer > -- > > Key: HBASE-13142 > URL: https://issues.apache.org/jira/browse/HBASE-13142 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Assignee: stack > Labels: beginner > Fix For: 2.0.0, 1.1.0 > > Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.0.98.txt, > 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, > hits.png, net.png, traces.2.svg, traces.svg > > > Running some scan profiling, flight recorder was mildly fingering resize of > the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It > was half-hearted blaming it for a few hundreds of ms over a five minute > sampling with a few tens of instances showing. > I tried then w/ flamegraph/lightweight profiler and this reported the buffer > allocations taking 22% of our total CPU. See attachment trace.svg. > I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and > indeed every allocation was doing a resize from initial allocation of 16k -- > the default up to 220k (this test returns ten randomly sized rows zipfian > sized between 0 and 8k). > Upping the allocation to 220k meant we now avoided the resize but the initial > allocation was now blamed for 10% of allocations (see trace.2.svg attached). > Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12586) Task 6 & 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection
[
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351413#comment-14351413
]
stack commented on HBASE-12586:
---
The former... "a lot of tests needs to be updated" This is good work.
> Task 6 & 7 from HBASE-9117, delete all public HTable constructors and delete
> ConnectionManager#{delete,get}Connection
> --
>
> Key: HBASE-12586
> URL: https://issues.apache.org/jira/browse/HBASE-12586
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: Mikhail Antonov
> Labels: beginner, beginners
> Attachments: HBASE-12586.patch, HBASE-12586.patch
>
>
> Finish cleanup from HBASE-9117 removing old API. This issue covers tasks 6
> and 7 from the list here:
> https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
> To be done in master branch only.
> Marked as beginner task. The idea is straight-forward. It is just a lot of
> work going through all tests converting to use new API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13159) Consider RangeReferenceFiles with transformations
[ https://issues.apache.org/jira/browse/HBASE-13159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351410#comment-14351410 ] stack commented on HBASE-13159: --- Tell us more about use cases. Give us example transforms (other than identity and 'drop'... or is that it?) Regards transforms, ain't that just filter at HFile level? How would you get the 'view' up to high level when transform is at the HFile level? bq. ...and can be referred from multiple regions Always shied from supporting this. Was afraid of the complexity and dangling pointers. I suppose if we had RangeReferenceFiles, could help at log split time. Same count of files so same NN traffic... we'd just be reading from one file. With current small files, you can remove them when done but with RangeReferenceFiles, you'd need to do refcounting... in meta? RangeReferenceFiles would not save NN traffic since we have to drop them in the FS but could save i/o not replicating Cells. @enis The index would be persisted in HFile or to a separate RangeReferenceFile? The benefit is smaller indices? Transform would add prefix back? Will that work? (maybe I'm not following) > Consider RangeReferenceFiles with transformations > - > > Key: HBASE-13159 > URL: https://issues.apache.org/jira/browse/HBASE-13159 > Project: HBase > Issue Type: Brainstorming >Reporter: Lars Hofhansl > > Currently we have References used by HalfStoreReaders and HFileLinks. > For various use cases we have here we have need for a RangeReferences with > simple transformation of the keys. > That would allow us to map HFiles between regions or even tables without > copying any data. > We can probably combine HalfStores, HFileLinks, and RangeReferences into a > single concept: > * RangeReference = arbitrary start and stop row, arbitrary key transformation > * HFileLink = start and stop keys set to the linked file's start/stop key, > transformation = identity > * (HalfStore) References = start/stop key set according to top or bottom > reference, transformation = identity > Note this is a *brainstorming* issue. :) > (Could start with just references with arbitrary start/stop keys, and do > transformations later) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13164) Update TestUsersOperationsWithSecureHadoop to use MiniKdc
[ https://issues.apache.org/jira/browse/HBASE-13164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351411#comment-14351411 ] Hudson commented on HBASE-13164: SUCCESS: Integrated in HBase-1.1 #258 (See [https://builds.apache.org/job/HBase-1.1/258/]) HBASE-13164 Update TestUsersOperationsWithSecureHadoop to use MiniKdc (Duo Zhang) (tedyu: rev e5efca212d457de003078c16da7e3be2f273b221) * hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestUsersOperationsWithSecureHadoop.java > Update TestUsersOperationsWithSecureHadoop to use MiniKdc > - > > Key: HBASE-13164 > URL: https://issues.apache.org/jira/browse/HBASE-13164 > Project: HBase > Issue Type: Sub-task > Components: security, test >Reporter: zhangduo >Assignee: zhangduo > Fix For: 2.0.0, 1.1.0 > > Attachments: HBASE-13164-branch-1.patch, HBASE-13164.patch > > > There is only one unit test besides TestSecureRPC which requires a kerberos > environment. Update it to use MiniKdc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12586) Task 6 & 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection
[
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351405#comment-14351405
]
Mikhail Antonov commented on HBASE-12586:
-
[~stack] Yeah... contrary in sense - "a _lot_ of tests needs to be updated", or
at more fundamental level? I hope to get tests fixed pretty soon and update the
patch. This patch doesn't yet include changes regarding {delete, get}Connection
methods - that's next.
> Task 6 & 7 from HBASE-9117, delete all public HTable constructors and delete
> ConnectionManager#{delete,get}Connection
> --
>
> Key: HBASE-12586
> URL: https://issues.apache.org/jira/browse/HBASE-12586
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: Mikhail Antonov
> Labels: beginner, beginners
> Attachments: HBASE-12586.patch, HBASE-12586.patch
>
>
> Finish cleanup from HBASE-9117 removing old API. This issue covers tasks 6
> and 7 from the list here:
> https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
> To be done in master branch only.
> Marked as beginner task. The idea is straight-forward. It is just a lot of
> work going through all tests converting to use new API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12586) Task 6 & 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection
[
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351392#comment-14351392
]
stack commented on HBASE-12586:
---
+1 on patches that remove more code than they add. This one could be a little
contrary to get in [~mantonov]... lots of presumption out in test land.
> Task 6 & 7 from HBASE-9117, delete all public HTable constructors and delete
> ConnectionManager#{delete,get}Connection
> --
>
> Key: HBASE-12586
> URL: https://issues.apache.org/jira/browse/HBASE-12586
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: Mikhail Antonov
> Labels: beginner, beginners
> Attachments: HBASE-12586.patch, HBASE-12586.patch
>
>
> Finish cleanup from HBASE-9117 removing old API. This issue covers tasks 6
> and 7 from the list here:
> https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
> To be done in master branch only.
> Marked as beginner task. The idea is straight-forward. It is just a lot of
> work going through all tests converting to use new API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13168) Backport HBASE-12590 "A solution for data skew in HBase-Mapreduce Job"
[ https://issues.apache.org/jira/browse/HBASE-13168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-13168: --- Attachment: (was: 13168-branch-1.patch) > Backport HBASE-12590 "A solution for data skew in HBase-Mapreduce Job" > -- > > Key: HBASE-13168 > URL: https://issues.apache.org/jira/browse/HBASE-13168 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Nick Dimiduk >Assignee: Ted Yu > Fix For: 1.1.0 > > Attachments: 13168-branch-1.patch > > > We should bring this back into active release branches. Seems like it > shouldn't break any API compatibility, but should be checked more closely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13168) Backport HBASE-12590 "A solution for data skew in HBase-Mapreduce Job"
[ https://issues.apache.org/jira/browse/HBASE-13168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-13168: --- Attachment: 13168-branch-1.patch > Backport HBASE-12590 "A solution for data skew in HBase-Mapreduce Job" > -- > > Key: HBASE-13168 > URL: https://issues.apache.org/jira/browse/HBASE-13168 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Nick Dimiduk >Assignee: Ted Yu > Fix For: 1.1.0 > > Attachments: 13168-branch-1.patch > > > We should bring this back into active release branches. Seems like it > shouldn't break any API compatibility, but should be checked more closely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12990) MetaScanner should be replaced by MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12990: -- Attachment: HBASE-12990.v7.patch Retry > MetaScanner should be replaced by MetaTableAccessor > --- > > Key: HBASE-12990 > URL: https://issues.apache.org/jira/browse/HBASE-12990 > Project: HBase > Issue Type: Improvement > Components: Client >Affects Versions: 2.0.0 >Reporter: Andrey Stepachev >Assignee: Andrey Stepachev > Attachments: HBASE-12990.patch, HBASE-12990.v2.patch, > HBASE-12990.v3.patch, HBASE-12990.v4.patch, HBASE-12990.v5.patch, > HBASE-12990.v5.patch, HBASE-12990.v5.patch, HBASE-12990.v6.patch, > HBASE-12990.v7.patch, HBASE-12990.v7.patch, HBASE-12990.v7.patch > > > MetaScanner and MetaTableAccessor do similar things, but seems they tend to > diverge. Let's have only one thing to enquiry META. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12405) WAL accounting by Store
[
https://issues.apache.org/jira/browse/HBASE-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351386#comment-14351386
]
Hadoop QA commented on HBASE-12405:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703207/HBASE-12405_5.patch
against master branch at commit 95fc53f1056238e0b5f65df7f3015f2aec3d0a20.
ATTACHMENT ID: 12703207
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ storeCompleteSequenceId_ =
java.util.Collections.unmodifiableList(storeCompleteSequenceId_);
+storeCompleteSequenceId_ =
java.util.Collections.unmodifiableList(storeCompleteSequenceId_);
+ new java.lang.String[] { "RegionSpecifier", "Stores",
"Storefiles", "StoreUncompressedSizeMB", "StorefileSizeMB", "MemstoreSizeMB",
"StorefileIndexSizeMB", "ReadRequestsCount", "WriteRequestsCount",
"TotalCompactingKVs", "CurrentCompactedKVs", "RootIndexSizeKB",
"TotalStaticIndexSizeKB", "TotalStaticBloomSizeKB", "CompleteSequenceId",
"DataLocality", "LastMajorCompactionTs", "StoreCompleteSequenceId", });
+ storeLastFlushedSequenceId_ =
java.util.Collections.unmodifiableList(storeLastFlushedSequenceId_);
+storeLastFlushedSequenceId_ =
java.util.Collections.unmodifiableList(storeLastFlushedSequenceId_);
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13129//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13129//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13129//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13129//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13129//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13129//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13129//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13129//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13129//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13129//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13129//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13129//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13129//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13129//console
This message is automatically generated.
> WAL accounting by Store
> ---
>
> Key: HBASE-12405
> URL: https://issues.apache.org/jira/browse/HBASE-12405
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 2.0.0, 1.1.0
>Reporter: zhangduo
>Assignee: zhangduo
> Fix For: 2.0.0, 1.1.0
>
> Attachments: HBASE-1240
[jira] [Commented] (HBASE-12586) Task 6 & 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection
[
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351378#comment-14351378
]
Hadoop QA commented on HBASE-12586:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703205/HBASE-12586.patch
against master branch at commit 95fc53f1056238e0b5f65df7f3015f2aec3d0a20.
ATTACHMENT ID: 12703205
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 27 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1931 checkstyle errors (more than the master's current 1930 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.wal.TestLogRollPeriod
org.apache.hadoop.hbase.master.handler.TestTableDescriptorModification
org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster
org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable
org.apache.hadoop.hbase.regionserver.wal.TestWALReplay
org.apache.hadoop.hbase.regionserver.TestJoinedScanners
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithACL
org.apache.hadoop.hbase.mapred.TestTableMapReduceUtil
org.apache.hadoop.hbase.master.TestGetLastFlushedSequenceId
org.apache.hadoop.hbase.master.TestMaster
org.apache.hadoop.hbase.regionserver.TestTags
org.apache.hadoop.hbase.security.access.TestCellACLWithMultipleVersions
org.apache.hadoop.hbase.security.access.TestAccessController
org.apache.hadoop.hbase.quotas.TestQuotaAdmin
org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.client.TestTimestampsFilter
org.apache.hadoop.hbase.TestMetaTableAccessorNoCluster
org.apache.hadoop.hbase.replication.TestReplicationKillMasterRS
org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint
org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat
org.apache.hadoop.hbase.security.access.TestAccessController2
org.apache.hadoop.hbase.trace.TestHTraceHooks
org.apache.hadoop.hbase.master.TestRegionPlacement
org.apache.hadoop.hbase.regionserver.TestCompactionState
org.apache.hadoop.hbase.mapreduce.TestImportTSVWithTTLs
org.apache.hadoop.hbase.fs.TestBlockReorder
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling
org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel
org.apache.hadoop.hbase.filter.TestMultiRowRangeFilter
org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
org.apache.hadoop.hbase.util.TestHBaseFsck
org.apache.hadoop.hbase.replication.TestReplicationDisableInactivePeer
org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan2
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildHole
org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink
org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
org.apache.hadoop.hbase.replication.TestReplicationKillSlaveRS
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed
org.apache.hadoop.hbase.filter.TestColumnRangeFilter
org.apache.hadoop.hbase.m
[jira] [Commented] (HBASE-12990) MetaScanner should be replaced by MetaTableAccessor
[
https://issues.apache.org/jira/browse/HBASE-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351375#comment-14351375
]
Hadoop QA commented on HBASE-12990:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703200/HBASE-12990.v7.patch
against master branch at commit 95fc53f1056238e0b5f65df7f3015f2aec3d0a20.
ATTACHMENT ID: 12703200
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 35 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestImportExport
org.apache.hadoop.hbase.util.TestProcessBasedCluster
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13127//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13127//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13127//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13127//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13127//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13127//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13127//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13127//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13127//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13127//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13127//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13127//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13127//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13127//console
This message is automatically generated.
> MetaScanner should be replaced by MetaTableAccessor
> ---
>
> Key: HBASE-12990
> URL: https://issues.apache.org/jira/browse/HBASE-12990
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Affects Versions: 2.0.0
>Reporter: Andrey Stepachev
>Assignee: Andrey Stepachev
> Attachments: HBASE-12990.patch, HBASE-12990.v2.patch,
> HBASE-12990.v3.patch, HBASE-12990.v4.patch, HBASE-12990.v5.patch,
> HBASE-12990.v5.patch, HBASE-12990.v5.patch, HBASE-12990.v6.patch,
> HBASE-12990.v7.patch, HBASE-12990.v7.patch
>
>
> MetaScanner and MetaTableAccessor do similar things, but seems they tend to
> diverge. Let's have only one thing to enquiry META.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13168) Backport HBASE-12590 "A solution for data skew in HBase-Mapreduce Job"
[ https://issues.apache.org/jira/browse/HBASE-13168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-13168: --- Fix Version/s: 1.1.0 > Backport HBASE-12590 "A solution for data skew in HBase-Mapreduce Job" > -- > > Key: HBASE-13168 > URL: https://issues.apache.org/jira/browse/HBASE-13168 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Nick Dimiduk >Assignee: Ted Yu > Fix For: 1.1.0 > > Attachments: 13168-branch-1.patch > > > We should bring this back into active release branches. Seems like it > shouldn't break any API compatibility, but should be checked more closely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13168) Backport HBASE-12590 "A solution for data skew in HBase-Mapreduce Job"
[ https://issues.apache.org/jira/browse/HBASE-13168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-13168: --- Status: Patch Available (was: Open) > Backport HBASE-12590 "A solution for data skew in HBase-Mapreduce Job" > -- > > Key: HBASE-13168 > URL: https://issues.apache.org/jira/browse/HBASE-13168 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Nick Dimiduk >Assignee: Ted Yu > Attachments: 13168-branch-1.patch > > > We should bring this back into active release branches. Seems like it > shouldn't break any API compatibility, but should be checked more closely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13168) Backport HBASE-12590 "A solution for data skew in HBase-Mapreduce Job"
[ https://issues.apache.org/jira/browse/HBASE-13168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-13168: --- Attachment: 13168-branch-1.patch > Backport HBASE-12590 "A solution for data skew in HBase-Mapreduce Job" > -- > > Key: HBASE-13168 > URL: https://issues.apache.org/jira/browse/HBASE-13168 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Nick Dimiduk >Assignee: Ted Yu > Fix For: 1.1.0 > > Attachments: 13168-branch-1.patch > > > We should bring this back into active release branches. Seems like it > shouldn't break any API compatibility, but should be checked more closely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13136) TestSplitLogManager.testGetPreviousRecoveryMode is flakey
[
https://issues.apache.org/jira/browse/HBASE-13136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351371#comment-14351371
]
zhangduo commented on HBASE-13136:
--
OK, will do it later.
> TestSplitLogManager.testGetPreviousRecoveryMode is flakey
> -
>
> Key: HBASE-13136
> URL: https://issues.apache.org/jira/browse/HBASE-13136
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.1.0
>Reporter: zhangduo
>Assignee: zhangduo
> Attachments: HBASE-13136.patch
>
>
> Add test code to run it 100 times then we can make it fail always.
> {code:title=TestSplitLogManager.java}
> @Test
> public void test() throws Exception {
> for (int i = 0; i < 100; i++) {
> setup();
> testGetPreviousRecoveryMode();
> teardown();
> }
> }
> {code}
> Add then add some ugly debug logs(Yeah I usually debug in this way...)
> {code:title=ZKSplitLogManagerCoordination.java}
> @Override
> public void setRecoveryMode(boolean isForInitialization) throws IOException
> {
> synchronized(this) {
> if (this.isDrainingDone) {
> // when there is no outstanding splitlogtask after master start up,
> we already have up to
> // date recovery mode
> return;
> }
> }
> if (this.watcher == null) {
> // when watcher is null(testing code) and recovery mode can only be
> LOG_SPLITTING
> synchronized(this) {
> this.isDrainingDone = true;
> this.recoveryMode = RecoveryMode.LOG_SPLITTING;
> }
> return;
> }
> boolean hasSplitLogTask = false;
> boolean hasRecoveringRegions = false;
> RecoveryMode previousRecoveryMode = RecoveryMode.UNKNOWN;
> RecoveryMode recoveryModeInConfig =
> (isDistributedLogReplay(conf)) ? RecoveryMode.LOG_REPLAY :
> RecoveryMode.LOG_SPLITTING;
> // Firstly check if there are outstanding recovering regions
> try {
> List regions = ZKUtil.listChildrenNoWatch(watcher,
> watcher.recoveringRegionsZNode);
> LOG.debug("===" + regions);
> if (regions != null && !regions.isEmpty()) {
> hasRecoveringRegions = true;
> previousRecoveryMode = RecoveryMode.LOG_REPLAY;
> }
> if (previousRecoveryMode == RecoveryMode.UNKNOWN) {
> // Secondly check if there are outstanding split log task
> List tasks = ZKUtil.listChildrenNoWatch(watcher,
> watcher.splitLogZNode);
> LOG.debug("===" + tasks);
> if (tasks != null && !tasks.isEmpty()) {
> hasSplitLogTask = true;
> if (isForInitialization) {
> // during initialization, try to get recovery mode from
> splitlogtask
> int listSize = tasks.size();
> for (int i = 0; i < listSize; i++) {
> String task = tasks.get(i);
> try {
> byte[] data =
> ZKUtil.getData(this.watcher,
> ZKUtil.joinZNode(watcher.splitLogZNode, task));
> if (data == null) continue;
> SplitLogTask slt = SplitLogTask.parseFrom(data);
> previousRecoveryMode = slt.getMode();
> if (previousRecoveryMode == RecoveryMode.UNKNOWN) {
> // created by old code base where we don't set recovery
> mode in splitlogtask
> // we can safely set to LOG_SPLITTING because we're in
> master initialization code
> // before SSH is enabled & there is no outstanding
> recovering regions
> previousRecoveryMode = RecoveryMode.LOG_SPLITTING;
> }
> break;
> } catch (DeserializationException e) {
> LOG.warn("Failed parse data for znode " + task, e);
> } catch (InterruptedException e) {
> throw new InterruptedIOException();
> }
> }
> }
> }
> }
> } catch (KeeperException e) {
> throw new IOException(e);
> }
> synchronized (this) {
> if (this.isDrainingDone) {
> return;
> }
> if (!hasSplitLogTask && !hasRecoveringRegions) {
> this.isDrainingDone = true;
> LOG.debug("set to " + recoveryModeInConfig);
> this.recoveryMode = recoveryModeInConfig;
> return;
> } else if (!isForInitialization) {
> // splitlogtask hasn't drained yet, keep existing recovery mode
> return;
> }
> if (previousRecoveryMode != RecoveryMode.UNKNOWN) {
> LOG.debug("set to " + previousRecoveryMode);
> this.isDrainingDone = (previousRecoveryMode == recoveryModeInConfig);
> this.recoveryMode = previousRecoveryMode;
> } else {
> LOG.debug("set to " + recoveryModeInConfig);
> this.re
[jira] [Commented] (HBASE-13136) TestSplitLogManager.testGetPreviousRecoveryMode is flakey
[
https://issues.apache.org/jira/browse/HBASE-13136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351370#comment-14351370
]
stack commented on HBASE-13136:
---
Add here [~Apache9]
> TestSplitLogManager.testGetPreviousRecoveryMode is flakey
> -
>
> Key: HBASE-13136
> URL: https://issues.apache.org/jira/browse/HBASE-13136
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.1.0
>Reporter: zhangduo
>Assignee: zhangduo
> Attachments: HBASE-13136.patch
>
>
> Add test code to run it 100 times then we can make it fail always.
> {code:title=TestSplitLogManager.java}
> @Test
> public void test() throws Exception {
> for (int i = 0; i < 100; i++) {
> setup();
> testGetPreviousRecoveryMode();
> teardown();
> }
> }
> {code}
> Add then add some ugly debug logs(Yeah I usually debug in this way...)
> {code:title=ZKSplitLogManagerCoordination.java}
> @Override
> public void setRecoveryMode(boolean isForInitialization) throws IOException
> {
> synchronized(this) {
> if (this.isDrainingDone) {
> // when there is no outstanding splitlogtask after master start up,
> we already have up to
> // date recovery mode
> return;
> }
> }
> if (this.watcher == null) {
> // when watcher is null(testing code) and recovery mode can only be
> LOG_SPLITTING
> synchronized(this) {
> this.isDrainingDone = true;
> this.recoveryMode = RecoveryMode.LOG_SPLITTING;
> }
> return;
> }
> boolean hasSplitLogTask = false;
> boolean hasRecoveringRegions = false;
> RecoveryMode previousRecoveryMode = RecoveryMode.UNKNOWN;
> RecoveryMode recoveryModeInConfig =
> (isDistributedLogReplay(conf)) ? RecoveryMode.LOG_REPLAY :
> RecoveryMode.LOG_SPLITTING;
> // Firstly check if there are outstanding recovering regions
> try {
> List regions = ZKUtil.listChildrenNoWatch(watcher,
> watcher.recoveringRegionsZNode);
> LOG.debug("===" + regions);
> if (regions != null && !regions.isEmpty()) {
> hasRecoveringRegions = true;
> previousRecoveryMode = RecoveryMode.LOG_REPLAY;
> }
> if (previousRecoveryMode == RecoveryMode.UNKNOWN) {
> // Secondly check if there are outstanding split log task
> List tasks = ZKUtil.listChildrenNoWatch(watcher,
> watcher.splitLogZNode);
> LOG.debug("===" + tasks);
> if (tasks != null && !tasks.isEmpty()) {
> hasSplitLogTask = true;
> if (isForInitialization) {
> // during initialization, try to get recovery mode from
> splitlogtask
> int listSize = tasks.size();
> for (int i = 0; i < listSize; i++) {
> String task = tasks.get(i);
> try {
> byte[] data =
> ZKUtil.getData(this.watcher,
> ZKUtil.joinZNode(watcher.splitLogZNode, task));
> if (data == null) continue;
> SplitLogTask slt = SplitLogTask.parseFrom(data);
> previousRecoveryMode = slt.getMode();
> if (previousRecoveryMode == RecoveryMode.UNKNOWN) {
> // created by old code base where we don't set recovery
> mode in splitlogtask
> // we can safely set to LOG_SPLITTING because we're in
> master initialization code
> // before SSH is enabled & there is no outstanding
> recovering regions
> previousRecoveryMode = RecoveryMode.LOG_SPLITTING;
> }
> break;
> } catch (DeserializationException e) {
> LOG.warn("Failed parse data for znode " + task, e);
> } catch (InterruptedException e) {
> throw new InterruptedIOException();
> }
> }
> }
> }
> }
> } catch (KeeperException e) {
> throw new IOException(e);
> }
> synchronized (this) {
> if (this.isDrainingDone) {
> return;
> }
> if (!hasSplitLogTask && !hasRecoveringRegions) {
> this.isDrainingDone = true;
> LOG.debug("set to " + recoveryModeInConfig);
> this.recoveryMode = recoveryModeInConfig;
> return;
> } else if (!isForInitialization) {
> // splitlogtask hasn't drained yet, keep existing recovery mode
> return;
> }
> if (previousRecoveryMode != RecoveryMode.UNKNOWN) {
> LOG.debug("set to " + previousRecoveryMode);
> this.isDrainingDone = (previousRecoveryMode == recoveryModeInConfig);
> this.recoveryMode = previousRecoveryMode;
> } else {
> LOG.debug("set to " + recoveryModeInConfig);
> this.recoveryMo
[jira] [Commented] (HBASE-12405) WAL accounting by Store
[ https://issues.apache.org/jira/browse/HBASE-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351369#comment-14351369 ] stack commented on HBASE-12405: --- [~Apache9] Ok. Lets verify we have the basics then interact with HBASE-11569. Let me work up an ITBLL to do above and get some results. > WAL accounting by Store > --- > > Key: HBASE-12405 > URL: https://issues.apache.org/jira/browse/HBASE-12405 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 2.0.0, 1.1.0 >Reporter: zhangduo >Assignee: zhangduo > Fix For: 2.0.0, 1.1.0 > > Attachments: HBASE-12405.patch, HBASE-12405_1.patch, > HBASE-12405_2.patch, HBASE-12405_3.patch, HBASE-12405_4.patch, > HBASE-12405_5.patch > > > HBASE-10201 has made flush decisions per Store, but has not done enough work > on HLog, so there are two problems: > 1. We record minSeqId both in HRegion and FSHLog, which is a duplication. > 2. There maybe holes in WAL accounting. > For example, assume family A with sequence id 1 and 3, family B with > seqId 2. If we flush family A, we can only record that WAL before sequence id > 1 can be removed safely. If we do a replay at this point, sequence id 3 will > also be replayed which is unnecessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13136) TestSplitLogManager.testGetPreviousRecoveryMode is flakey
[
https://issues.apache.org/jira/browse/HBASE-13136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351368#comment-14351368
]
zhangduo commented on HBASE-13136:
--
So raise a back port issue or just prepare patch for 1.0 and 0.98 here? Thanks~
> TestSplitLogManager.testGetPreviousRecoveryMode is flakey
> -
>
> Key: HBASE-13136
> URL: https://issues.apache.org/jira/browse/HBASE-13136
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.1.0
>Reporter: zhangduo
>Assignee: zhangduo
> Attachments: HBASE-13136.patch
>
>
> Add test code to run it 100 times then we can make it fail always.
> {code:title=TestSplitLogManager.java}
> @Test
> public void test() throws Exception {
> for (int i = 0; i < 100; i++) {
> setup();
> testGetPreviousRecoveryMode();
> teardown();
> }
> }
> {code}
> Add then add some ugly debug logs(Yeah I usually debug in this way...)
> {code:title=ZKSplitLogManagerCoordination.java}
> @Override
> public void setRecoveryMode(boolean isForInitialization) throws IOException
> {
> synchronized(this) {
> if (this.isDrainingDone) {
> // when there is no outstanding splitlogtask after master start up,
> we already have up to
> // date recovery mode
> return;
> }
> }
> if (this.watcher == null) {
> // when watcher is null(testing code) and recovery mode can only be
> LOG_SPLITTING
> synchronized(this) {
> this.isDrainingDone = true;
> this.recoveryMode = RecoveryMode.LOG_SPLITTING;
> }
> return;
> }
> boolean hasSplitLogTask = false;
> boolean hasRecoveringRegions = false;
> RecoveryMode previousRecoveryMode = RecoveryMode.UNKNOWN;
> RecoveryMode recoveryModeInConfig =
> (isDistributedLogReplay(conf)) ? RecoveryMode.LOG_REPLAY :
> RecoveryMode.LOG_SPLITTING;
> // Firstly check if there are outstanding recovering regions
> try {
> List regions = ZKUtil.listChildrenNoWatch(watcher,
> watcher.recoveringRegionsZNode);
> LOG.debug("===" + regions);
> if (regions != null && !regions.isEmpty()) {
> hasRecoveringRegions = true;
> previousRecoveryMode = RecoveryMode.LOG_REPLAY;
> }
> if (previousRecoveryMode == RecoveryMode.UNKNOWN) {
> // Secondly check if there are outstanding split log task
> List tasks = ZKUtil.listChildrenNoWatch(watcher,
> watcher.splitLogZNode);
> LOG.debug("===" + tasks);
> if (tasks != null && !tasks.isEmpty()) {
> hasSplitLogTask = true;
> if (isForInitialization) {
> // during initialization, try to get recovery mode from
> splitlogtask
> int listSize = tasks.size();
> for (int i = 0; i < listSize; i++) {
> String task = tasks.get(i);
> try {
> byte[] data =
> ZKUtil.getData(this.watcher,
> ZKUtil.joinZNode(watcher.splitLogZNode, task));
> if (data == null) continue;
> SplitLogTask slt = SplitLogTask.parseFrom(data);
> previousRecoveryMode = slt.getMode();
> if (previousRecoveryMode == RecoveryMode.UNKNOWN) {
> // created by old code base where we don't set recovery
> mode in splitlogtask
> // we can safely set to LOG_SPLITTING because we're in
> master initialization code
> // before SSH is enabled & there is no outstanding
> recovering regions
> previousRecoveryMode = RecoveryMode.LOG_SPLITTING;
> }
> break;
> } catch (DeserializationException e) {
> LOG.warn("Failed parse data for znode " + task, e);
> } catch (InterruptedException e) {
> throw new InterruptedIOException();
> }
> }
> }
> }
> }
> } catch (KeeperException e) {
> throw new IOException(e);
> }
> synchronized (this) {
> if (this.isDrainingDone) {
> return;
> }
> if (!hasSplitLogTask && !hasRecoveringRegions) {
> this.isDrainingDone = true;
> LOG.debug("set to " + recoveryModeInConfig);
> this.recoveryMode = recoveryModeInConfig;
> return;
> } else if (!isForInitialization) {
> // splitlogtask hasn't drained yet, keep existing recovery mode
> return;
> }
> if (previousRecoveryMode != RecoveryMode.UNKNOWN) {
> LOG.debug("set to " + previousRecoveryMode);
> this.isDrainingDone = (previousRecoveryMode == recoveryModeInConfig);
> this.recoveryMode = previousRecoveryMode;
> } else {
> LOG.debu
[jira] [Assigned] (HBASE-13168) Backport HBASE-12590 "A solution for data skew in HBase-Mapreduce Job"
[ https://issues.apache.org/jira/browse/HBASE-13168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-13168: -- Assignee: Ted Yu > Backport HBASE-12590 "A solution for data skew in HBase-Mapreduce Job" > -- > > Key: HBASE-13168 > URL: https://issues.apache.org/jira/browse/HBASE-13168 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Nick Dimiduk >Assignee: Ted Yu > > We should bring this back into active release branches. Seems like it > shouldn't break any API compatibility, but should be checked more closely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13169) ModifyTable increasing the region replica count should also auto-setup RRRE
[
https://issues.apache.org/jira/browse/HBASE-13169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351366#comment-14351366
]
Hadoop QA commented on HBASE-13169:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703197/hbase-13169_v1.patch
against master branch at commit 95fc53f1056238e0b5f65df7f3015f2aec3d0a20.
ATTACHMENT ID: 12703197
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13125//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13125//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13125//console
This message is automatically generated.
> ModifyTable increasing the region replica count should also auto-setup RRRE
> ---
>
> Key: HBASE-13169
> URL: https://issues.apache.org/jira/browse/HBASE-13169
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.1.0
>
> Attachments: hbase-13169_v1.patch
>
>
> This is a follow up to Async WAL replication jira (HBASE-11568). In
> HBASE-11568 we setup replication automatically in create table if the
> configuration is enabled.
> We should do the same in case a table with region replication = 1 is
> modified, and region replication is set to >1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12706) Support multiple port numbers in ZK quorum string
[
https://issues.apache.org/jira/browse/HBASE-12706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351360#comment-14351360
]
Hudson commented on HBASE-12706:
SUCCESS: Integrated in HBase-TRUNK #6220 (See
[https://builds.apache.org/job/HBase-TRUNK/6220/])
HBASE-12706 Support multiple port numbers in ZK quorum string (Stephen Yuan
Jiang) (enis: rev 95fc53f1056238e0b5f65df7f3015f2aec3d0a20)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/zookeeper/TestHQuorumPeer.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMasterCommandLine.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/zookeeper/TestRecoverableZooKeeper.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperMainServer.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperMainServer.java
> Support multiple port numbers in ZK quorum string
> -
>
> Key: HBASE-12706
> URL: https://issues.apache.org/jira/browse/HBASE-12706
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Stephen Yuan Jiang
>Assignee: Stephen Yuan Jiang
>Priority: Critical
> Fix For: 2.0.0, 1.1.0
>
> Attachments: HBASE-12706.v1-branch-1.patch,
> HBASE-12706.v2-branch-1.patch, HBASE-12706.v2-master.patch,
> HBASE-12706.v3-branch-1.patch, HBASE-12706.v3-master.patch,
> hbase-12706-alternate.patch
>
>
> HBase does not allow the zk quorum string to contain port numbers in this
> format:
> {noformat}
> hostname1:port1,hostname2:port2,hostname3:port3
> {noformat}
> Instead it expects the string to be in this format:
> {noformat}
> hostname1,hostname2,hostname3:port3
> {noformat}
> And port 3 is used for all the client ports. We should flex the parsing so
> that both forms are accepted.
> A sample exception:
> {code}
> java.io.IOException: Cluster key passed
> host1:2181,host2:2181,host3:2181,host4:2181,host5:2181:2181:/hbase is
> invalid, the format should
> be:hbase.zookeeper.quorum:hbase.zookeeper.client.port:zookeeper.znode.parent
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.transformClusterKey(ZKUtil.java:403)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.applyClusterKeyToConf(ZKUtil.java:386)
> at
> org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.getPeerConf(ReplicationPeersZKImpl.java:304)
> at
> org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.createPeer(ReplicationPeersZKImpl.java:435)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12466) Document visibility scan label generator usage and behavior
[ https://issues.apache.org/jira/browse/HBASE-12466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351357#comment-14351357 ] Jerry He commented on HBASE-12466: -- Hi, Misty bq. link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/security/visibility/VisibilityClient.htmlsgetAuths%28org.apache.hadoop.conf.Configuration,%20java.lang.String%29[setAuths()] method Inside the link, the method should be VisibilityClient.htmls *setAuths*, with the url for setAuths http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/security/visibility/VisibilityClient.html#setAuths(org.apache.hadoop.conf.Configuration,%20java.lang.String[],%20java.lang.String) Same problem for HBASE-12468 > Document visibility scan label generator usage and behavior > --- > > Key: HBASE-12466 > URL: https://issues.apache.org/jira/browse/HBASE-12466 > Project: HBase > Issue Type: Task > Components: documentation, security >Reporter: Jerry He >Assignee: Misty Stanley-Jones > Attachments: HBASE-12466-v1.patch, HBASE-12466.patch > > > Document the SLGs we provide and their behavior. > Document the out-of-box default behavior. > Document how to configure SLG if non-default behavior is desired. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12706) Support multiple port numbers in ZK quorum string
[
https://issues.apache.org/jira/browse/HBASE-12706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351352#comment-14351352
]
Hudson commented on HBASE-12706:
FAILURE: Integrated in HBase-1.1 #257 (See
[https://builds.apache.org/job/HBase-1.1/257/])
HBASE-12706 Support multiple port numbers in ZK quorum string (Stephen Yuan
Jiang) (enis: rev 5619f20e1af9958e5bb45a8f570b4d86768341fc)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/zookeeper/TestRecoverableZooKeeper.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/zookeeper/TestHQuorumPeer.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperMainServer.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMasterCommandLine.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperMainServer.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java
> Support multiple port numbers in ZK quorum string
> -
>
> Key: HBASE-12706
> URL: https://issues.apache.org/jira/browse/HBASE-12706
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Stephen Yuan Jiang
>Assignee: Stephen Yuan Jiang
>Priority: Critical
> Fix For: 2.0.0, 1.1.0
>
> Attachments: HBASE-12706.v1-branch-1.patch,
> HBASE-12706.v2-branch-1.patch, HBASE-12706.v2-master.patch,
> HBASE-12706.v3-branch-1.patch, HBASE-12706.v3-master.patch,
> hbase-12706-alternate.patch
>
>
> HBase does not allow the zk quorum string to contain port numbers in this
> format:
> {noformat}
> hostname1:port1,hostname2:port2,hostname3:port3
> {noformat}
> Instead it expects the string to be in this format:
> {noformat}
> hostname1,hostname2,hostname3:port3
> {noformat}
> And port 3 is used for all the client ports. We should flex the parsing so
> that both forms are accepted.
> A sample exception:
> {code}
> java.io.IOException: Cluster key passed
> host1:2181,host2:2181,host3:2181,host4:2181,host5:2181:2181:/hbase is
> invalid, the format should
> be:hbase.zookeeper.quorum:hbase.zookeeper.client.port:zookeeper.znode.parent
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.transformClusterKey(ZKUtil.java:403)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.applyClusterKeyToConf(ZKUtil.java:386)
> at
> org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.getPeerConf(ReplicationPeersZKImpl.java:304)
> at
> org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.createPeer(ReplicationPeersZKImpl.java:435)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13121) Async wal replication for region replicas and dist log replay does not work together
[
https://issues.apache.org/jira/browse/HBASE-13121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351340#comment-14351340
]
Hadoop QA commented on HBASE-13121:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703181/hbase-13121_v2.patch
against master branch at commit 5845f72ad60a7a7f5a04c01b68d841c78c38f79a.
ATTACHMENT ID: 12703181
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13124//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13124//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13124//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13124//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13124//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13124//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13124//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13124//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13124//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13124//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13124//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13124//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13124//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13124//console
This message is automatically generated.
> Async wal replication for region replicas and dist log replay does not work
> together
>
>
> Key: HBASE-13121
> URL: https://issues.apache.org/jira/browse/HBASE-13121
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.1.0
>
> Attachments: hbase-13121_v1.patch, hbase-13121_v2.patch
>
>
> We had not tested dist log replay while testing async wal replication for
> region replicas. There seems to be a couple of issues, but fixable.
> The distinction for dist log replay is that, the region will be opened for
> recovery and regular writes when a primary fails over. This causes the region
> open event marker to be written to WAL, but at this time, the region actually
> does not contain all the edits flushed (since it is still recovering). If
> secondary regions see this event, and picks up all the files in the region
> open event marker, then they can drop edits.
> The solution is:
> - Only write the region open event marker to WAL when region is out of
> recovering mode.
> - Force a flush out of recovering mode. This ensures that
[jira] [Commented] (HBASE-13170) Allow block cache to be external
[
https://issues.apache.org/jira/browse/HBASE-13170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351339#comment-14351339
]
Elliott Clark commented on HBASE-13170:
---
{code}
18:32:02 elliott@elliott-mbp ~ echo stats | nc 0 11211 | grep get_
STAT get_hits 584320
STAT get_misses 72526
{code}
For example here's a run of PE randomRead after restarting the local mode
instance.
> Allow block cache to be external
>
>
> Key: HBASE-13170
> URL: https://issues.apache.org/jira/browse/HBASE-13170
> Project: HBase
> Issue Type: New Feature
> Components: io
>Affects Versions: 2.0.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 1.1.0
>
>
> Allow an external service to provide the block cache. This has the nice
> property of allowing failover/upgrades to happen without causing a fully cold
> cache.
> Additionally this allows read replicas to share some of the same memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13164) Update TestUsersOperationsWithSecureHadoop to use MiniKdc
[ https://issues.apache.org/jira/browse/HBASE-13164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-13164: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Update TestUsersOperationsWithSecureHadoop to use MiniKdc > - > > Key: HBASE-13164 > URL: https://issues.apache.org/jira/browse/HBASE-13164 > Project: HBase > Issue Type: Sub-task > Components: security, test >Reporter: zhangduo >Assignee: zhangduo > Fix For: 2.0.0, 1.1.0 > > Attachments: HBASE-13164-branch-1.patch, HBASE-13164.patch > > > There is only one unit test besides TestSecureRPC which requires a kerberos > environment. Update it to use MiniKdc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13170) Allow block cache to be external
[ https://issues.apache.org/jira/browse/HBASE-13170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-13170: -- Description: Allow an external service to provide the block cache. This has the nice property of allowing failover/upgrades to happen without causing a fully cold cache. Additionally this allows read replicas to share some of the same memory. > Allow block cache to be external > > > Key: HBASE-13170 > URL: https://issues.apache.org/jira/browse/HBASE-13170 > Project: HBase > Issue Type: New Feature > Components: io >Affects Versions: 2.0.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 1.1.0 > > > Allow an external service to provide the block cache. This has the nice > property of allowing failover/upgrades to happen without causing a fully cold > cache. > Additionally this allows read replicas to share some of the same memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13170) Allow block cache to be external
[ https://issues.apache.org/jira/browse/HBASE-13170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351334#comment-14351334 ] Elliott Clark commented on HBASE-13170: --- https://reviews.facebook.net/D34635 It's no where near finished or even ready to really think about in it's current form. However it's an interesting thought > Allow block cache to be external > > > Key: HBASE-13170 > URL: https://issues.apache.org/jira/browse/HBASE-13170 > Project: HBase > Issue Type: New Feature > Components: io >Affects Versions: 2.0.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 1.1.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12405) WAL accounting by Store
[ https://issues.apache.org/jira/browse/HBASE-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangduo updated HBASE-12405: - Attachment: HBASE-12405_5.patch Edit comments for LastSequenceId > WAL accounting by Store > --- > > Key: HBASE-12405 > URL: https://issues.apache.org/jira/browse/HBASE-12405 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 2.0.0, 1.1.0 >Reporter: zhangduo >Assignee: zhangduo > Fix For: 2.0.0, 1.1.0 > > Attachments: HBASE-12405.patch, HBASE-12405_1.patch, > HBASE-12405_2.patch, HBASE-12405_3.patch, HBASE-12405_4.patch, > HBASE-12405_5.patch > > > HBASE-10201 has made flush decisions per Store, but has not done enough work > on HLog, so there are two problems: > 1. We record minSeqId both in HRegion and FSHLog, which is a duplication. > 2. There maybe holes in WAL accounting. > For example, assume family A with sequence id 1 and 3, family B with > seqId 2. If we flush family A, we can only record that WAL before sequence id > 1 can be removed safely. If we do a replay at this point, sequence id 3 will > also be replayed which is unnecessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13170) Allow block cache to be external
[ https://issues.apache.org/jira/browse/HBASE-13170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351323#comment-14351323 ] Nick Dimiduk commented on HBASE-13170: -- bq. Allow block cache to be external Go on... :) > Allow block cache to be external > > > Key: HBASE-13170 > URL: https://issues.apache.org/jira/browse/HBASE-13170 > Project: HBase > Issue Type: New Feature > Components: io >Affects Versions: 2.0.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 1.1.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12405) WAL accounting by Store
[ https://issues.apache.org/jira/browse/HBASE-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351321#comment-14351321 ] zhangduo commented on HBASE-12405: -- [~stack] I think the test plan is good. And I'm a little worried about 'HBASE-11569 Flush / Compaction handling from secondary region replicas'. HRegion.replayWALFlushCommitMarker also plays with maxFlushedSeqId, and they just use flush sequence number as maxFlushedSeqId. Perhaps this is not right when per column family flush is turned on since we may not always flush all stores. Thanks~ > WAL accounting by Store > --- > > Key: HBASE-12405 > URL: https://issues.apache.org/jira/browse/HBASE-12405 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 2.0.0, 1.1.0 >Reporter: zhangduo >Assignee: zhangduo > Fix For: 2.0.0, 1.1.0 > > Attachments: HBASE-12405.patch, HBASE-12405_1.patch, > HBASE-12405_2.patch, HBASE-12405_3.patch, HBASE-12405_4.patch > > > HBASE-10201 has made flush decisions per Store, but has not done enough work > on HLog, so there are two problems: > 1. We record minSeqId both in HRegion and FSHLog, which is a duplication. > 2. There maybe holes in WAL accounting. > For example, assume family A with sequence id 1 and 3, family B with > seqId 2. If we flush family A, we can only record that WAL before sequence id > 1 can be removed safely. If we do a replay at this point, sequence id 3 will > also be replayed which is unnecessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12586) Task 6 & 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection
[
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-12586:
Status: Open (was: Patch Available)
> Task 6 & 7 from HBASE-9117, delete all public HTable constructors and delete
> ConnectionManager#{delete,get}Connection
> --
>
> Key: HBASE-12586
> URL: https://issues.apache.org/jira/browse/HBASE-12586
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: Mikhail Antonov
> Labels: beginner, beginners
> Attachments: HBASE-12586.patch, HBASE-12586.patch
>
>
> Finish cleanup from HBASE-9117 removing old API. This issue covers tasks 6
> and 7 from the list here:
> https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
> To be done in master branch only.
> Marked as beginner task. The idea is straight-forward. It is just a lot of
> work going through all tests converting to use new API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12586) Task 6 & 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection
[
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-12586:
Status: Patch Available (was: Open)
> Task 6 & 7 from HBASE-9117, delete all public HTable constructors and delete
> ConnectionManager#{delete,get}Connection
> --
>
> Key: HBASE-12586
> URL: https://issues.apache.org/jira/browse/HBASE-12586
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: Mikhail Antonov
> Labels: beginner, beginners
> Attachments: HBASE-12586.patch, HBASE-12586.patch
>
>
> Finish cleanup from HBASE-9117 removing old API. This issue covers tasks 6
> and 7 from the list here:
> https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
> To be done in master branch only.
> Marked as beginner task. The idea is straight-forward. It is just a lot of
> work going through all tests converting to use new API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12586) Task 6 & 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection
[
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-12586:
Attachment: HBASE-12586.patch
Incorrectly-exported patch, removed files didn't get included.. Updating.
> Task 6 & 7 from HBASE-9117, delete all public HTable constructors and delete
> ConnectionManager#{delete,get}Connection
> --
>
> Key: HBASE-12586
> URL: https://issues.apache.org/jira/browse/HBASE-12586
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: Mikhail Antonov
> Labels: beginner, beginners
> Attachments: HBASE-12586.patch, HBASE-12586.patch
>
>
> Finish cleanup from HBASE-9117 removing old API. This issue covers tasks 6
> and 7 from the list here:
> https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
> To be done in master branch only.
> Marked as beginner task. The idea is straight-forward. It is just a lot of
> work going through all tests converting to use new API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12586) Task 6 & 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection
[
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351311#comment-14351311
]
Hadoop QA commented on HBASE-12586:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703199/HBASE-12586.patch
against master branch at commit 95fc53f1056238e0b5f65df7f3015f2aec3d0a20.
ATTACHMENT ID: 12703199
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 27 new
or modified tests.
{color:red}-1 javac{color}. The patch appears to cause mvn compile goal to
fail with Hadoop version 2.4.1.
Compilation errors resume:
[ERROR] COMPILATION ERROR :
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableFactory.java:[41,14]
no suitable constructor found for
HTable(org.apache.hadoop.conf.Configuration,org.apache.hadoop.hbase.TableName)
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on
project hbase-client: Compilation failure
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableFactory.java:[41,14]
no suitable constructor found for
HTable(org.apache.hadoop.conf.Configuration,org.apache.hadoop.hbase.TableName)
[ERROR] constructor
org.apache.hadoop.hbase.client.HTable.HTable(org.apache.hadoop.hbase.client.ClusterConnection,org.apache.hadoop.hbase.client.BufferedMutatorParams)
is not applicable
[ERROR] (actual argument org.apache.hadoop.conf.Configuration cannot be
converted to org.apache.hadoop.hbase.client.ClusterConnection by method
invocation conversion)
[ERROR] constructor
org.apache.hadoop.hbase.client.HTable.HTable(org.apache.hadoop.hbase.TableName,org.apache.hadoop.hbase.client.ClusterConnection,org.apache.hadoop.hbase.client.TableConfiguration,org.apache.hadoop.hbase.client.RpcRetryingCallerFactory,org.apache.hadoop.hbase.ipc.RpcControllerFactory,java.util.concurrent.ExecutorService)
is not applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hbase-client
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13126//console
This message is automatically generated.
> Task 6 & 7 from HBASE-9117, delete all public HTable constructors and delete
> ConnectionManager#{delete,get}Connection
> --
>
> Key: HBASE-12586
> URL: https://issues.apache.org/jira/browse/HBASE-12586
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: Mikhail Antonov
> Labels: beginner, beginners
> Attachments: HBASE-12586.patch
>
>
> Finish cleanup from HBASE-9117 removing old API. This issue covers tasks 6
> and 7 from the list here:
> https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
> To be done in master branch only.
> Marked as beginner task. The idea is straight-forward. It is just a lot of
> work going through all tests converting to use new API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351310#comment-14351310 ] Hudson commented on HBASE-13082: SUCCESS: Integrated in HBase-TRUNK #6219 (See [https://builds.apache.org/job/HBase-TRUNK/6219/]) Revert "HBASE-13082 Coarsen StoreScanner locks to RegionScanner." (larsh: rev 5845f72ad60a7a7f5a04c01b68d841c78c38f79a) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedStoreScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java > Coarsen StoreScanner locks to RegionScanner > --- > > Key: HBASE-13082 > URL: https://issues.apache.org/jira/browse/HBASE-13082 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 2.0.0, 1.1.0 > > Attachments: 13082-test.txt, 13082-v2.txt, 13082.txt, 13082.txt, > gc.png, gc.png, gc.png, hits.png, next.png, next.png > > > Continuing where HBASE-10015 left of. > We can avoid locking (and memory fencing) inside StoreScanner by deferring to > the lock already held by the RegionScanner. > In tests this shows quite a scan improvement and reduced CPU (the fences make > the cores wait for memory fetches). > There are some drawbacks too: > * All calls to RegionScanner need to be remain synchronized > * Implementors of coprocessors need to be diligent in following the locking > contract. For example Phoenix does not lock RegionScanner.nextRaw() and > required in the documentation (not picking on Phoenix, this one is my fault > as I told them it's OK) > * possible starving of flushes and compaction with heavy read load. > RegionScanner operations would keep getting the locks and the > flushes/compactions would not be able finalize the set of files. > I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13170) Allow block cache to be external
Elliott Clark created HBASE-13170: - Summary: Allow block cache to be external Key: HBASE-13170 URL: https://issues.apache.org/jira/browse/HBASE-13170 Project: HBase Issue Type: New Feature Components: io Affects Versions: 2.0.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 1.1.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351304#comment-14351304 ] Hudson commented on HBASE-13142: FAILURE: Integrated in HBase-1.1 #256 (See [https://builds.apache.org/job/HBase-1.1/256/]) HBASE-13142 [PERF] Reuse the IPCUtil#buildCellBlock buffer: ADDENDUM -- Fix build breakage (stack: rev 880215b728c7f5086705e37a0815074d54280a38) * hbase-common/src/test/java/org/apache/hadoop/hbase/io/TestBoundedByteBufferPool.java > [PERF] Reuse the IPCUtil#buildCellBlock buffer > -- > > Key: HBASE-13142 > URL: https://issues.apache.org/jira/browse/HBASE-13142 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Assignee: stack > Labels: beginner > Fix For: 2.0.0, 1.1.0 > > Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.0.98.txt, > 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, > hits.png, net.png, traces.2.svg, traces.svg > > > Running some scan profiling, flight recorder was mildly fingering resize of > the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It > was half-hearted blaming it for a few hundreds of ms over a five minute > sampling with a few tens of instances showing. > I tried then w/ flamegraph/lightweight profiler and this reported the buffer > allocations taking 22% of our total CPU. See attachment trace.svg. > I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and > indeed every allocation was doing a resize from initial allocation of 16k -- > the default up to 220k (this test returns ten randomly sized rows zipfian > sized between 0 and 8k). > Upping the allocation to 220k meant we now avoided the resize but the initial > allocation was now blamed for 10% of allocations (see trace.2.svg attached). > Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13169) ModifyTable increasing the region replica count should also auto-setup RRRE
[ https://issues.apache.org/jira/browse/HBASE-13169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351303#comment-14351303 ] Mikhail Antonov commented on HBASE-13169: - Looks good to me. Wondering if any special handling is needed if modification goes in "opposite" direction, say from 3 to 1 num of reps? > ModifyTable increasing the region replica count should also auto-setup RRRE > --- > > Key: HBASE-13169 > URL: https://issues.apache.org/jira/browse/HBASE-13169 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.1.0 > > Attachments: hbase-13169_v1.patch > > > This is a follow up to Async WAL replication jira (HBASE-11568). In > HBASE-11568 we setup replication automatically in create table if the > configuration is enabled. > We should do the same in case a table with region replication = 1 is > modified, and region replication is set to >1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12990) MetaScanner should be replaced by MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12990: -- Attachment: HBASE-12990.v7.patch +1 on this. You going to commit it [~octo47] (trying hadoopqa again in case it rotted meantime). You think it should go in master and branch-1.1? > MetaScanner should be replaced by MetaTableAccessor > --- > > Key: HBASE-12990 > URL: https://issues.apache.org/jira/browse/HBASE-12990 > Project: HBase > Issue Type: Improvement > Components: Client >Affects Versions: 2.0.0 >Reporter: Andrey Stepachev >Assignee: Andrey Stepachev > Attachments: HBASE-12990.patch, HBASE-12990.v2.patch, > HBASE-12990.v3.patch, HBASE-12990.v4.patch, HBASE-12990.v5.patch, > HBASE-12990.v5.patch, HBASE-12990.v5.patch, HBASE-12990.v6.patch, > HBASE-12990.v7.patch, HBASE-12990.v7.patch > > > MetaScanner and MetaTableAccessor do similar things, but seems they tend to > diverge. Let's have only one thing to enquiry META. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12586) Task 6 & 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection
[
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-12586:
Status: Patch Available (was: In Progress)
> Task 6 & 7 from HBASE-9117, delete all public HTable constructors and delete
> ConnectionManager#{delete,get}Connection
> --
>
> Key: HBASE-12586
> URL: https://issues.apache.org/jira/browse/HBASE-12586
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: Mikhail Antonov
> Labels: beginner, beginners
> Attachments: HBASE-12586.patch
>
>
> Finish cleanup from HBASE-9117 removing old API. This issue covers tasks 6
> and 7 from the list here:
> https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
> To be done in master branch only.
> Marked as beginner task. The idea is straight-forward. It is just a lot of
> work going through all tests converting to use new API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12586) Task 6 & 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#{delete,get}Connection
[
https://issues.apache.org/jira/browse/HBASE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-12586:
Attachment: HBASE-12586.patch
Draft patch.
> Task 6 & 7 from HBASE-9117, delete all public HTable constructors and delete
> ConnectionManager#{delete,get}Connection
> --
>
> Key: HBASE-12586
> URL: https://issues.apache.org/jira/browse/HBASE-12586
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: Mikhail Antonov
> Labels: beginner, beginners
> Attachments: HBASE-12586.patch
>
>
> Finish cleanup from HBASE-9117 removing old API. This issue covers tasks 6
> and 7 from the list here:
> https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
> To be done in master branch only.
> Marked as beginner task. The idea is straight-forward. It is just a lot of
> work going through all tests converting to use new API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12405) WAL accounting by Store
[ https://issues.apache.org/jira/browse/HBASE-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351289#comment-14351289 ] stack commented on HBASE-12405: --- So, I'll commit in a day or so. [~Apache9] So, I want to do a ITBLL that exercises this new functionality. We talked about what it would look like, of how I'd have to write 3 columns; one small, one medium, and then a large one. I suppose the ITBLL column could be the middle-sized one? The aim is that by hbase-1.1, this is on and as tested as anything is around these parts. > WAL accounting by Store > --- > > Key: HBASE-12405 > URL: https://issues.apache.org/jira/browse/HBASE-12405 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 2.0.0, 1.1.0 >Reporter: zhangduo >Assignee: zhangduo > Fix For: 2.0.0, 1.1.0 > > Attachments: HBASE-12405.patch, HBASE-12405_1.patch, > HBASE-12405_2.patch, HBASE-12405_3.patch, HBASE-12405_4.patch > > > HBASE-10201 has made flush decisions per Store, but has not done enough work > on HLog, so there are two problems: > 1. We record minSeqId both in HRegion and FSHLog, which is a duplication. > 2. There maybe holes in WAL accounting. > For example, assume family A with sequence id 1 and 3, family B with > seqId 2. If we flush family A, we can only record that WAL before sequence id > 1 can be removed safely. If we do a replay at this point, sequence id 3 will > also be replayed which is unnecessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12405) WAL accounting by Store
[ https://issues.apache.org/jira/browse/HBASE-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351277#comment-14351277 ] stack commented on HBASE-12405: --- Porting my +1 from RB here. > WAL accounting by Store > --- > > Key: HBASE-12405 > URL: https://issues.apache.org/jira/browse/HBASE-12405 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 2.0.0, 1.1.0 >Reporter: zhangduo >Assignee: zhangduo > Fix For: 2.0.0, 1.1.0 > > Attachments: HBASE-12405.patch, HBASE-12405_1.patch, > HBASE-12405_2.patch, HBASE-12405_3.patch, HBASE-12405_4.patch > > > HBASE-10201 has made flush decisions per Store, but has not done enough work > on HLog, so there are two problems: > 1. We record minSeqId both in HRegion and FSHLog, which is a duplication. > 2. There maybe holes in WAL accounting. > For example, assume family A with sequence id 1 and 3, family B with > seqId 2. If we flush family A, we can only record that WAL before sequence id > 1 can be removed safely. If we do a replay at this point, sequence id 3 will > also be replayed which is unnecessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13169) ModifyTable increasing the region replica count should also auto-setup RRRE
[ https://issues.apache.org/jira/browse/HBASE-13169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-13169: -- Status: Patch Available (was: Open) > ModifyTable increasing the region replica count should also auto-setup RRRE > --- > > Key: HBASE-13169 > URL: https://issues.apache.org/jira/browse/HBASE-13169 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.1.0 > > Attachments: hbase-13169_v1.patch > > > This is a follow up to Async WAL replication jira (HBASE-11568). In > HBASE-11568 we setup replication automatically in create table if the > configuration is enabled. > We should do the same in case a table with region replication = 1 is > modified, and region replication is set to >1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13169) ModifyTable increasing the region replica count should also auto-setup RRRE
[ https://issues.apache.org/jira/browse/HBASE-13169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-13169: -- Attachment: hbase-13169_v1.patch Attaching simple patch. > ModifyTable increasing the region replica count should also auto-setup RRRE > --- > > Key: HBASE-13169 > URL: https://issues.apache.org/jira/browse/HBASE-13169 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.1.0 > > Attachments: hbase-13169_v1.patch > > > This is a follow up to Async WAL replication jira (HBASE-11568). In > HBASE-11568 we setup replication automatically in create table if the > configuration is enabled. > We should do the same in case a table with region replication = 1 is > modified, and region replication is set to >1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13169) ModifyTable increasing the region replica count should also auto-setup RRRE
Enis Soztutar created HBASE-13169: - Summary: ModifyTable increasing the region replica count should also auto-setup RRRE Key: HBASE-13169 URL: https://issues.apache.org/jira/browse/HBASE-13169 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 2.0.0, 1.1.0 This is a follow up to Async WAL replication jira (HBASE-11568). In HBASE-11568 we setup replication automatically in create table if the configuration is enabled. We should do the same in case a table with region replication = 1 is modified, and region replication is set to >1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13122) Improve efficiency for return codes of some filters
[ https://issues.apache.org/jira/browse/HBASE-13122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351273#comment-14351273 ] stack commented on HBASE-13122: --- What [~apurtell] said. > Improve efficiency for return codes of some filters > --- > > Key: HBASE-13122 > URL: https://issues.apache.org/jira/browse/HBASE-13122 > Project: HBase > Issue Type: Improvement > Components: Filters >Affects Versions: 0.94.24, 1.0.1, 0.98.10.1 >Reporter: Shuaifeng Zhou > Fix For: 2.0.0, 1.1.0 > > Attachments: 13122-master.patch, 13122-master.patch, > 13122-master.patch, 13122.patch > > > ColumnRangeFilter: > when minColumnInclusive is false, it means all the cells at the current > row&column not fit the condition, so it should skip to next column, return > code should be NEXT_COL, not SKIP. > FamilyFilter is the similar sitution. > Currently, SKIP will not causing error, but not efficent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12706) Support multiple port numbers in ZK quorum string
[
https://issues.apache.org/jira/browse/HBASE-12706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-12706:
--
Resolution: Fixed
Release Note: hbase.zookeeper.quorum configuration now allows servers
together with client ports consistent with the way Zookeeper java client
accepts the quorum string. In this case, using hbase.zookeeper.clientPort is
not needed. eg.
hbase.zookeeper.quorum=myserver1:2181,myserver2:2,myserver3:3
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I have pushed this to 1.1 and master. Thanks Stephen for the patch and bearing
with me through the reviews.
> Support multiple port numbers in ZK quorum string
> -
>
> Key: HBASE-12706
> URL: https://issues.apache.org/jira/browse/HBASE-12706
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Stephen Yuan Jiang
>Assignee: Stephen Yuan Jiang
>Priority: Critical
> Fix For: 2.0.0, 1.1.0
>
> Attachments: HBASE-12706.v1-branch-1.patch,
> HBASE-12706.v2-branch-1.patch, HBASE-12706.v2-master.patch,
> HBASE-12706.v3-branch-1.patch, HBASE-12706.v3-master.patch,
> hbase-12706-alternate.patch
>
>
> HBase does not allow the zk quorum string to contain port numbers in this
> format:
> {noformat}
> hostname1:port1,hostname2:port2,hostname3:port3
> {noformat}
> Instead it expects the string to be in this format:
> {noformat}
> hostname1,hostname2,hostname3:port3
> {noformat}
> And port 3 is used for all the client ports. We should flex the parsing so
> that both forms are accepted.
> A sample exception:
> {code}
> java.io.IOException: Cluster key passed
> host1:2181,host2:2181,host3:2181,host4:2181,host5:2181:2181:/hbase is
> invalid, the format should
> be:hbase.zookeeper.quorum:hbase.zookeeper.client.port:zookeeper.znode.parent
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.transformClusterKey(ZKUtil.java:403)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.applyClusterKeyToConf(ZKUtil.java:386)
> at
> org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.getPeerConf(ReplicationPeersZKImpl.java:304)
> at
> org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.createPeer(ReplicationPeersZKImpl.java:435)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-13142: -- Attachment: 13142v5.0.98.txt Here is 0.98 patch. Lets see what hadoopqa thinks of it. > [PERF] Reuse the IPCUtil#buildCellBlock buffer > -- > > Key: HBASE-13142 > URL: https://issues.apache.org/jira/browse/HBASE-13142 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Assignee: stack > Labels: beginner > Fix For: 2.0.0, 1.1.0 > > Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.0.98.txt, > 13142v5.txt, 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, > hits.png, net.png, traces.2.svg, traces.svg > > > Running some scan profiling, flight recorder was mildly fingering resize of > the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It > was half-hearted blaming it for a few hundreds of ms over a five minute > sampling with a few tens of instances showing. > I tried then w/ flamegraph/lightweight profiler and this reported the buffer > allocations taking 22% of our total CPU. See attachment trace.svg. > I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and > indeed every allocation was doing a resize from initial allocation of 16k -- > the default up to 220k (this test returns ten randomly sized rows zipfian > sized between 0 and 8k). > Upping the allocation to 220k meant we now avoided the resize but the initial > allocation was now blamed for 10% of allocations (see trace.2.svg attached). > Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13063) Allow to turn off memstore replication for region replicas
[
https://issues.apache.org/jira/browse/HBASE-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351226#comment-14351226
]
Enis Soztutar commented on HBASE-13063:
---
We can also document this in the book. There is a section for region replicas
already with all config options.
> Allow to turn off memstore replication for region replicas
> --
>
> Key: HBASE-13063
> URL: https://issues.apache.org/jira/browse/HBASE-13063
> Project: HBase
> Issue Type: New Feature
> Components: regionserver, Replication
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 2.0.0, 1.1.0
>
> Attachments: HBASE-13063-v0.patch, HBASE-13063-v1-branch-1.patch,
> HBASE-13063-v1.patch, HBASE-13063-v1.patch, HBASE-13063-v2.patch,
> HBASE-13063-v2.patch
>
>
> HBASE-11568 allows to use replication to send wal edits from the primary to
> the replicas.
> sometimes the memstore replication is not required, so it will be nice have a
> flag to disable it.
> the result will be more or less the same to what we have in phase-1, but with
> the row-level consistency provided by the "flush" edit transfered via
> replication to refresh the hfiles.
> create 't1', 'f', {REGION_REPLICATION => 2, REGION_MEMSTORE_REPLICATION =>
> false}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13121) Async wal replication for region replicas and dist log replay does not work together
[ https://issues.apache.org/jira/browse/HBASE-13121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-13121: -- Status: Patch Available (was: Open) > Async wal replication for region replicas and dist log replay does not work > together > > > Key: HBASE-13121 > URL: https://issues.apache.org/jira/browse/HBASE-13121 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.1.0 > > Attachments: hbase-13121_v1.patch, hbase-13121_v2.patch > > > We had not tested dist log replay while testing async wal replication for > region replicas. There seems to be a couple of issues, but fixable. > The distinction for dist log replay is that, the region will be opened for > recovery and regular writes when a primary fails over. This causes the region > open event marker to be written to WAL, but at this time, the region actually > does not contain all the edits flushed (since it is still recovering). If > secondary regions see this event, and picks up all the files in the region > open event marker, then they can drop edits. > The solution is: > - Only write the region open event marker to WAL when region is out of > recovering mode. > - Force a flush out of recovering mode. This ensures that all data is force > flushed in this case. Before the region open event marker is written, we > guarantee that all data in the region is flushed, so the list of files in the > event marker is complete. > - Edits coming from recovery are re-written to WAL when recovery is in > action. These edits will have a larger seqId then their "original" seqId. If > this is the case, we do not replicate these edits to the secondary replicas. > Since the dist log replay recovers edits out of order (coming from parallel > replays from WAL file split tasks), this ensures that TIMELINE consistency is > respected and edits are not seen out of order in secondaries. These edits are > seen from secondaries via the forced flush event. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13121) Async wal replication for region replicas and dist log replay does not work together
[ https://issues.apache.org/jira/browse/HBASE-13121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-13121: -- Attachment: hbase-13121_v2.patch v2 patch addressed Jeffrey's comments, and adds one more simple test. > Async wal replication for region replicas and dist log replay does not work > together > > > Key: HBASE-13121 > URL: https://issues.apache.org/jira/browse/HBASE-13121 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.1.0 > > Attachments: hbase-13121_v1.patch, hbase-13121_v2.patch > > > We had not tested dist log replay while testing async wal replication for > region replicas. There seems to be a couple of issues, but fixable. > The distinction for dist log replay is that, the region will be opened for > recovery and regular writes when a primary fails over. This causes the region > open event marker to be written to WAL, but at this time, the region actually > does not contain all the edits flushed (since it is still recovering). If > secondary regions see this event, and picks up all the files in the region > open event marker, then they can drop edits. > The solution is: > - Only write the region open event marker to WAL when region is out of > recovering mode. > - Force a flush out of recovering mode. This ensures that all data is force > flushed in this case. Before the region open event marker is written, we > guarantee that all data in the region is flushed, so the list of files in the > event marker is complete. > - Edits coming from recovery are re-written to WAL when recovery is in > action. These edits will have a larger seqId then their "original" seqId. If > this is the case, we do not replicate these edits to the secondary replicas. > Since the dist log replay recovers edits out of order (coming from parallel > replays from WAL file split tasks), this ensures that TIMELINE consistency is > respected and edits are not seen out of order in secondaries. These edits are > seen from secondaries via the forced flush event. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13168) Backport HBASE-12590 "A solution for data skew in HBase-Mapreduce Job"
Nick Dimiduk created HBASE-13168: Summary: Backport HBASE-12590 "A solution for data skew in HBase-Mapreduce Job" Key: HBASE-13168 URL: https://issues.apache.org/jira/browse/HBASE-13168 Project: HBase Issue Type: Improvement Components: mapreduce Reporter: Nick Dimiduk We should bring this back into active release branches. Seems like it shouldn't break any API compatibility, but should be checked more closely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13122) Improve efficiency for return codes of some filters
[ https://issues.apache.org/jira/browse/HBASE-13122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351215#comment-14351215 ] Andrew Purtell commented on HBASE-13122: The commit, then revert, then commit here could have been avoided by allowing more time for others to look at the issue. No fault of anyone, maybe give more time in the future. There's no rush. > Improve efficiency for return codes of some filters > --- > > Key: HBASE-13122 > URL: https://issues.apache.org/jira/browse/HBASE-13122 > Project: HBase > Issue Type: Improvement > Components: Filters >Affects Versions: 0.94.24, 1.0.1, 0.98.10.1 >Reporter: Shuaifeng Zhou > Fix For: 2.0.0, 1.1.0 > > Attachments: 13122-master.patch, 13122-master.patch, > 13122-master.patch, 13122.patch > > > ColumnRangeFilter: > when minColumnInclusive is false, it means all the cells at the current > row&column not fit the condition, so it should skip to next column, return > code should be NEXT_COL, not SKIP. > FamilyFilter is the similar sitution. > Currently, SKIP will not causing error, but not efficent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351213#comment-14351213 ] Hudson commented on HBASE-13142: FAILURE: Integrated in HBase-TRUNK #6218 (See [https://builds.apache.org/job/HBase-TRUNK/6218/]) HBASE-13142 [PERF] Reuse the IPCUtil#buildCellBlock buffer (stack: rev 9159c82fbf6bf0b31eba58095eeca33348e2c389) * hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferOutputStream.java * hbase-common/src/test/java/org/apache/hadoop/hbase/io/TestBoundedByteBufferPool.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestByteBufferOutputStream.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/BoundedByteBufferPool.java > [PERF] Reuse the IPCUtil#buildCellBlock buffer > -- > > Key: HBASE-13142 > URL: https://issues.apache.org/jira/browse/HBASE-13142 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Assignee: stack > Labels: beginner > Fix For: 2.0.0, 1.1.0 > > Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.txt, > 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, > net.png, traces.2.svg, traces.svg > > > Running some scan profiling, flight recorder was mildly fingering resize of > the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It > was half-hearted blaming it for a few hundreds of ms over a five minute > sampling with a few tens of instances showing. > I tried then w/ flamegraph/lightweight profiler and this reported the buffer > allocations taking 22% of our total CPU. See attachment trace.svg. > I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and > indeed every allocation was doing a resize from initial allocation of 16k -- > the default up to 220k (this test returns ten randomly sized rows zipfian > sized between 0 and 8k). > Upping the allocation to 220k meant we now avoided the resize but the initial > allocation was now blamed for 10% of allocations (see trace.2.svg attached). > Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12562) Handling memory pressure for secondary region replicas
[ https://issues.apache.org/jira/browse/HBASE-12562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351212#comment-14351212 ] Hudson commented on HBASE-12562: FAILURE: Integrated in HBase-TRUNK #6218 (See [https://builds.apache.org/job/HBase-TRUNK/6218/]) HBASE-12562 Handling memory pressure for secondary region replicas (enis: rev 4ac42a2f56b91ce864d1bcb04f1f9950e527aab1) * hbase-server/src/main/java/org/apache/hadoop/hbase/io/FileLink.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java > Handling memory pressure for secondary region replicas > -- > > Key: HBASE-12562 > URL: https://issues.apache.org/jira/browse/HBASE-12562 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.1.0 > > Attachments: hbase-12562_v1.patch, hbase-12562_v2.patch, > hbase-12562_v2.patch, hbase-12562_v2.patch, hbase-12562_v3.patch > > > This issue will track the implementation of how to handle the memory pressure > for secondary region replicas. Since the replicas cannot flush by themselves, > the region server might get blocked or cause extensive flushing for its > primary regions. The design doc attached at HBASE-11183 contains two possible > solutions that we can pursue. The first one is to not allow secondary region > replicas to not flush by themselves, but instead of needed allow them to > refresh their store files on demand (which possibly allows them to drop their > memstore snapshots or memstores). The second approach is to allow the > secondaries to flush to a temporary space. > Both have pros and cons, but for simplicity and to not cause extra write > amplification, we have implemented the first approach. More details can be > found in the design doc, but we can also discuss other options here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351209#comment-14351209 ] Andrew Purtell commented on HBASE-13142: +1 for 0.98 if you're up for a backport [~stack] > [PERF] Reuse the IPCUtil#buildCellBlock buffer > -- > > Key: HBASE-13142 > URL: https://issues.apache.org/jira/browse/HBASE-13142 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Assignee: stack > Labels: beginner > Fix For: 2.0.0, 1.1.0 > > Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.txt, > 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, > net.png, traces.2.svg, traces.svg > > > Running some scan profiling, flight recorder was mildly fingering resize of > the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It > was half-hearted blaming it for a few hundreds of ms over a five minute > sampling with a few tens of instances showing. > I tried then w/ flamegraph/lightweight profiler and this reported the buffer > allocations taking 22% of our total CPU. See attachment trace.svg. > I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and > indeed every allocation was doing a resize from initial allocation of 16k -- > the default up to 220k (this test returns ten randomly sized rows zipfian > sized between 0 and 8k). > Upping the allocation to 220k meant we now avoided the resize but the initial > allocation was now blamed for 10% of allocations (see trace.2.svg attached). > Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12466) Document visibility scan label generator usage and behavior
[ https://issues.apache.org/jira/browse/HBASE-12466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351203#comment-14351203 ] Andrew Purtell commented on HBASE-12466: lgtm, commit unless [~jinghe] has more comments I'd say > Document visibility scan label generator usage and behavior > --- > > Key: HBASE-12466 > URL: https://issues.apache.org/jira/browse/HBASE-12466 > Project: HBase > Issue Type: Task > Components: documentation, security >Reporter: Jerry He >Assignee: Misty Stanley-Jones > Attachments: HBASE-12466-v1.patch, HBASE-12466.patch > > > Document the SLGs we provide and their behavior. > Document the out-of-box default behavior. > Document how to configure SLG if non-default behavior is desired. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13121) Async wal replication for region replicas and dist log replay does not work together
[ https://issues.apache.org/jira/browse/HBASE-13121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351157#comment-14351157 ] Enis Soztutar commented on HBASE-13121: --- Thanks Jeffrey for review. bq. 1) Could you still set recovering to false and then submit the rest work into executor I thought about this. I think it is safer to do it this way. The expectation is that there will be executors available. bq. 2) openSeqNum in the following code may still use the old value? Good catch. I've modified it so that we get a new seqId and use it as the open seqId. Since during replay, we can accept more writes, the seqId maybe bigger than the original open seq id. > Async wal replication for region replicas and dist log replay does not work > together > > > Key: HBASE-13121 > URL: https://issues.apache.org/jira/browse/HBASE-13121 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.1.0 > > Attachments: hbase-13121_v1.patch > > > We had not tested dist log replay while testing async wal replication for > region replicas. There seems to be a couple of issues, but fixable. > The distinction for dist log replay is that, the region will be opened for > recovery and regular writes when a primary fails over. This causes the region > open event marker to be written to WAL, but at this time, the region actually > does not contain all the edits flushed (since it is still recovering). If > secondary regions see this event, and picks up all the files in the region > open event marker, then they can drop edits. > The solution is: > - Only write the region open event marker to WAL when region is out of > recovering mode. > - Force a flush out of recovering mode. This ensures that all data is force > flushed in this case. Before the region open event marker is written, we > guarantee that all data in the region is flushed, so the list of files in the > event marker is complete. > - Edits coming from recovery are re-written to WAL when recovery is in > action. These edits will have a larger seqId then their "original" seqId. If > this is the case, we do not replicate these edits to the secondary replicas. > Since the dist log replay recovers edits out of order (coming from parallel > replays from WAL file split tasks), this ensures that TIMELINE consistency is > respected and edits are not seen out of order in secondaries. These edits are > seen from secondaries via the forced flush event. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351127#comment-14351127 ] Hudson commented on HBASE-13142: FAILURE: Integrated in HBase-1.1 #255 (See [https://builds.apache.org/job/HBase-1.1/255/]) HBASE-13142 [PERF] Reuse the IPCUtil#buildCellBlock buffer (stack: rev 3e3276d7fa2d20815c35b72b386efb5796db7939) * hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestByteBufferOutputStream.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferOutputStream.java * hbase-common/src/test/java/org/apache/hadoop/hbase/io/TestBoundedByteBufferPool.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/BoundedByteBufferPool.java > [PERF] Reuse the IPCUtil#buildCellBlock buffer > -- > > Key: HBASE-13142 > URL: https://issues.apache.org/jira/browse/HBASE-13142 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Assignee: stack > Labels: beginner > Fix For: 2.0.0, 1.1.0 > > Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.txt, > 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, > net.png, traces.2.svg, traces.svg > > > Running some scan profiling, flight recorder was mildly fingering resize of > the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It > was half-hearted blaming it for a few hundreds of ms over a five minute > sampling with a few tens of instances showing. > I tried then w/ flamegraph/lightweight profiler and this reported the buffer > allocations taking 22% of our total CPU. See attachment trace.svg. > I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and > indeed every allocation was doing a resize from initial allocation of 16k -- > the default up to 220k (this test returns ten randomly sized rows zipfian > sized between 0 and 8k). > Upping the allocation to 220k meant we now avoided the resize but the initial > allocation was now blamed for 10% of allocations (see trace.2.svg attached). > Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12562) Handling memory pressure for secondary region replicas
[ https://issues.apache.org/jira/browse/HBASE-12562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351126#comment-14351126 ] Hudson commented on HBASE-12562: FAILURE: Integrated in HBase-1.1 #255 (See [https://builds.apache.org/job/HBase-1.1/255/]) HBASE-12562 Handling memory pressure for secondary region replicas (enis: rev 6e5e5d8ccef17c8d38f3284204bebd20f92525bd) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/FileLink.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java > Handling memory pressure for secondary region replicas > -- > > Key: HBASE-12562 > URL: https://issues.apache.org/jira/browse/HBASE-12562 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.1.0 > > Attachments: hbase-12562_v1.patch, hbase-12562_v2.patch, > hbase-12562_v2.patch, hbase-12562_v2.patch, hbase-12562_v3.patch > > > This issue will track the implementation of how to handle the memory pressure > for secondary region replicas. Since the replicas cannot flush by themselves, > the region server might get blocked or cause extensive flushing for its > primary regions. The design doc attached at HBASE-11183 contains two possible > solutions that we can pursue. The first one is to not allow secondary region > replicas to not flush by themselves, but instead of needed allow them to > refresh their store files on demand (which possibly allows them to drop their > memstore snapshots or memstores). The second approach is to allow the > secondaries to flush to a temporary space. > Both have pros and cons, but for simplicity and to not cause extra write > amplification, we have implemented the first approach. More details can be > found in the design doc, but we can also discuss other options here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351125#comment-14351125 ] Hudson commented on HBASE-13082: FAILURE: Integrated in HBase-1.1 #255 (See [https://builds.apache.org/job/HBase-1.1/255/]) Revert "HBASE-13082 Coarsen StoreScanner locks to RegionScanner." (larsh: rev c8610ce36f398f74aefbe46c3f65fcc9a6b4e394) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedStoreScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Coarsen StoreScanner locks to RegionScanner > --- > > Key: HBASE-13082 > URL: https://issues.apache.org/jira/browse/HBASE-13082 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 2.0.0, 1.1.0 > > Attachments: 13082-test.txt, 13082-v2.txt, 13082.txt, 13082.txt, > gc.png, gc.png, gc.png, hits.png, next.png, next.png > > > Continuing where HBASE-10015 left of. > We can avoid locking (and memory fencing) inside StoreScanner by deferring to > the lock already held by the RegionScanner. > In tests this shows quite a scan improvement and reduced CPU (the fences make > the cores wait for memory fetches). > There are some drawbacks too: > * All calls to RegionScanner need to be remain synchronized > * Implementors of coprocessors need to be diligent in following the locking > contract. For example Phoenix does not lock RegionScanner.nextRaw() and > required in the documentation (not picking on Phoenix, this one is my fault > as I told them it's OK) > * possible starving of flushes and compaction with heavy read load. > RegionScanner operations would keep getting the locks and the > flushes/compactions would not be able finalize the set of files. > I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-13082: -- Attachment: 13082-v2.txt For reference, here's the rebased patch that I had committed (and subsequently reverted) > Coarsen StoreScanner locks to RegionScanner > --- > > Key: HBASE-13082 > URL: https://issues.apache.org/jira/browse/HBASE-13082 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 2.0.0, 1.1.0 > > Attachments: 13082-test.txt, 13082-v2.txt, 13082.txt, 13082.txt, > gc.png, gc.png, gc.png, hits.png, next.png, next.png > > > Continuing where HBASE-10015 left of. > We can avoid locking (and memory fencing) inside StoreScanner by deferring to > the lock already held by the RegionScanner. > In tests this shows quite a scan improvement and reduced CPU (the fences make > the cores wait for memory fetches). > There are some drawbacks too: > * All calls to RegionScanner need to be remain synchronized > * Implementors of coprocessors need to be diligent in following the locking > contract. For example Phoenix does not lock RegionScanner.nextRaw() and > required in the documentation (not picking on Phoenix, this one is my fault > as I told them it's OK) > * possible starving of flushes and compaction with heavy read load. > RegionScanner operations would keep getting the locks and the > flushes/compactions would not be able finalize the set of files. > I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl reopened HBASE-13082: --- Reverted... Meh :( > Coarsen StoreScanner locks to RegionScanner > --- > > Key: HBASE-13082 > URL: https://issues.apache.org/jira/browse/HBASE-13082 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 2.0.0, 1.1.0 > > Attachments: 13082-test.txt, 13082.txt, 13082.txt, gc.png, gc.png, > gc.png, hits.png, next.png, next.png > > > Continuing where HBASE-10015 left of. > We can avoid locking (and memory fencing) inside StoreScanner by deferring to > the lock already held by the RegionScanner. > In tests this shows quite a scan improvement and reduced CPU (the fences make > the cores wait for memory fetches). > There are some drawbacks too: > * All calls to RegionScanner need to be remain synchronized > * Implementors of coprocessors need to be diligent in following the locking > contract. For example Phoenix does not lock RegionScanner.nextRaw() and > required in the documentation (not picking on Phoenix, this one is my fault > as I told them it's OK) > * possible starving of flushes and compaction with heavy read load. > RegionScanner operations would keep getting the locks and the > flushes/compactions would not be able finalize the set of files. > I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351103#comment-14351103 ] Lars Hofhansl commented on HBASE-13082: --- I'll revert until I can work out what the issue is. Sorry about this. > Coarsen StoreScanner locks to RegionScanner > --- > > Key: HBASE-13082 > URL: https://issues.apache.org/jira/browse/HBASE-13082 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 2.0.0, 1.1.0 > > Attachments: 13082-test.txt, 13082.txt, 13082.txt, gc.png, gc.png, > gc.png, hits.png, next.png, next.png > > > Continuing where HBASE-10015 left of. > We can avoid locking (and memory fencing) inside StoreScanner by deferring to > the lock already held by the RegionScanner. > In tests this shows quite a scan improvement and reduced CPU (the fences make > the cores wait for memory fetches). > There are some drawbacks too: > * All calls to RegionScanner need to be remain synchronized > * Implementors of coprocessors need to be diligent in following the locking > contract. For example Phoenix does not lock RegionScanner.nextRaw() and > required in the documentation (not picking on Phoenix, this one is my fault > as I told them it's OK) > * possible starving of flushes and compaction with heavy read load. > RegionScanner operations would keep getting the locks and the > flushes/compactions would not be able finalize the set of files. > I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351100#comment-14351100 ] Lars Hofhansl commented on HBASE-13082: --- Huh? Lemme double check the test failures. The preCommit did not indicate those. > Coarsen StoreScanner locks to RegionScanner > --- > > Key: HBASE-13082 > URL: https://issues.apache.org/jira/browse/HBASE-13082 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 2.0.0, 1.1.0 > > Attachments: 13082-test.txt, 13082.txt, 13082.txt, gc.png, gc.png, > gc.png, hits.png, next.png, next.png > > > Continuing where HBASE-10015 left of. > We can avoid locking (and memory fencing) inside StoreScanner by deferring to > the lock already held by the RegionScanner. > In tests this shows quite a scan improvement and reduced CPU (the fences make > the cores wait for memory fetches). > There are some drawbacks too: > * All calls to RegionScanner need to be remain synchronized > * Implementors of coprocessors need to be diligent in following the locking > contract. For example Phoenix does not lock RegionScanner.nextRaw() and > required in the documentation (not picking on Phoenix, this one is my fault > as I told them it's OK) > * possible starving of flushes and compaction with heavy read load. > RegionScanner operations would keep getting the locks and the > flushes/compactions would not be able finalize the set of files. > I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351102#comment-14351102 ] Lars Hofhansl commented on HBASE-13082: --- Well, actually it did... Missed it. Looking... > Coarsen StoreScanner locks to RegionScanner > --- > > Key: HBASE-13082 > URL: https://issues.apache.org/jira/browse/HBASE-13082 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 2.0.0, 1.1.0 > > Attachments: 13082-test.txt, 13082.txt, 13082.txt, gc.png, gc.png, > gc.png, hits.png, next.png, next.png > > > Continuing where HBASE-10015 left of. > We can avoid locking (and memory fencing) inside StoreScanner by deferring to > the lock already held by the RegionScanner. > In tests this shows quite a scan improvement and reduced CPU (the fences make > the cores wait for memory fetches). > There are some drawbacks too: > * All calls to RegionScanner need to be remain synchronized > * Implementors of coprocessors need to be diligent in following the locking > contract. For example Phoenix does not lock RegionScanner.nextRaw() and > required in the documentation (not picking on Phoenix, this one is my fault > as I told them it's OK) > * possible starving of flushes and compaction with heavy read load. > RegionScanner operations would keep getting the locks and the > flushes/compactions would not be able finalize the set of files. > I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12562) Handling memory pressure for secondary region replicas
[ https://issues.apache.org/jira/browse/HBASE-12562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12562: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've pushed this. Thanks for review. > Handling memory pressure for secondary region replicas > -- > > Key: HBASE-12562 > URL: https://issues.apache.org/jira/browse/HBASE-12562 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.1.0 > > Attachments: hbase-12562_v1.patch, hbase-12562_v2.patch, > hbase-12562_v2.patch, hbase-12562_v2.patch, hbase-12562_v3.patch > > > This issue will track the implementation of how to handle the memory pressure > for secondary region replicas. Since the replicas cannot flush by themselves, > the region server might get blocked or cause extensive flushing for its > primary regions. The design doc attached at HBASE-11183 contains two possible > solutions that we can pursue. The first one is to not allow secondary region > replicas to not flush by themselves, but instead of needed allow them to > refresh their store files on demand (which possibly allows them to drop their > memstore snapshots or memstores). The second approach is to allow the > secondaries to flush to a temporary space. > Both have pros and cons, but for simplicity and to not cause extra write > amplification, we have implemented the first approach. More details can be > found in the design doc, but we can also discuss other options here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351079#comment-14351079 ] Hudson commented on HBASE-13082: FAILURE: Integrated in HBase-1.1 #254 (See [https://builds.apache.org/job/HBase-1.1/254/]) HBASE-13082 Coarsen StoreScanner locks to RegionScanner. (larsh: rev 02522615d186e900de39c22d61d6592113f3d134) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedStoreScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java > Coarsen StoreScanner locks to RegionScanner > --- > > Key: HBASE-13082 > URL: https://issues.apache.org/jira/browse/HBASE-13082 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 2.0.0, 1.1.0 > > Attachments: 13082-test.txt, 13082.txt, 13082.txt, gc.png, gc.png, > gc.png, hits.png, next.png, next.png > > > Continuing where HBASE-10015 left of. > We can avoid locking (and memory fencing) inside StoreScanner by deferring to > the lock already held by the RegionScanner. > In tests this shows quite a scan improvement and reduced CPU (the fences make > the cores wait for memory fetches). > There are some drawbacks too: > * All calls to RegionScanner need to be remain synchronized > * Implementors of coprocessors need to be diligent in following the locking > contract. For example Phoenix does not lock RegionScanner.nextRaw() and > required in the documentation (not picking on Phoenix, this one is my fault > as I told them it's OK) > * possible starving of flushes and compaction with heavy read load. > RegionScanner operations would keep getting the locks and the > flushes/compactions would not be able finalize the set of files. > I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351074#comment-14351074 ] stack commented on HBASE-13142: --- bq. ...how ugly it can get when big rows going on To see, enable TRACE on IPUtils for a few seconds on prod regionserver. > [PERF] Reuse the IPCUtil#buildCellBlock buffer > -- > > Key: HBASE-13142 > URL: https://issues.apache.org/jira/browse/HBASE-13142 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Assignee: stack > Labels: beginner > Fix For: 2.0.0, 1.1.0 > > Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.txt, > 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, > net.png, traces.2.svg, traces.svg > > > Running some scan profiling, flight recorder was mildly fingering resize of > the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It > was half-hearted blaming it for a few hundreds of ms over a five minute > sampling with a few tens of instances showing. > I tried then w/ flamegraph/lightweight profiler and this reported the buffer > allocations taking 22% of our total CPU. See attachment trace.svg. > I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and > indeed every allocation was doing a resize from initial allocation of 16k -- > the default up to 220k (this test returns ten randomly sized rows zipfian > sized between 0 and 8k). > Upping the allocation to 220k meant we now avoided the resize but the initial > allocation was now blamed for 10% of allocations (see trace.2.svg attached). > Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351069#comment-14351069 ] stack commented on HBASE-13142: --- Might be candidate for 0.98 [~apurtell] or for 1.0.x [~enis] given how ugly it can get when big rows going on > [PERF] Reuse the IPCUtil#buildCellBlock buffer > -- > > Key: HBASE-13142 > URL: https://issues.apache.org/jira/browse/HBASE-13142 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Assignee: stack > Labels: beginner > Fix For: 2.0.0, 1.1.0 > > Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.txt, > 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, > net.png, traces.2.svg, traces.svg > > > Running some scan profiling, flight recorder was mildly fingering resize of > the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It > was half-hearted blaming it for a few hundreds of ms over a five minute > sampling with a few tens of instances showing. > I tried then w/ flamegraph/lightweight profiler and this reported the buffer > allocations taking 22% of our total CPU. See attachment trace.svg. > I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and > indeed every allocation was doing a resize from initial allocation of 16k -- > the default up to 220k (this test returns ten randomly sized rows zipfian > sized between 0 and 8k). > Upping the allocation to 220k meant we now avoided the resize but the initial > allocation was now blamed for 10% of allocations (see trace.2.svg attached). > Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13159) Consider RangeReferenceFiles with transformations
[ https://issues.apache.org/jira/browse/HBASE-13159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351071#comment-14351071 ] Enis Soztutar commented on HBASE-13159: --- Big +1. Was thinking along the same lines (excluding transformations). HFile link implements soft links, and Reference files implement soft links with top/bottom limit. We can actually benefit from another related concept, where an HFile contains information for multiple regions, and can be referred from multiple regions (with region name + boundary). This is pretty important for distributed log splitting, where we do not need to create so many small files per WAL file (we end up creating regions x WAL files many files). I believe in the original paper, BigTable achieves something like this with softlinks implemented via META (the regions files are what is there in the meta, not what is there in the file system). A range on the reference file will allow the region to be splittable after another split, but before compaction. It also allows the region to be split into more than two pieces. For the transformation, I think even local index can make use of it. If we can make persist the index data without the region start key prefix, we can apply an on-demand transformation for adding the prefix to the cells, so that after local index region split, the data does not have to be rewritten. > Consider RangeReferenceFiles with transformations > - > > Key: HBASE-13159 > URL: https://issues.apache.org/jira/browse/HBASE-13159 > Project: HBase > Issue Type: Brainstorming >Reporter: Lars Hofhansl > > Currently we have References used by HalfStoreReaders and HFileLinks. > For various use cases we have here we have need for a RangeReferences with > simple transformation of the keys. > That would allow us to map HFiles between regions or even tables without > copying any data. > We can probably combine HalfStores, HFileLinks, and RangeReferences into a > single concept: > * RangeReference = arbitrary start and stop row, arbitrary key transformation > * HFileLink = start and stop keys set to the linked file's start/stop key, > transformation = identity > * (HalfStore) References = start/stop key set according to top or bottom > reference, transformation = identity > Note this is a *brainstorming* issue. :) > (Could start with just references with arbitrary start/stop keys, and do > transformations later) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-13142: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks [~eclark] > [PERF] Reuse the IPCUtil#buildCellBlock buffer > -- > > Key: HBASE-13142 > URL: https://issues.apache.org/jira/browse/HBASE-13142 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Assignee: stack > Labels: beginner > Fix For: 2.0.0, 1.1.0 > > Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.txt, > 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, > net.png, traces.2.svg, traces.svg > > > Running some scan profiling, flight recorder was mildly fingering resize of > the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It > was half-hearted blaming it for a few hundreds of ms over a five minute > sampling with a few tens of instances showing. > I tried then w/ flamegraph/lightweight profiler and this reported the buffer > allocations taking 22% of our total CPU. See attachment trace.svg. > I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and > indeed every allocation was doing a resize from initial allocation of 16k -- > the default up to 220k (this test returns ten randomly sized rows zipfian > sized between 0 and 8k). > Upping the allocation to 220k meant we now avoided the resize but the initial > allocation was now blamed for 10% of allocations (see trace.2.svg attached). > Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351034#comment-14351034 ] Hudson commented on HBASE-13082: FAILURE: Integrated in HBase-TRUNK #6217 (See [https://builds.apache.org/job/HBase-TRUNK/6217/]) HBASE-13082 Coarsen StoreScanner locks to RegionScanner. (larsh: rev ec1eff9b69d37d2340f057361d2b269cc5fffd56) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedStoreScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Coarsen StoreScanner locks to RegionScanner > --- > > Key: HBASE-13082 > URL: https://issues.apache.org/jira/browse/HBASE-13082 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 2.0.0, 1.1.0 > > Attachments: 13082-test.txt, 13082.txt, 13082.txt, gc.png, gc.png, > gc.png, hits.png, next.png, next.png > > > Continuing where HBASE-10015 left of. > We can avoid locking (and memory fencing) inside StoreScanner by deferring to > the lock already held by the RegionScanner. > In tests this shows quite a scan improvement and reduced CPU (the fences make > the cores wait for memory fetches). > There are some drawbacks too: > * All calls to RegionScanner need to be remain synchronized > * Implementors of coprocessors need to be diligent in following the locking > contract. For example Phoenix does not lock RegionScanner.nextRaw() and > required in the documentation (not picking on Phoenix, this one is my fault > as I told them it's OK) > * possible starving of flushes and compaction with heavy read load. > RegionScanner operations would keep getting the locks and the > flushes/compactions would not be able finalize the set of files. > I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13090) Progress heartbeats for long running scanners
[ https://issues.apache.org/jira/browse/HBASE-13090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351027#comment-14351027 ] Andrew Purtell commented on HBASE-13090: Sounds ok to me too, will be interested to see the details in a patch. > Progress heartbeats for long running scanners > - > > Key: HBASE-13090 > URL: https://issues.apache.org/jira/browse/HBASE-13090 > Project: HBase > Issue Type: New Feature >Reporter: Andrew Purtell > > It can be necessary to set very long timeouts for clients that issue scans > over large regions when all data in the region might be filtered out > depending on scan criteria. This is a usability concern because it can be > hard to identify what worst case timeout to use until scans are > occasionally/intermittently failing in production, depending on variable scan > criteria. It would be better if the client-server scan protocol can send back > periodic progress heartbeats to clients as long as server scanners are alive > and making progress. > This is related but orthogonal to streaming scan (HBASE-13071). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350984#comment-14350984 ] Elliott Clark commented on HBASE-13142: --- +1 Ship it. > [PERF] Reuse the IPCUtil#buildCellBlock buffer > -- > > Key: HBASE-13142 > URL: https://issues.apache.org/jira/browse/HBASE-13142 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Assignee: stack > Labels: beginner > Fix For: 2.0.0, 1.1.0 > > Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.txt, > 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, > net.png, traces.2.svg, traces.svg > > > Running some scan profiling, flight recorder was mildly fingering resize of > the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It > was half-hearted blaming it for a few hundreds of ms over a five minute > sampling with a few tens of instances showing. > I tried then w/ flamegraph/lightweight profiler and this reported the buffer > allocations taking 22% of our total CPU. See attachment trace.svg. > I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and > indeed every allocation was doing a resize from initial allocation of 16k -- > the default up to 220k (this test returns ten randomly sized rows zipfian > sized between 0 and 8k). > Upping the allocation to 220k meant we now avoided the resize but the initial > allocation was now blamed for 10% of allocations (see trace.2.svg attached). > Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13090) Progress heartbeats for long running scanners
[ https://issues.apache.org/jira/browse/HBASE-13090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350965#comment-14350965 ] stack commented on HBASE-13090: --- That sounds good. A reset or a new session because there may be other stuff the RegionScanner wants to reset on session start other than just timers. > Progress heartbeats for long running scanners > - > > Key: HBASE-13090 > URL: https://issues.apache.org/jira/browse/HBASE-13090 > Project: HBase > Issue Type: New Feature >Reporter: Andrew Purtell > > It can be necessary to set very long timeouts for clients that issue scans > over large regions when all data in the region might be filtered out > depending on scan criteria. This is a usability concern because it can be > hard to identify what worst case timeout to use until scans are > occasionally/intermittently failing in production, depending on variable scan > criteria. It would be better if the client-server scan protocol can send back > periodic progress heartbeats to clients as long as server scanners are alive > and making progress. > This is related but orthogonal to streaming scan (HBASE-13071). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350958#comment-14350958 ] stack commented on HBASE-13142: --- Any chance of review? > [PERF] Reuse the IPCUtil#buildCellBlock buffer > -- > > Key: HBASE-13142 > URL: https://issues.apache.org/jira/browse/HBASE-13142 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Assignee: stack > Labels: beginner > Fix For: 2.0.0, 1.1.0 > > Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.txt, > 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png, > net.png, traces.2.svg, traces.svg > > > Running some scan profiling, flight recorder was mildly fingering resize of > the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It > was half-hearted blaming it for a few hundreds of ms over a five minute > sampling with a few tens of instances showing. > I tried then w/ flamegraph/lightweight profiler and this reported the buffer > allocations taking 22% of our total CPU. See attachment trace.svg. > I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and > indeed every allocation was doing a resize from initial allocation of 16k -- > the default up to 220k (this test returns ten randomly sized rows zipfian > sized between 0 and 8k). > Upping the allocation to 220k meant we now avoided the resize but the initial > allocation was now blamed for 10% of allocations (see trace.2.svg attached). > Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13063) Allow to turn off memstore replication for region replicas
[
https://issues.apache.org/jira/browse/HBASE-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350925#comment-14350925
]
Hadoop QA commented on HBASE-13063:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703106/HBASE-13063-v2.patch
against master branch at commit 6d4a4a48fe90b889c39e3ed42448fc7597a46973.
ATTACHMENT ID: 12703106
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13122//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13122//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13122//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13122//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13122//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13122//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13122//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13122//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13122//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13122//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13122//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13122//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13122//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13122//console
This message is automatically generated.
> Allow to turn off memstore replication for region replicas
> --
>
> Key: HBASE-13063
> URL: https://issues.apache.org/jira/browse/HBASE-13063
> Project: HBase
> Issue Type: New Feature
> Components: regionserver, Replication
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 2.0.0, 1.1.0
>
> Attachments: HBASE-13063-v0.patch, HBASE-13063-v1-branch-1.patch,
> HBASE-13063-v1.patch, HBASE-13063-v1.patch, HBASE-13063-v2.patch,
> HBASE-13063-v2.patch
>
>
> HBASE-11568 allows to use replication to send wal edits from the primary to
> the replicas.
> sometimes the memstore replication is not required, so it will be nice have a
> flag to disable it.
> the result will be more or less the same to what we have in phase-1, but with
> the row-level consistency provided by the "flush" edit transfered via
> replication to refresh the hfiles.
> create 't1', 'f', {REGION_REPLICATION => 2, REGION_MEMSTORE_REPLICATION =>
> false}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12562) Handling memory pressure for secondary region replicas
[
https://issues.apache.org/jira/browse/HBASE-12562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350911#comment-14350911
]
Hadoop QA commented on HBASE-12562:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703110/hbase-12562_v3.patch
against master branch at commit 6d4a4a48fe90b889c39e3ed42448fc7597a46973.
ATTACHMENT ID: 12703110
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13121//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13121//console
This message is automatically generated.
> Handling memory pressure for secondary region replicas
> --
>
> Key: HBASE-12562
> URL: https://issues.apache.org/jira/browse/HBASE-12562
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.1.0
>
> Attachments: hbase-12562_v1.patch, hbase-12562_v2.patch,
> hbase-12562_v2.patch, hbase-12562_v2.patch, hbase-12562_v3.patch
>
>
> This issue will track the implementation of how to handle the memory pressure
> for secondary region replicas. Since the replicas cannot flush by themselves,
> the region server might get blocked or cause extensive flushing for its
> primary regions. The design doc attached at HBASE-11183 contains two possible
> solutions that we can pursue. The first one is to not allow secondary region
> replicas to not flush by themselves, but instead of needed allow them to
> refresh their store files on demand (which possibly allows them to drop their
> memstore snapshots or memstores). The second approach is to allow the
> secondaries to flush to a temporary space.
> Both have pros and cons, but for simplicity and to not cause extra write
> amplifica
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[
https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350910#comment-14350910
]
Hadoop QA commented on HBASE-13142:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703111/13142v5.txt
against master branch at commit 6d4a4a48fe90b889c39e3ed42448fc7597a46973.
ATTACHMENT ID: 12703111
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13120//console
This message is automatically generated.
> [PERF] Reuse the IPCUtil#buildCellBlock buffer
> --
>
> Key: HBASE-13142
> URL: https://issues.apache.org/jira/browse/HBASE-13142
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Reporter: stack
>Assignee: stack
> Labels: beginner
> Fix For: 2.0.0, 1.1.0
>
> Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.txt,
> 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png,
> net.png, traces.2.svg, traces.svg
>
>
> Running some scan profiling, flight recorder was mildly fingering resize of
> the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It
> was half-hearted blaming it for a few hundreds of ms over a five minute
> sampling with a few tens of instances showing.
> I tried then w/ flamegraph/lightweight profiler and this reported the buffer
> allocations taking 22% of our total CPU. See attachment trace.svg.
> I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and
> indeed every allocation was doing a resize from initial allocation of 16k --
> the default up to 220k (this test returns ten randomly sized rows zipfian
> sized between
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[
https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350880#comment-14350880
]
Hadoop QA commented on HBASE-13142:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703095/13142v5.txt
against master branch at commit 6d4a4a48fe90b889c39e3ed42448fc7597a46973.
ATTACHMENT ID: 12703095
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13119//console
This message is automatically generated.
> [PERF] Reuse the IPCUtil#buildCellBlock buffer
> --
>
> Key: HBASE-13142
> URL: https://issues.apache.org/jira/browse/HBASE-13142
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Reporter: stack
>Assignee: stack
> Labels: beginner
> Fix For: 2.0.0, 1.1.0
>
> Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.txt,
> 13142v5.txt, buffers.svg, clean.svg, gc.png, gc_time_spent.png, hits.png,
> net.png, traces.2.svg, traces.svg
>
>
> Running some scan profiling, flight recorder was mildly fingering resize of
> the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It
> was half-hearted blaming it for a few hundreds of ms over a five minute
> sampling with a few tens of instances showing.
> I tried then w/ flamegraph/lightweight profiler and this reported the buffer
> allocations taking 22% of our total CPU. See attachment trace.svg.
> I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and
> indeed every allocation was doing a resize from initial allocation of 16k --
> the default up to 220k (this test returns ten randomly sized rows zipfian
> sized between
[jira] [Commented] (HBASE-13090) Progress heartbeats for long running scanners
[ https://issues.apache.org/jira/browse/HBASE-13090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350874#comment-14350874 ] Jonathan Lawlor commented on HBASE-13090: - bq. Doesn't RegionScanner get created when the scan starts? Can it not run the timer? That's true, the RegionScanner is created when the scan starts. It also remains open Server-side as long as the Client does not close it (as is the case in non-small scans). To keep all accounting of the timer within the RegionScanner we could add a call to RegionScanner#updateTimeoutTimestamp (or something along those lines) into RsRpcServices each time we either create or retrieve the RegionScanner (would avoid accounting for timeouts within RsRpcServices). The timestamp would need to be updated on each RsRpcServices#scan call to make sure that we aren't using a previously defined timeout timestamp that would be too restrictive at this point. Then all timeout information would be communicated to RsRpcServices via the newly defined NextState from HBASE-11544. > Progress heartbeats for long running scanners > - > > Key: HBASE-13090 > URL: https://issues.apache.org/jira/browse/HBASE-13090 > Project: HBase > Issue Type: New Feature >Reporter: Andrew Purtell > > It can be necessary to set very long timeouts for clients that issue scans > over large regions when all data in the region might be filtered out > depending on scan criteria. This is a usability concern because it can be > hard to identify what worst case timeout to use until scans are > occasionally/intermittently failing in production, depending on variable scan > criteria. It would be better if the client-server scan protocol can send back > periodic progress heartbeats to clients as long as server scanners are alive > and making progress. > This is related but orthogonal to streaming scan (HBASE-13071). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-13082: -- Resolution: Fixed Assignee: Lars Hofhansl Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to 1.1 and 2.0. > Coarsen StoreScanner locks to RegionScanner > --- > > Key: HBASE-13082 > URL: https://issues.apache.org/jira/browse/HBASE-13082 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 2.0.0, 1.1.0 > > Attachments: 13082-test.txt, 13082.txt, 13082.txt, gc.png, gc.png, > gc.png, hits.png, next.png, next.png > > > Continuing where HBASE-10015 left of. > We can avoid locking (and memory fencing) inside StoreScanner by deferring to > the lock already held by the RegionScanner. > In tests this shows quite a scan improvement and reduced CPU (the fences make > the cores wait for memory fetches). > There are some drawbacks too: > * All calls to RegionScanner need to be remain synchronized > * Implementors of coprocessors need to be diligent in following the locking > contract. For example Phoenix does not lock RegionScanner.nextRaw() and > required in the documentation (not picking on Phoenix, this one is my fault > as I told them it's OK) > * possible starving of flushes and compaction with heavy read load. > RegionScanner operations would keep getting the locks and the > flushes/compactions would not be able finalize the set of files. > I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8927) Use nano time instead of mili time everywhere
[ https://issues.apache.org/jira/browse/HBASE-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350822#comment-14350822 ] stack commented on HBASE-8927: -- [~lhofhansl] There is no config by default if I read this right. New tables will get a non-1 multiplier. > Use nano time instead of mili time everywhere > - > > Key: HBASE-8927 > URL: https://issues.apache.org/jira/browse/HBASE-8927 > Project: HBase > Issue Type: Bug >Reporter: stack > Labels: phoenix > Attachments: 8927.txt > > > Less collisions and we are paying the price of a long anyways so might as > well fill it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8927) Use nano time instead of mili time everywhere
[ https://issues.apache.org/jira/browse/HBASE-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350820#comment-14350820 ] stack commented on HBASE-8927: -- I like it. Simple. [~lhofhansl] > Use nano time instead of mili time everywhere > - > > Key: HBASE-8927 > URL: https://issues.apache.org/jira/browse/HBASE-8927 > Project: HBase > Issue Type: Bug >Reporter: stack > Labels: phoenix > Attachments: 8927.txt > > > Less collisions and we are paying the price of a long anyways so might as > well fill it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350808#comment-14350808 ] Lars Hofhansl commented on HBASE-13082: --- Alright... Thanks [~stack]. I'll target 1.1+. > Coarsen StoreScanner locks to RegionScanner > --- > > Key: HBASE-13082 > URL: https://issues.apache.org/jira/browse/HBASE-13082 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Fix For: 2.0.0, 1.1.0 > > Attachments: 13082-test.txt, 13082.txt, 13082.txt, gc.png, gc.png, > gc.png, hits.png, next.png, next.png > > > Continuing where HBASE-10015 left of. > We can avoid locking (and memory fencing) inside StoreScanner by deferring to > the lock already held by the RegionScanner. > In tests this shows quite a scan improvement and reduced CPU (the fences make > the cores wait for memory fetches). > There are some drawbacks too: > * All calls to RegionScanner need to be remain synchronized > * Implementors of coprocessors need to be diligent in following the locking > contract. For example Phoenix does not lock RegionScanner.nextRaw() and > required in the documentation (not picking on Phoenix, this one is my fault > as I told them it's OK) > * possible starving of flushes and compaction with heavy read load. > RegionScanner operations would keep getting the locks and the > flushes/compactions would not be able finalize the set of files. > I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-13082: -- Fix Version/s: 1.1.0 2.0.0 > Coarsen StoreScanner locks to RegionScanner > --- > > Key: HBASE-13082 > URL: https://issues.apache.org/jira/browse/HBASE-13082 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Fix For: 2.0.0, 1.1.0 > > Attachments: 13082-test.txt, 13082.txt, 13082.txt, gc.png, gc.png, > gc.png, hits.png, next.png, next.png > > > Continuing where HBASE-10015 left of. > We can avoid locking (and memory fencing) inside StoreScanner by deferring to > the lock already held by the RegionScanner. > In tests this shows quite a scan improvement and reduced CPU (the fences make > the cores wait for memory fetches). > There are some drawbacks too: > * All calls to RegionScanner need to be remain synchronized > * Implementors of coprocessors need to be diligent in following the locking > contract. For example Phoenix does not lock RegionScanner.nextRaw() and > required in the documentation (not picking on Phoenix, this one is my fault > as I told them it's OK) > * possible starving of flushes and compaction with heavy read load. > RegionScanner operations would keep getting the locks and the > flushes/compactions would not be able finalize the set of files. > I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8927) Use nano time instead of mili time everywhere
[ https://issues.apache.org/jira/browse/HBASE-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350806#comment-14350806 ] Lars Hofhansl commented on HBASE-8927: -- Coming back to this... I think in order to make TTL work we need to be able to at least distinguish old and new tables and/or column families. I still think the easiest is to add TIMESTAMP_MULTIPLIER as a table or CF option. That way we: # can handle old tables (the multiplier would default to 1) # handle new table (there the multiplier could default to 1000 - for microsecs, or 100 - for nanosec) # allow folks to tweak their their time range against their resolution # (could even allow to set the multiplier to 0, indicating we're not dealing with time at all in the TS fields, and then we can disallow TTL) # when checking the TTL, we'd scale the TTL with this value, and it would do the right thing. Note that we would not actually have a higher resolution per se, we'd just make space in the KVs. We can either set the free bits to 0, or fill them with a random numbers, or do some nanotime trickery. It's a new option (sorry [~stack]), but I do not see another way out. Thoughts? > Use nano time instead of mili time everywhere > - > > Key: HBASE-8927 > URL: https://issues.apache.org/jira/browse/HBASE-8927 > Project: HBase > Issue Type: Bug >Reporter: stack > Labels: phoenix > Attachments: 8927.txt > > > Less collisions and we are paying the price of a long anyways so might as > well fill it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
