[jira] [Commented] (HBASE-11279) Block cache could be disabled by mistake
[
https://issues.apache.org/jira/browse/HBASE-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14014937#comment-14014937
]
Nick Dimiduk commented on HBASE-11279:
--
Strange. Do you know what the value of cacheConfig was?
Block cache could be disabled by mistake
Key: HBASE-11279
URL: https://issues.apache.org/jira/browse/HBASE-11279
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
There is a weird test failure:
{noformat}
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.hbase.io.hfile.TestPrefetch.readStoreFile(TestPrefetch.java:96)
at
org.apache.hadoop.hbase.io.hfile.TestPrefetch.testPrefetch(TestPrefetch.java:66)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{noformat}
Looked into it, and found it could be because CacheConfig tries to compare a
float with a long. Probably we should do this instead:
{noformat}
-if (cachePercentage == 0L) {
+if (cachePercentage = 0.0001f) {
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11277) RPCServer threads can wedge under high load
[
https://issues.apache.org/jira/browse/HBASE-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14015018#comment-14015018
]
Lars Hofhansl commented on HBASE-11277:
---
Still delay on apache email?
[~apurtell], agree with what you said.
RPCServer threads can wedge under high load
---
Key: HBASE-11277
URL: https://issues.apache.org/jira/browse/HBASE-11277
Project: HBase
Issue Type: Bug
Affects Versions: 0.96.2, 0.98.3
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Critical
Fix For: 0.99.0, 0.96.3, 0.98.3
Attachments: HBASE-11277.patch
This is with 0.98.0 in an insecure setup with 7u55 and 7u60. Under high load,
RPCServer threads can wedge, fail to make progess, and consume 100% CPU time
on a core indefinitely.
Dumping threads, all threads are in BLOCKED or IN_NATIVE state. The IN_NATIVE
threads are mostly in EPollArrayWrapper.epollWait or
FileDispatcherImpl.read0. The number of threads found in
FileDispatcherImpl.read0 correspond to the number of runaway threads expected
based on looking at 'top' output. These look like:
{noformat}
Thread 64758: (state = IN_NATIVE)
- sun.nio.ch.FileDispatcherImpl.read0(java.io.FileDescriptor, long, int)
@bci=0 (Compiled frame; information may be imprecise)
- sun.nio.ch.SocketDispatcher.read(java.io.FileDescriptor, long, int)
@bci=4, line=39 (Compiled frame)
- sun.nio.ch.IOUtil.readIntoNativeBuffer(java.io.FileDescriptor,
java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher) @bci=114, line=223
(Compil
ed frame)
- sun.nio.ch.IOUtil.read(java.io.FileDescriptor, java.nio.ByteBuffer, long,
sun.nio.ch.NativeDispatcher) @bci=48, line=197 (Compiled frame)
- sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=234, line=379
(Compiled frame)
-
org.apache.hadoop.hbase.ipc.RpcServer.channelRead(java.nio.channels.ReadableByteChannel,
java.nio.ByteBuffer) @bci=12, line=2224 (Compiled frame)
- org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess()
@bci=509, line=1488 (Compiled frame)
-
org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(java.nio.channels.SelectionKey)
@bci=23, line=790 (Compiled frame)
- org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop() @bci=97,
line=581 (Compiled frame)
- org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run() @bci=1,
line=556 (Interpreted frame)
-
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1145 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615
(Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-6701) Revisit thrust of paragraph on splitting
[ https://issues.apache.org/jira/browse/HBASE-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-6701: --- Attachment: (was: HBASE-6701-rebased.patch) Revisit thrust of paragraph on splitting Key: HBASE-6701 URL: https://issues.apache.org/jira/browse/HBASE-6701 Project: HBase Issue Type: Bug Components: documentation Reporter: stack Assignee: Misty Stanley-Jones Attachments: HBASE-6701.patch See the thread 'md5 hash key and splits' for the confusion our paragraph on splitting seems to bring on (as well as good input on when manual splitting might be favored). The user is under the impression that he needs to manually split though his keys have md5 salt. The paragraph needs to make sure it does not bring on such confusion as it would seem to in this case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-6701) Revisit thrust of paragraph on splitting
[ https://issues.apache.org/jira/browse/HBASE-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-6701: --- Attachment: HBASE-6701-rebased.patch Revisit thrust of paragraph on splitting Key: HBASE-6701 URL: https://issues.apache.org/jira/browse/HBASE-6701 Project: HBase Issue Type: Bug Components: documentation Reporter: stack Assignee: Misty Stanley-Jones Attachments: HBASE-6701.patch See the thread 'md5 hash key and splits' for the confusion our paragraph on splitting seems to bring on (as well as good input on when manual splitting might be favored). The user is under the impression that he needs to manually split though his keys have md5 salt. The paragraph needs to make sure it does not bring on such confusion as it would seem to in this case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-6701) Revisit thrust of paragraph on splitting
[ https://issues.apache.org/jira/browse/HBASE-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-6701: --- Attachment: HBASE-6701-rebased.patch Rebased -- this patch seems to contain some whitespace-related changes. Sorry about that. Revisit thrust of paragraph on splitting Key: HBASE-6701 URL: https://issues.apache.org/jira/browse/HBASE-6701 Project: HBase Issue Type: Bug Components: documentation Reporter: stack Assignee: Misty Stanley-Jones Attachments: HBASE-6701-rebased.patch, HBASE-6701.patch See the thread 'md5 hash key and splits' for the confusion our paragraph on splitting seems to bring on (as well as good input on when manual splitting might be favored). The user is under the impression that he needs to manually split though his keys have md5 salt. The paragraph needs to make sure it does not bring on such confusion as it would seem to in this case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-6701) Revisit thrust of paragraph on splitting
[ https://issues.apache.org/jira/browse/HBASE-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-6701: --- Attachment: (was: HBASE-6701-rebased.patch) Revisit thrust of paragraph on splitting Key: HBASE-6701 URL: https://issues.apache.org/jira/browse/HBASE-6701 Project: HBase Issue Type: Bug Components: documentation Reporter: stack Assignee: Misty Stanley-Jones Attachments: HBASE-6701.patch See the thread 'md5 hash key and splits' for the confusion our paragraph on splitting seems to bring on (as well as good input on when manual splitting might be favored). The user is under the impression that he needs to manually split though his keys have md5 salt. The paragraph needs to make sure it does not bring on such confusion as it would seem to in this case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11278) Port HBASE-11201 'Enable global procedure members to return values to procedure master' to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14015135#comment-14015135 ] Andrew Purtell commented on HBASE-11278: Who is requesting the backport? Is there a use case? Port HBASE-11201 'Enable global procedure members to return values to procedure master' to 0.98 --- Key: HBASE-11278 URL: https://issues.apache.org/jira/browse/HBASE-11278 Project: HBase Issue Type: Task Reporter: Ted Yu HBASE-11201 adds the capability for procedure members to return data back to the master. This issue is to backport the feature to 0.98 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-6701) Revisit thrust of paragraph on splitting
[ https://issues.apache.org/jira/browse/HBASE-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-6701: --- Attachment: HBASE-6701-rebased.patch Revisit thrust of paragraph on splitting Key: HBASE-6701 URL: https://issues.apache.org/jira/browse/HBASE-6701 Project: HBase Issue Type: Bug Components: documentation Reporter: stack Assignee: Misty Stanley-Jones Attachments: HBASE-6701-rebased.patch, HBASE-6701.patch See the thread 'md5 hash key and splits' for the confusion our paragraph on splitting seems to bring on (as well as good input on when manual splitting might be favored). The user is under the impression that he needs to manually split though his keys have md5 salt. The paragraph needs to make sure it does not bring on such confusion as it would seem to in this case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-6701) Revisit thrust of paragraph on splitting
[
https://issues.apache.org/jira/browse/HBASE-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14015140#comment-14015140
]
Hadoop QA commented on HBASE-6701:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12647844/HBASE-6701-rebased.patch
against trunk revision .
ATTACHMENT ID: 12647844
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9659//console
This message is automatically generated.
Revisit thrust of paragraph on splitting
Key: HBASE-6701
URL: https://issues.apache.org/jira/browse/HBASE-6701
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: stack
Assignee: Misty Stanley-Jones
Attachments: HBASE-6701-rebased.patch, HBASE-6701.patch
See the thread 'md5 hash key and splits' for the confusion our paragraph on
splitting seems to bring on (as well as good input on when manual splitting
might be favored). The user is under the impression that he needs to
manually split though his keys have md5 salt. The paragraph needs to make
sure it does not bring on such confusion as it would seem to in this case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HBASE-11204) Document bandwidth consumption limit feature for ExportSnapshot
[ https://issues.apache.org/jira/browse/HBASE-11204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones reassigned HBASE-11204: --- Assignee: Misty Stanley-Jones Document bandwidth consumption limit feature for ExportSnapshot --- Key: HBASE-11204 URL: https://issues.apache.org/jira/browse/HBASE-11204 Project: HBase Issue Type: Task Components: snapshots Reporter: Ted Yu Assignee: Misty Stanley-Jones Priority: Minor http://hbase.apache.org/book.html#ops.snapshots.export should document bandwidth consumption limit feature which is implemented by HBASE-11083 and HBASE-11090 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-6701) Revisit thrust of paragraph on splitting
[ https://issues.apache.org/jira/browse/HBASE-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-6701: --- Attachment: (was: HBASE-6701-rebased.patch) Revisit thrust of paragraph on splitting Key: HBASE-6701 URL: https://issues.apache.org/jira/browse/HBASE-6701 Project: HBase Issue Type: Bug Components: documentation Reporter: stack Assignee: Misty Stanley-Jones Attachments: HBASE-6701.patch See the thread 'md5 hash key and splits' for the confusion our paragraph on splitting seems to bring on (as well as good input on when manual splitting might be favored). The user is under the impression that he needs to manually split though his keys have md5 salt. The paragraph needs to make sure it does not bring on such confusion as it would seem to in this case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-6701) Revisit thrust of paragraph on splitting
[ https://issues.apache.org/jira/browse/HBASE-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-6701: --- Attachment: HBASE-6701-rebased.patch Had to use --no-prefix to get the patch to work. Revisit thrust of paragraph on splitting Key: HBASE-6701 URL: https://issues.apache.org/jira/browse/HBASE-6701 Project: HBase Issue Type: Bug Components: documentation Reporter: stack Assignee: Misty Stanley-Jones Attachments: HBASE-6701-rebased.patch, HBASE-6701.patch See the thread 'md5 hash key and splits' for the confusion our paragraph on splitting seems to bring on (as well as good input on when manual splitting might be favored). The user is under the impression that he needs to manually split though his keys have md5 salt. The paragraph needs to make sure it does not bring on such confusion as it would seem to in this case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11280) Document distributed log replay
Misty Stanley-Jones created HBASE-11280: --- Summary: Document distributed log replay Key: HBASE-11280 URL: https://issues.apache.org/jira/browse/HBASE-11280 Project: HBase Issue Type: Sub-task Components: MTTR, wal Reporter: Misty Stanley-Jones Assignee: stack Fix For: 0.99.0 Enable 'distributed log replay' by default. Depends on hfilev3 being enabled. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HBASE-11280) Document distributed log replay
[ https://issues.apache.org/jira/browse/HBASE-11280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones reassigned HBASE-11280: --- Assignee: Misty Stanley-Jones (was: stack) Document distributed log replay --- Key: HBASE-11280 URL: https://issues.apache.org/jira/browse/HBASE-11280 Project: HBase Issue Type: Sub-task Components: documentation, MTTR, wal Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Fix For: 0.99.0 Enable 'distributed log replay' by default. Depends on hfilev3 being enabled. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11280) Document distributed log replay
[ https://issues.apache.org/jira/browse/HBASE-11280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11280: Component/s: documentation Document distributed log replay --- Key: HBASE-11280 URL: https://issues.apache.org/jira/browse/HBASE-11280 Project: HBase Issue Type: Sub-task Components: documentation, MTTR, wal Reporter: Misty Stanley-Jones Assignee: stack Fix For: 0.99.0 Enable 'distributed log replay' by default. Depends on hfilev3 being enabled. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11238) Add info about SlabCache and BucketCache to Ref Guide
[ https://issues.apache.org/jira/browse/HBASE-11238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14015169#comment-14015169 ] Dima Spivak commented on HBASE-11238: - Just as a thought: it might be worth noting on which machines particular files need to be modified (e.g. those running RegionServers) for things like this. Playing around with setting up BucketCache earlier today, I inadvertently modified hbase-env.sh on my gateway before understanding why my RS' couldn't restart without throwing OutOfMemoryError. Add info about SlabCache and BucketCache to Ref Guide - Key: HBASE-11238 URL: https://issues.apache.org/jira/browse/HBASE-11238 Project: HBase Issue Type: Bug Components: documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11238-2.patch, HBASE-11238-3.patch, HBASE-11238.patch Upstream issues: HBASE-11171 and HBASE-11098. Could back port some of what is in these issues, the package-info.java class for instance. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-6701) Revisit thrust of paragraph on splitting
[
https://issues.apache.org/jira/browse/HBASE-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14015171#comment-14015171
]
Hadoop QA commented on HBASE-6701:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12647848/HBASE-6701-rebased.patch
against trunk revision .
ATTACHMENT ID: 12647848
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ paraThe balancer is a periodic operation which is run on the
master to redistribute regions on the cluster. It is configured via
+ paraSee xref linkend=master.processes.loadbalancer / for more
information on the LoadBalancer.
+ paraDo not turn off block cache (You'd do it by setting
varnamehbase.block.cache.size/varname to zero).
+ Currently we do not do well if you do this because the regionserver
will spend all its time loading hfile
+ indices over and over again. If your working set it such that
block cache does you no good, at least
+ size the block cache such that hfile indices will stay up in the
cache (you can get a rough idea
+ on the size you need by surveying regionserver UIs; you'll see
index block size accounted near the
+ titlelink
xlink:href=http://en.wikipedia.org/wiki/Nagle's_algorithmNagle's/link or
the small package problem/title
+ link
xlink:href=http://search-hadoop.com/m/pduLg2fydtE/Inconsistent+scan+performance+with+caching+set+amp;subj=Re+Inconsistent+scan+performance+with+caching+set+to+1;Inconsistent
scan performance with caching set to 1/link
+ see the graphs on the tail of link
xlink:href=https://issues.apache.org/jira/browse/HBASE-7008;HBASE-7008 Set
scanner caching to a better default/link
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestMultiParallel
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9660//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9660//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9660//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9660//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9660//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9660//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9660//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9660//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9660//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9660//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9660//console
This message is automatically generated.
Revisit thrust of paragraph on splitting
Key: HBASE-6701
URL: https://issues.apache.org/jira/browse/HBASE-6701
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: stack
Assignee: Misty Stanley-Jones
Attachments: HBASE-6701-rebased.patch, HBASE-6701.patch
See the thread 'md5 hash key and splits' for the confusion our paragraph on
splitting seems to bring on (as well as good input on when manual splitting
might be favored). The user is under the impression that he needs
[jira] [Updated] (HBASE-11238) Add info about SlabCache and BucketCache to Ref Guide
[ https://issues.apache.org/jira/browse/HBASE-11238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11238: Attachment: HBASE-11238-4.patch Rebased again and added Dima's info about where the configuration should happen. I also did a bit more spot-refactoring but didn't change any content. Add info about SlabCache and BucketCache to Ref Guide - Key: HBASE-11238 URL: https://issues.apache.org/jira/browse/HBASE-11238 Project: HBase Issue Type: Bug Components: documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11238-2.patch, HBASE-11238-3.patch, HBASE-11238-4.patch, HBASE-11238.patch Upstream issues: HBASE-11171 and HBASE-11098. Could back port some of what is in these issues, the package-info.java class for instance. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11280) Document distributed log replay
[ https://issues.apache.org/jira/browse/HBASE-11280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11280: Release Note: (was: Enable Distributed Log Replay (DLR) as default WAL/hlog splitting mechanism. Replaces Distributed Log Splitting (DLS) system. Improves MTTR (the more regions a server is carrying, the better the improvement. Also, we can start taking writes during WAL/hlog recovery). Requires HFileV3 (On by default in 0.99/1.0). See HBASE-7006 to learn more about DLR.) Document distributed log replay --- Key: HBASE-11280 URL: https://issues.apache.org/jira/browse/HBASE-11280 Project: HBase Issue Type: Sub-task Components: documentation, MTTR, wal Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Fix For: 0.99.0 Enable 'distributed log replay' by default. Depends on hfilev3 being enabled. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11280) Document distributed log replay
[ https://issues.apache.org/jira/browse/HBASE-11280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14015173#comment-14015173 ] Misty Stanley-Jones commented on HBASE-11280: - http://blog.cloudera.com/blog/2012/07/hbase-log-splitting/ Document distributed log replay --- Key: HBASE-11280 URL: https://issues.apache.org/jira/browse/HBASE-11280 Project: HBase Issue Type: Sub-task Components: documentation, MTTR, wal Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Fix For: 0.99.0 Enable 'distributed log replay' by default. Depends on hfilev3 being enabled. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10935) support snapshot policy where flush memstore can be skipped to prevent production cluster freeze
[ https://issues.apache.org/jira/browse/HBASE-10935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14015176#comment-14015176 ] Tianying Chang commented on HBASE-10935: [~mbertozzi] Sure! Thanks! support snapshot policy where flush memstore can be skipped to prevent production cluster freeze Key: HBASE-10935 URL: https://issues.apache.org/jira/browse/HBASE-10935 Project: HBase Issue Type: New Feature Components: shell, snapshots Affects Versions: 0.94.7, 0.94.18 Reporter: Tianying Chang Assignee: Tianying Chang Priority: Minor Fix For: 0.94.7, 0.99.0, 0.94.20 Attachments: HBASE-10935-0.94-v1.patch, HBASE-10935-0.98-v1.patch, HBASE-10935-trunk-v1.patch, hbase-10935-94.patch, hbase-10935-trunk.patch We are using snapshot feature to do HBase disaster recovery. We will do snapshot in our production cluster periodically. The current flush snapshot policy require all regions of the table to coordinate to prevent write and do flush at the same time. Since we use WALPlayer to complete the data that is not in the snapshot HFile, we don't need the snapshot to do coordinated flush. The snapshot just recored all the HFile that are already there. I added the parameter in the HBase shell. So people can choose to use the NoFlush snapshot when they need, like below. Otherwise, the default flush snpahot support is not impacted. snaphot 'TestTable', 'TestSnapshot', 'skipFlush' -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11278) Port HBASE-11201 'Enable global procedure members to return values to procedure master' to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14015183#comment-14015183 ] Jerry He commented on HBASE-11278: -- Hi, Andrew, Ted I plan to use HBASE-11201 for HBASE-11148. If there is a need for HBASE-11148 to go into 0.98 eventually, there will be a need for this backport. Port HBASE-11201 'Enable global procedure members to return values to procedure master' to 0.98 --- Key: HBASE-11278 URL: https://issues.apache.org/jira/browse/HBASE-11278 Project: HBase Issue Type: Task Reporter: Ted Yu HBASE-11201 adds the capability for procedure members to return data back to the master. This issue is to backport the feature to 0.98 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-8073) HFileOutputFormat support for offline operation
[ https://issues.apache.org/jira/browse/HBASE-8073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-8073: Attachment: HBASE-8073-trunk-v1.patch HFileOutputFormat support for offline operation --- Key: HBASE-8073 URL: https://issues.apache.org/jira/browse/HBASE-8073 Project: HBase Issue Type: Sub-task Components: mapreduce Reporter: Nick Dimiduk Fix For: 0.99.0 Attachments: HBASE-8073-trunk-v0.patch, HBASE-8073-trunk-v1.patch When using HFileOutputFormat to generate HFiles, it inspects the region topology of the target table. The split points from that table are used to guide the TotalOrderPartitioner. If the target table does not exist, it is first created. This imposes an unnecessary dependence on an online HBase and existing table. If the table exists, it can be used. However, the job can be smarter. For example, if there's far more data going into the HFiles than the table currently contains, the table regions aren't very useful for data split points. Instead, the input data can be sampled to produce split points more meaningful to the dataset. LoadIncrementalHFiles is already capable of handling divergence between HFile boundaries and table regions, so this should not pose any additional burdon at load time. The proper method of sampling the data likely requires a custom input format and an additional map-reduce job perform the sampling. See a relevant implementation: https://github.com/alexholmes/hadoop-book/blob/master/src/main/java/com/manning/hip/ch4/sampler/ReservoirSamplerInputFormat.java -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8073) HFileOutputFormat support for offline operation
[ https://issues.apache.org/jira/browse/HBASE-8073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14015185#comment-14015185 ] Jerry He commented on HBASE-8073: - Attached v1. Fixed the test case errors: need to mock a HTableDescriptor in the two test cases. HFileOutputFormat support for offline operation --- Key: HBASE-8073 URL: https://issues.apache.org/jira/browse/HBASE-8073 Project: HBase Issue Type: Sub-task Components: mapreduce Reporter: Nick Dimiduk Fix For: 0.99.0 Attachments: HBASE-8073-trunk-v0.patch, HBASE-8073-trunk-v1.patch When using HFileOutputFormat to generate HFiles, it inspects the region topology of the target table. The split points from that table are used to guide the TotalOrderPartitioner. If the target table does not exist, it is first created. This imposes an unnecessary dependence on an online HBase and existing table. If the table exists, it can be used. However, the job can be smarter. For example, if there's far more data going into the HFiles than the table currently contains, the table regions aren't very useful for data split points. Instead, the input data can be sampled to produce split points more meaningful to the dataset. LoadIncrementalHFiles is already capable of handling divergence between HFile boundaries and table regions, so this should not pose any additional burdon at load time. The proper method of sampling the data likely requires a custom input format and an additional map-reduce job perform the sampling. See a relevant implementation: https://github.com/alexholmes/hadoop-book/blob/master/src/main/java/com/manning/hip/ch4/sampler/ReservoirSamplerInputFormat.java -- This message was sent by Atlassian JIRA (v6.2#6252)
