[jira] [Commented] (HBASE-10954) Fix TestCloseRegionHandler.testFailedFlushAborts
[
https://issues.apache.org/jira/browse/HBASE-10954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965043#comment-13965043
]
Hadoop QA commented on HBASE-10954:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639527/HBASE-10954.patch
against trunk revision .
ATTACHMENT ID: 12639527
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9241//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9241//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9241//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9241//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9241//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9241//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9241//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9241//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9241//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9241//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9241//console
This message is automatically generated.
Fix TestCloseRegionHandler.testFailedFlushAborts
Key: HBASE-10954
URL: https://issues.apache.org/jira/browse/HBASE-10954
Project: HBase
Issue Type: Bug
Components: regionserver
Reporter: Mikhail Antonov
Assignee: Mikhail Antonov
Fix For: 0.99.0
Attachments: HBASE-10954.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10953) Remove unused @private SimpleBlockCache (and RandomSeek tool that uses it)
[
https://issues.apache.org/jira/browse/HBASE-10953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965045#comment-13965045
]
Hadoop QA commented on HBASE-10953:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639524/10953.txt
against trunk revision .
ATTACHMENT ID: 12639524
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.procedure.TestZKProcedure
org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9240//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9240//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9240//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9240//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9240//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9240//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9240//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9240//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9240//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9240//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9240//console
This message is automatically generated.
Remove unused @private SimpleBlockCache (and RandomSeek tool that uses it)
--
Key: HBASE-10953
URL: https://issues.apache.org/jira/browse/HBASE-10953
Project: HBase
Issue Type: Task
Reporter: stack
Assignee: stack
Attachments: 10953.txt
Tripped over these today. Remove. Not used. @Private
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10929) Change ScanQueryMatcher to use Cells instead of KeyValue.
[ https://issues.apache.org/jira/browse/HBASE-10929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10929: --- Status: Open (was: Patch Available) Change ScanQueryMatcher to use Cells instead of KeyValue. - Key: HBASE-10929 URL: https://issues.apache.org/jira/browse/HBASE-10929 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10929.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10929) Change ScanQueryMatcher to use Cells instead of KeyValue.
[ https://issues.apache.org/jira/browse/HBASE-10929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10929: --- Attachment: HBASE-10929_1.patch Updated patch that has the changes in StoreScanner also that would be needed with SQM. [~saint@gmail.com] Trying to get some perf numbers with these changes. Will publish once am done with it. Change ScanQueryMatcher to use Cells instead of KeyValue. - Key: HBASE-10929 URL: https://issues.apache.org/jira/browse/HBASE-10929 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10929.patch, HBASE-10929_1.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10929) Change ScanQueryMatcher to use Cells instead of KeyValue.
[ https://issues.apache.org/jira/browse/HBASE-10929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10929: --- Status: Patch Available (was: Open) Change ScanQueryMatcher to use Cells instead of KeyValue. - Key: HBASE-10929 URL: https://issues.apache.org/jira/browse/HBASE-10929 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10929.patch, HBASE-10929_1.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[
https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965067#comment-13965067
]
Matt Corgan commented on HBASE-10771:
-
taking a look at patch_2 after a mention in HBASE-10861
{code}PairLong, Integer getVLong(int index) throws IOException{code}this
returns a fairly heavyweight object. maybe it should just return a primitive
long and we could have a public static method int numVLongBytes(long n) that
quickly computes the number of bytes. here is a different VLong format i wrote
a while back. i don't know what the numBytes formula for this particular VLong
would be, but you can see how efficiently it could possibly be calculated.
probably much cheaper than creating 3 objects
{code}
public static int numBytes(long in){// do a check for illegal arguments
if not protected
if(in == 0){ return 1; }// doesn't work with the formula below
return (70 - Long.numberOfLeadingZeros(in)) / 7;// 70 comes
from 64+(7-1)
}
public static byte[] getBytes(long value){
int numBytes = numBytes(value);
byte[] bytes = new byte[numBytes];
long remainder = value;
for(int i = 0; i numBytes - 1; ++i){
bytes[i] = (byte)((remainder LONG_7_RIGHT_BITS_SET) |
LONG_8TH_BIT_SET);// set the left bit
remainder = 7;
}
bytes[numBytes - 1] = (byte)(remainder
LONG_7_RIGHT_BITS_SET);// do not set the left bit
return bytes;
}
{code}
also not sure the checked IOException is necessary since all of these methods
could encounter similar corruption errors, plus the actual IO has presumably
already been done earlier.
{code}
+ @Override
+ public long getLong(int index) {
+return Bytes.toLong(bytes, index);
+ }
{code}
does the hbase Bytes util have the exact same format that ByteBuffers use?
native java format can be seen in java.nio.Bits.java
Primitive type put/get APIs in ByteRange
-
Key: HBASE-10771
URL: https://issues.apache.org/jira/browse/HBASE-10771
Project: HBase
Issue Type: Improvement
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.99.0
Attachments: HBASE-10771.patch, HBASE-10771_V2.patch
While doing HBASE-10713 I came across the need to write int/long (and read
also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add
such APIs.
Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion
under HBASE-10191 suggest we can have BR backed HFileBlocks etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10861) Supporting API in ByteRange
[
https://issues.apache.org/jira/browse/HBASE-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965070#comment-13965070
]
Matt Corgan commented on HBASE-10861:
-
{quote}In HBASE-10771 we had some discussion regarding that. Matt Corgan mind
taking a look at the patch on that Jira.{quote}sure, i left some comments over
there.
{quote}So do we have a consensus regarding the BR backed by offheap ?{quote}i
hate advocate too strongly for it since there are so many unknowns, but I do
think we need some sort of wrapper object for byte[]'s and ByteBuffers. i just
think it strange that java doesn't include a wrapper for byte[]'s like String
wraps char[]. The biggest difference from String is that we're talking about a
mutable wrapper, but I think we could get a lot of mileage out of recycling the
ByteRange wrapper in performance critical spots - just need to be careful.
Supporting API in ByteRange
---
Key: HBASE-10861
URL: https://issues.apache.org/jira/browse/HBASE-10861
Project: HBase
Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-10861.patch, HBASE-10861_2.patch
We would need APIs that would
setLimit(int limit)
getLimt()
asReadOnly()
These APIs would help in implementations that have Buffers offheap (for now
BRs backed by DBB).
If anything more is needed could be added when needed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10801) Ensure DBE interfaces can work with Cell
[ https://issues.apache.org/jira/browse/HBASE-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965073#comment-13965073 ] Matt Corgan commented on HBASE-10801: - Hmm- I thought we had a CellScanner interface with methods advance() and current(). The idea there is to advance right before calling current(), whereas the code above looks like it's preemtively calling next(). I wonder if it really needs to advance to the next key before it's needed. I'm not sure... Ensure DBE interfaces can work with Cell Key: HBASE-10801 URL: https://issues.apache.org/jira/browse/HBASE-10801 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10801.patch, HBASE-10801_1.patch Some changes to the interfaces may be needed for DBEs or may be the way it works currently may be need to be modified inorder to make DBEs work with Cells. Suggestions and ideas welcome. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10947) Allow subclassing of HTable and HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-10947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965072#comment-13965072 ] ramkrishna.s.vasudevan commented on HBASE-10947: Regarding HBaseAdmin, atleast in 0.94 we don't have an interface for that. Hence subclassing HBaseAdmin would help to define a wrapper around the core admin functionalities that we need. Wrt HTable, we can implement the HTableInterface(or even extend the HTableInterface), but code in the MR side like TableInputFormatBase return HTable for APIs like getHTable(). So in these places if we need the wrapper of the HTable to be returned then need to change the MR code. If we are ok, then would change the MR code. Thoughts!! Allow subclassing of HTable and HBaseAdmin -- Key: HBASE-10947 URL: https://issues.apache.org/jira/browse/HBASE-10947 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan To extend functionality of HTable and HBaseAdmin we may need to subclass them. This JIRA allows to add a default constructor and probably remove the final variables in them so that we could subclass them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10801) Ensure DBE interfaces can work with Cell
[
https://issues.apache.org/jira/browse/HBASE-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965074#comment-13965074
]
Matt Corgan commented on HBASE-10801:
-
{quote} the code above looks like it's preemtively calling
next(){quote}preemtively calling hfs.next(). i wonder if that could be delayed.
Ensure DBE interfaces can work with Cell
Key: HBASE-10801
URL: https://issues.apache.org/jira/browse/HBASE-10801
Project: HBase
Issue Type: Sub-task
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0
Attachments: HBASE-10801.patch, HBASE-10801_1.patch
Some changes to the interfaces may be needed for DBEs or may be the way it
works currently may be need to be modified inorder to make DBEs work with
Cells. Suggestions and ideas welcome.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10929) Change ScanQueryMatcher to use Cells instead of KeyValue.
[
https://issues.apache.org/jira/browse/HBASE-10929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965078#comment-13965078
]
Hadoop QA commented on HBASE-10929:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639541/HBASE-10929_1.patch
against trunk revision .
ATTACHMENT ID: 12639541
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 javac{color}. The patch appears to cause mvn compile goal to
fail.
{color:red}-1 findbugs{color}. The patch appears to cause Findbugs
(version 1.3.9) to fail.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9243//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9243//console
This message is automatically generated.
Change ScanQueryMatcher to use Cells instead of KeyValue.
-
Key: HBASE-10929
URL: https://issues.apache.org/jira/browse/HBASE-10929
Project: HBase
Issue Type: Sub-task
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0
Attachments: HBASE-10929.patch, HBASE-10929_1.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-7319) Extend Cell usage through read path
[ https://issues.apache.org/jira/browse/HBASE-7319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965081#comment-13965081 ] ramkrishna.s.vasudevan commented on HBASE-7319: --- I would rename to getCell() in another JIRA after the pending things are done. Extend Cell usage through read path --- Key: HBASE-7319 URL: https://issues.apache.org/jira/browse/HBASE-7319 Project: HBase Issue Type: Umbrella Components: Compaction, Performance, regionserver, Scanners Reporter: Matt Corgan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-7319.patch, HBASE-7319_1.patch, HBASE-7319_1.patch, HBASE-7319_2.patch Umbrella issue for eliminating Cell copying. The Cell interface allows us to work with a reference to underlying bytes in the block cache without copying each Cell into consecutive bytes in an array (KeyValue). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10801) Ensure DBE interfaces can work with Cell
[ https://issues.apache.org/jira/browse/HBASE-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10801: --- Status: Patch Available (was: Open) Ensure DBE interfaces can work with Cell Key: HBASE-10801 URL: https://issues.apache.org/jira/browse/HBASE-10801 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10801.patch, HBASE-10801_1.patch, HBASE-10801_2.patch Some changes to the interfaces may be needed for DBEs or may be the way it works currently may be need to be modified inorder to make DBEs work with Cells. Suggestions and ideas welcome. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10801) Ensure DBE interfaces can work with Cell
[ https://issues.apache.org/jira/browse/HBASE-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10801: --- Status: Open (was: Patch Available) Ensure DBE interfaces can work with Cell Key: HBASE-10801 URL: https://issues.apache.org/jira/browse/HBASE-10801 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10801.patch, HBASE-10801_1.patch, HBASE-10801_2.patch Some changes to the interfaces may be needed for DBEs or may be the way it works currently may be need to be modified inorder to make DBEs work with Cells. Suggestions and ideas welcome. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10801) Ensure DBE interfaces can work with Cell
[ https://issues.apache.org/jira/browse/HBASE-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10801: --- Attachment: HBASE-10801_2.patch Updated patch addressing the comments. Moved those primitive members to an inner class. Anyway to get the full benefit we need to see if the call to hfs.next() can be delayed as per Matt. Ensure DBE interfaces can work with Cell Key: HBASE-10801 URL: https://issues.apache.org/jira/browse/HBASE-10801 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10801.patch, HBASE-10801_1.patch, HBASE-10801_2.patch Some changes to the interfaces may be needed for DBEs or may be the way it works currently may be need to be modified inorder to make DBEs work with Cells. Suggestions and ideas welcome. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10915) Decouple region closing from ZooKeeper
[
https://issues.apache.org/jira/browse/HBASE-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965112#comment-13965112
]
Hadoop QA commented on HBASE-10915:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639535/HBASE-10915.patch
against trunk revision .
ATTACHMENT ID: 12639535
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestMultiParallel
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9242//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9242//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9242//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9242//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9242//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9242//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9242//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9242//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9242//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9242//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9242//console
This message is automatically generated.
Decouple region closing from ZooKeeper
--
Key: HBASE-10915
URL: https://issues.apache.org/jira/browse/HBASE-10915
Project: HBase
Issue Type: Sub-task
Components: Zookeeper
Reporter: Mikhail Antonov
Assignee: Mikhail Antonov
Attachments: HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch,
HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch
Decouple CloseRegionHandler class and code using it (HRegionServer,
ProtobufUtil) from ZK API.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10929) Change ScanQueryMatcher to use Cells instead of KeyValue.
[ https://issues.apache.org/jira/browse/HBASE-10929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10929: --- Status: Open (was: Patch Available) Change ScanQueryMatcher to use Cells instead of KeyValue. - Key: HBASE-10929 URL: https://issues.apache.org/jira/browse/HBASE-10929 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10929.patch, HBASE-10929_1.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10934) Provide HBaseAdminInterface to abstract HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-10934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965148#comment-13965148 ] Enis Soztutar commented on HBASE-10934: --- I was thinking about this, and I think we should consider whether we want the Admin / Table interfaces to be interfaces or abstract classes. Interfaces are natural choice, but if we will support users to have their own implementations of these interfaces, then adding new methods would break the compilation. Abstract classes are more flexible in that sense. If we go the interface way, I think we should document that although the API would be public, we might add methods freely. Provide HBaseAdminInterface to abstract HBaseAdmin -- Key: HBASE-10934 URL: https://issues.apache.org/jira/browse/HBASE-10934 Project: HBase Issue Type: Improvement Components: Client Reporter: Carter Priority: Blocker Fix For: 0.99.0 As HBaseAdmin is essentially the administrative API, it would seem to follow Java best practices to provide an interface to access it instead of requiring applications to use the raw object. I am proposing (and would be happy to develop): * A new interface, HBaseAdminInterface, that captures the signatures of the API (HBaseAdmin will implement this interface) * A new method, HConnection.getHBaseAdmin(), that returns an instance of the interface -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10070) HBase read high-availability using eventually consistent region replicas
[ https://issues.apache.org/jira/browse/HBASE-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965150#comment-13965150 ] Enis Soztutar commented on HBASE-10070: --- bq. Enis Soztutar Is the 3rd option of Async WAL replication feature already implemented? Not yet, we are finishing the phase 1 as identified in the doc. Only the scan support is left for that. Afterwards, we plan to continue on implementing the asyn wal approach with a design proposal of it's own. Are you interested in using async wal replication separately, or you are asking whether the feature as a whole is available? HBase read high-availability using eventually consistent region replicas Key: HBASE-10070 URL: https://issues.apache.org/jira/browse/HBASE-10070 Project: HBase Issue Type: New Feature Reporter: Enis Soztutar Assignee: Enis Soztutar Attachments: HighAvailabilityDesignforreadsApachedoc.pdf In the present HBase architecture, it is hard, probably impossible, to satisfy constraints like 99th percentile of the reads will be served under 10 ms. One of the major factors that affects this is the MTTR for regions. There are three phases in the MTTR process - detection, assignment, and recovery. Of these, the detection is usually the longest and is presently in the order of 20-30 seconds. During this time, the clients would not be able to read the region data. However, some clients will be better served if regions will be available for reads during recovery for doing eventually consistent reads. This will help with satisfying low latency guarantees for some class of applications which can work with stale reads. For improving read availability, we propose a replicated read-only region serving design, also referred as secondary regions, or region shadows. Extending current model of a region being opened for reads and writes in a single region server, the region will be also opened for reading in region servers. The region server which hosts the region for reads and writes (as in current case) will be declared as PRIMARY, while 0 or more region servers might be hosting the region as SECONDARY. There may be more than one secondary (replica count 2). Will attach a design doc shortly which contains most of the details and some thoughts about development approaches. Reviews are more than welcome. We also have a proof of concept patch, which includes the master and regions server side of changes. Client side changes will be coming soon as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10943) Backport HBASE-7329 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-10943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-10943: Attachment: HBASE-10943-0.94-v1.diff Backport HBASE-7329 to 0.94 --- Key: HBASE-10943 URL: https://issues.apache.org/jira/browse/HBASE-10943 Project: HBase Issue Type: Improvement Affects Versions: 0.94.18 Reporter: Liu Shaohui Priority: Minor Attachments: HBASE-10943-0.94-v1.diff See HBASE-7329 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10943) Backport HBASE-7329 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-10943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965168#comment-13965168 ] Liu Shaohui commented on HBASE-10943: - [~lhofhansl] What's your suggestion about this patch? Backport HBASE-7329 to 0.94 --- Key: HBASE-10943 URL: https://issues.apache.org/jira/browse/HBASE-10943 Project: HBase Issue Type: Improvement Affects Versions: 0.94.18 Reporter: Liu Shaohui Priority: Minor Attachments: HBASE-10943-0.94-v1.diff See HBASE-7329 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10943) Backport HBASE-7329 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-10943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Xie updated HBASE-10943: -- Assignee: Liu Shaohui Backport HBASE-7329 to 0.94 --- Key: HBASE-10943 URL: https://issues.apache.org/jira/browse/HBASE-10943 Project: HBase Issue Type: Improvement Affects Versions: 0.94.18 Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Attachments: HBASE-10943-0.94-v1.diff See HBASE-7329 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10943) Backport HBASE-7329 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-10943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Xie updated HBASE-10943: -- Status: Patch Available (was: Open) Backport HBASE-7329 to 0.94 --- Key: HBASE-10943 URL: https://issues.apache.org/jira/browse/HBASE-10943 Project: HBase Issue Type: Improvement Affects Versions: 0.94.18 Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Attachments: HBASE-10943-0.94-v1.diff See HBASE-7329 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10943) Backport HBASE-7329 to 0.94
[
https://issues.apache.org/jira/browse/HBASE-10943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965191#comment-13965191
]
Hadoop QA commented on HBASE-10943:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639555/HBASE-10943-0.94-v1.diff
against trunk revision .
ATTACHMENT ID: 12639555
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 14 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9245//console
This message is automatically generated.
Backport HBASE-7329 to 0.94
---
Key: HBASE-10943
URL: https://issues.apache.org/jira/browse/HBASE-10943
Project: HBase
Issue Type: Improvement
Affects Versions: 0.94.18
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
Attachments: HBASE-10943-0.94-v1.diff
See HBASE-7329
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10801) Ensure DBE interfaces can work with Cell
[
https://issues.apache.org/jira/browse/HBASE-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965195#comment-13965195
]
Hadoop QA commented on HBASE-10801:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639546/HBASE-10801_2.patch
against trunk revision .
ATTACHMENT ID: 12639546
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ currentStateMembers.qualifierOffset =
currentStateMembers.familyOffset + currentStateMembers.familyLength;
+ - (int)
KeyValue.getKeyDataStructureSize(currentStateMembers.rowLength,
currentStateMembers.familyLength, 0);
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestRegionFavoredNodes
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9244//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9244//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9244//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9244//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9244//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9244//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9244//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9244//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9244//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9244//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9244//console
This message is automatically generated.
Ensure DBE interfaces can work with Cell
Key: HBASE-10801
URL: https://issues.apache.org/jira/browse/HBASE-10801
Project: HBase
Issue Type: Sub-task
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0
Attachments: HBASE-10801.patch, HBASE-10801_1.patch,
HBASE-10801_2.patch
Some changes to the interfaces may be needed for DBEs or may be the way it
works currently may be need to be modified inorder to make DBEs work with
Cells. Suggestions and ideas welcome.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-6618) Implement FuzzyRowFilter with ranges support
[
https://issues.apache.org/jira/browse/HBASE-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965203#comment-13965203
]
Igor Kuzmitshov commented on HBASE-6618:
[~alexb], you are right about keeping the mask separate, somehow I forgot that
? can be a “normal byte”, sorry.
I have just checked other Filters, it seems that all are quite low-level and
use byte arrays as constructor parameters. It makes sense to use byte arrays as
parameters to be consistent, but adding a builder could be nice as well.
For me, the biggest “inconvenience” (especially when using HBase shell) of
constructing a FuzzyRowFilter is not in byte arrays themselves, but in Lists of
Pairs (or Triples) of byte arrays. I would add a simpler constructor for one
rule (I guess one rule would be enough quite often) and a separate method to
add rules:
{code}
FuzzyRowFilter(byte[] fuzzyInfo, byte[] lowerBytes, byte[] upperBytes)
void addRule(byte[] fuzzyInfo, byte[] lowerBytes, byte[] upperBytes)
{code}
Implement FuzzyRowFilter with ranges support
Key: HBASE-6618
URL: https://issues.apache.org/jira/browse/HBASE-6618
Project: HBase
Issue Type: New Feature
Components: Filters
Reporter: Alex Baranau
Assignee: Alex Baranau
Priority: Minor
Fix For: 0.99.0
Attachments: HBASE-6618-algo-desc-bits.png, HBASE-6618-algo.patch,
HBASE-6618.patch, HBASE-6618_2.path, HBASE-6618_3.path, HBASE-6618_4.patch,
HBASE-6618_5.patch
Apart from current ability to specify fuzzy row filter e.g. for
userId_actionId format as _0004 (where 0004 - actionId) it would be
great to also have ability to specify the fuzzy range , e.g. _0004,
..., _0099.
See initial discussion here: http://search-hadoop.com/m/WVLJdX0Z65
Note: currently it is possible to provide multiple fuzzy row rules to
existing FuzzyRowFilter, but in case when the range is big (contains
thousands of values) it is not efficient.
Filter should perform efficient fast-forwarding during the scan (this is what
distinguishes it from regex row filter).
While such functionality may seem like a proper fit for custom filter (i.e.
not including into standard filter set) it looks like the filter may be very
re-useable. We may judge based on the implementation that will hopefully be
added.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10947) Allow subclassing of HTable and HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-10947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965222#comment-13965222 ] Enis Soztutar commented on HBASE-10947: --- Actually, we want to move away from HTable() constructors altogether and move into getting the table object from connection. Issues HBASE-10602, HBASE-10934 and HBASE-9117 contains some details. We may rethink the MR APIs as well. Allow subclassing of HTable and HBaseAdmin -- Key: HBASE-10947 URL: https://issues.apache.org/jira/browse/HBASE-10947 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan To extend functionality of HTable and HBaseAdmin we may need to subclass them. This JIRA allows to add a default constructor and probably remove the final variables in them so that we could subclass them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10801) Ensure DBE interfaces can work with Cell
[ https://issues.apache.org/jira/browse/HBASE-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10801: --- Status: Open (was: Patch Available) Ensure DBE interfaces can work with Cell Key: HBASE-10801 URL: https://issues.apache.org/jira/browse/HBASE-10801 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10801.patch, HBASE-10801_1.patch, HBASE-10801_2.patch Some changes to the interfaces may be needed for DBEs or may be the way it works currently may be need to be modified inorder to make DBEs work with Cells. Suggestions and ideas welcome. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10801) Ensure DBE interfaces can work with Cell
[ https://issues.apache.org/jira/browse/HBASE-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10801: --- Attachment: HBASE-10801_3.patch [~mcorgan] Could you take a look at this patch. I am not sure if we can change the logic of how the hfs.next() is getting called. Generally the next KV is also fetched and the comparison is done to load the next store file if the key is already seeked key is already bigger. Changing that would be a bigger task too. So for now I have done a work around way where only the key is deepcloned whereas the value is not copied just referred to the actual underlying array. (This is the power of Cell). So in cases where the value part is bigger we don't do any copy of that and this would help in all the comparisons and the fetching of the exact KV and the copy happens at the last stage where we need KVs. Two things Instead of deep copying of the value[] we create other objects and need to check how much is that a overhead. Because our code still works with KV as an end result, the key copying would happen twice - once in the BDE and other when the KeyValueUtil.ensureKeyValue is called. Ensure DBE interfaces can work with Cell Key: HBASE-10801 URL: https://issues.apache.org/jira/browse/HBASE-10801 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10801.patch, HBASE-10801_1.patch, HBASE-10801_2.patch, HBASE-10801_3.patch Some changes to the interfaces may be needed for DBEs or may be the way it works currently may be need to be modified inorder to make DBEs work with Cells. Suggestions and ideas welcome. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10801) Ensure DBE interfaces can work with Cell
[ https://issues.apache.org/jira/browse/HBASE-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10801: --- Status: Patch Available (was: Open) Ensure DBE interfaces can work with Cell Key: HBASE-10801 URL: https://issues.apache.org/jira/browse/HBASE-10801 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10801.patch, HBASE-10801_1.patch, HBASE-10801_2.patch, HBASE-10801_3.patch Some changes to the interfaces may be needed for DBEs or may be the way it works currently may be need to be modified inorder to make DBEs work with Cells. Suggestions and ideas welcome. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10934) Provide HBaseAdminInterface to abstract HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-10934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965309#comment-13965309 ] Nicolas Liochon commented on HBASE-10934: - bq. If we go the interface way, I think we should document that although the API would be public, we might add methods freely. +1 to use interface, and +1 to document that we can add methods at any time. Interfaces are useful because they are easily proxied. Provide HBaseAdminInterface to abstract HBaseAdmin -- Key: HBASE-10934 URL: https://issues.apache.org/jira/browse/HBASE-10934 Project: HBase Issue Type: Improvement Components: Client Reporter: Carter Priority: Blocker Fix For: 0.99.0 As HBaseAdmin is essentially the administrative API, it would seem to follow Java best practices to provide an interface to access it instead of requiring applications to use the raw object. I am proposing (and would be happy to develop): * A new interface, HBaseAdminInterface, that captures the signatures of the API (HBaseAdmin will implement this interface) * A new method, HConnection.getHBaseAdmin(), that returns an instance of the interface -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10934) Provide HBaseAdminInterface to abstract HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-10934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965387#comment-13965387 ] Carter commented on HBASE-10934: As a developer, I would expect new methods to appear as versions progress and the interface is extended -- as long as nothing breaks backwards compatibility between major releases. I would also be a little puzzled if I found an API defined with an abstract class but not an interface. Provide HBaseAdminInterface to abstract HBaseAdmin -- Key: HBASE-10934 URL: https://issues.apache.org/jira/browse/HBASE-10934 Project: HBase Issue Type: Improvement Components: Client Reporter: Carter Priority: Blocker Fix For: 0.99.0 As HBaseAdmin is essentially the administrative API, it would seem to follow Java best practices to provide an interface to access it instead of requiring applications to use the raw object. I am proposing (and would be happy to develop): * A new interface, HBaseAdminInterface, that captures the signatures of the API (HBaseAdmin will implement this interface) * A new method, HConnection.getHBaseAdmin(), that returns an instance of the interface -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10951) Use PBKDF2 to generate test encryption keys in the shell
[ https://issues.apache.org/jira/browse/HBASE-10951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965388#comment-13965388 ] Andrew Purtell commented on HBASE-10951: No compatability issues that I can see given this isn't the way to generate data keys for production. This is so one can use the shell to create a schema with all encryption related attributes set up properly for basic functional testing or integration tests. Use PBKDF2 to generate test encryption keys in the shell Key: HBASE-10951 URL: https://issues.apache.org/jira/browse/HBASE-10951 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Trivial Fix For: 0.99.0, 0.98.2 Attachments: HBASE-10951.patch We provide some support in the shell for setting the column family data encryption key, which enables some simple testing when kicking the tires. (CF data key management should be done using the Java API.) Despite the very modest goal there might be an objection to using a hash instead of a key derivation function, so just go ahead and do that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10813) Possible over-catch of exceptions
[
https://issues.apache.org/jira/browse/HBASE-10813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965392#comment-13965392
]
Nicolas Liochon commented on HBASE-10813:
-
Thanks for the fix, Michael. Thanks for committing it Stack.
Possible over-catch of exceptions
-
Key: HBASE-10813
URL: https://issues.apache.org/jira/browse/HBASE-10813
Project: HBase
Issue Type: Improvement
Components: regionserver, util
Affects Versions: 0.96.1
Reporter: Ding Yuan
Assignee: Ding Yuan
Fix For: 0.99.0
Attachments: HBASE-10813_trunk_fixed_tests.patch,
HBase-10813-trunk.patch
There are a few cases found by a tool that are possibly over-catch of
exceptions, especially those that will abort the server. Over-catching these
exceptions may unexpectedly abort the server, and may cause problems in the
future when code in the try-block evolves. I am attaching a patch against
trunk that constrains the catch blocks to the exact exceptions that were
thrown.
My tool actually found one more case in 0.96.1, but I found it has already
been fixed in trunk:
{noformat}
Line: 1175, File: org/apache/hadoop/hbase/master/SplitLogManager.java
1173: try {
1174: Thread.sleep(20);
1175:-} catch (Exception ignoreE) {
1175:+ } catch (InterruptedException e) {
1176: // ignore
1177: }
{noformat}
Any feedbacks are much appreciated!
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10813) Possible over-catch of exceptions
[
https://issues.apache.org/jira/browse/HBASE-10813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965397#comment-13965397
]
Ding Yuan commented on HBASE-10813:
---
Thanks all!
Possible over-catch of exceptions
-
Key: HBASE-10813
URL: https://issues.apache.org/jira/browse/HBASE-10813
Project: HBase
Issue Type: Improvement
Components: regionserver, util
Affects Versions: 0.96.1
Reporter: Ding Yuan
Assignee: Ding Yuan
Fix For: 0.99.0
Attachments: HBASE-10813_trunk_fixed_tests.patch,
HBase-10813-trunk.patch
There are a few cases found by a tool that are possibly over-catch of
exceptions, especially those that will abort the server. Over-catching these
exceptions may unexpectedly abort the server, and may cause problems in the
future when code in the try-block evolves. I am attaching a patch against
trunk that constrains the catch blocks to the exact exceptions that were
thrown.
My tool actually found one more case in 0.96.1, but I found it has already
been fixed in trunk:
{noformat}
Line: 1175, File: org/apache/hadoop/hbase/master/SplitLogManager.java
1173: try {
1174: Thread.sleep(20);
1175:-} catch (Exception ignoreE) {
1175:+ } catch (InterruptedException e) {
1176: // ignore
1177: }
{noformat}
Any feedbacks are much appreciated!
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HBASE-10955) HBCK leaves the region in masters in-memory RegionStates if region hdfs dir is lost
Enis Soztutar created HBASE-10955:
-
Summary: HBCK leaves the region in masters in-memory RegionStates
if region hdfs dir is lost
Key: HBASE-10955
URL: https://issues.apache.org/jira/browse/HBASE-10955
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.99.0, 0.98.2
One of our tests removes the hdfs directory for the region, and invokes HBCK to
fix the issue. This test fails flakily because the region is removed from meta
and unassigned, but the region is not offlined from the masters in-memory. This
affects further LB runs and disable table, etc.
In case of {{inMeta !inHdfs isDeployed}}, we should not just close the
region from RS, but call master.unassign().
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10801) Ensure DBE interfaces can work with Cell
[
https://issues.apache.org/jira/browse/HBASE-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965402#comment-13965402
]
Hadoop QA commented on HBASE-10801:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639565/HBASE-10801_3.patch
against trunk revision .
ATTACHMENT ID: 12639565
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9246//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9246//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9246//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9246//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9246//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9246//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9246//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9246//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9246//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9246//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9246//console
This message is automatically generated.
Ensure DBE interfaces can work with Cell
Key: HBASE-10801
URL: https://issues.apache.org/jira/browse/HBASE-10801
Project: HBase
Issue Type: Sub-task
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0
Attachments: HBASE-10801.patch, HBASE-10801_1.patch,
HBASE-10801_2.patch, HBASE-10801_3.patch
Some changes to the interfaces may be needed for DBEs or may be the way it
works currently may be need to be modified inorder to make DBEs work with
Cells. Suggestions and ideas welcome.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10956) Upgrade hadoop-2 dependency to 2.4.0
[ https://issues.apache.org/jira/browse/HBASE-10956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-10956: --- Attachment: 10956-v1.txt Upgrade hadoop-2 dependency to 2.4.0 Key: HBASE-10956 URL: https://issues.apache.org/jira/browse/HBASE-10956 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Attachments: 10956-v1.txt Hadoop 2.4.0 has been released: http://search-hadoop.com/m/LgpTk2YKhUf This JIRA is to upgrade hadoop-2 dependency to 2.4.0 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-10956) Upgrade hadoop-2 dependency to 2.4.0
Ted Yu created HBASE-10956: -- Summary: Upgrade hadoop-2 dependency to 2.4.0 Key: HBASE-10956 URL: https://issues.apache.org/jira/browse/HBASE-10956 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Attachments: 10956-v1.txt Hadoop 2.4.0 has been released: http://search-hadoop.com/m/LgpTk2YKhUf This JIRA is to upgrade hadoop-2 dependency to 2.4.0 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10956) Upgrade hadoop-2 dependency to 2.4.0
[ https://issues.apache.org/jira/browse/HBASE-10956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-10956: --- Status: Patch Available (was: Open) Upgrade hadoop-2 dependency to 2.4.0 Key: HBASE-10956 URL: https://issues.apache.org/jira/browse/HBASE-10956 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Attachments: 10956-v1.txt Hadoop 2.4.0 has been released: http://search-hadoop.com/m/LgpTk2YKhUf This JIRA is to upgrade hadoop-2 dependency to 2.4.0 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10955) HBCK leaves the region in masters in-memory RegionStates if region hdfs dir is lost
[
https://issues.apache.org/jira/browse/HBASE-10955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-10955:
--
Attachment: hbase-10955.v1.patch
This fixes the condition defined above. Also we no longer create the HLog
unnecessarily.
HBCK leaves the region in masters in-memory RegionStates if region hdfs dir
is lost
---
Key: HBASE-10955
URL: https://issues.apache.org/jira/browse/HBASE-10955
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.99.0, 0.98.2
Attachments: hbase-10955.v1.patch
One of our tests removes the hdfs directory for the region, and invokes HBCK
to fix the issue. This test fails flakily because the region is removed from
meta and unassigned, but the region is not offlined from the masters
in-memory. This affects further LB runs and disable table, etc.
In case of {{inMeta !inHdfs isDeployed}}, we should not just close the
region from RS, but call master.unassign().
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10785) Metas own location should be cached
[
https://issues.apache.org/jira/browse/HBASE-10785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965409#comment-13965409
]
Enis Soztutar commented on HBASE-10785:
---
I think I've got +1s for earlier versions. The v3 does not add anything new.
I'll commit this tomorrow unless objection.
Metas own location should be cached
---
Key: HBASE-10785
URL: https://issues.apache.org/jira/browse/HBASE-10785
Project: HBase
Issue Type: Improvement
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: hbase-10070
Attachments: hbase-10785_v1.patch, hbase-10785_v2.patch,
hbase-10785_v3.patch
With ROOT table gone, we no longer cache the location of the meta table (in
MetaCache) in 96+. I've checked 94 code, and there we cache meta, but not
root.
However, not caching the metas own location means that we are doing a
zookeeper request every time we want to look up a regions location from meta.
This means that there is a significant spike in zk requests whenever a region
server goes down.
This affects trunk,0.98 and 0.96 as well as hbase-10070 branch. I've
discovered the issue in hbase-10070 because of the integration test
(HBASE-10572) results in 150K requests to zk in 10min.
A thread dump from one of the runs have 100+ threads from client in this
stack trace:
{code}
TimeBoundedMultiThreadedReaderThread_20 prio=10
tid=0x7f852c2f2000 nid=0x57b6 in Object.wait() [0x7f85059e7000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at
org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309)
- locked 0xea71aa78 (a
org.apache.zookeeper.ClientCnxn$Packet)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1149)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:337)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:684)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.blockUntilAvailable(ZKUtil.java:1853)
at
org.apache.hadoop.hbase.zookeeper.MetaRegionTracker.blockUntilAvailable(MetaRegionTracker.java:186)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:60)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1126)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1112)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1220)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1129)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:321)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.call(RpcRetryingCallerWithReadReplicas.java:257)
- locked 0xe9bcf238 (a
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:818)
at
org.apache.hadoop.hbase.util.MultiThreadedReader$HBaseReaderThread.queryKey(MultiThreadedReader.java:288)
at
org.apache.hadoop.hbase.util.MultiThreadedReader$HBaseReaderThread.readKey(MultiThreadedReader.java:249)
at
org.apache.hadoop.hbase.util.MultiThreadedReader$HBaseReaderThread.runReader(MultiThreadedReader.java:192)
at
org.apache.hadoop.hbase.util.MultiThreadedReader$HBaseReaderThread.run(MultiThreadedReader.java:150)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10956) Upgrade hadoop-2 dependency to 2.4.0
[
https://issues.apache.org/jira/browse/HBASE-10956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965510#comment-13965510
]
Hadoop QA commented on HBASE-10956:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639585/10956-v1.txt
against trunk revision .
ATTACHMENT ID: 12639585
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9247//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9247//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9247//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9247//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9247//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9247//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9247//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9247//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9247//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9247//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9247//console
This message is automatically generated.
Upgrade hadoop-2 dependency to 2.4.0
Key: HBASE-10956
URL: https://issues.apache.org/jira/browse/HBASE-10956
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 10956-v1.txt
Hadoop 2.4.0 has been released:
http://search-hadoop.com/m/LgpTk2YKhUf
This JIRA is to upgrade hadoop-2 dependency to 2.4.0
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10884) [REST] Do not disable block caching when scanning
[ https://issues.apache.org/jira/browse/HBASE-10884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965513#comment-13965513 ] Andrew Purtell commented on HBASE-10884: Test failure is not related, will commit soon. Ping [~stack] and [~lhofhansl], I'm going to assume no objection to trivial patch and perf improvement, but limited to REST use cases. [REST] Do not disable block caching when scanning - Key: HBASE-10884 URL: https://issues.apache.org/jira/browse/HBASE-10884 Project: HBase Issue Type: Improvement Affects Versions: 0.98.1, 0.96.1.1, 0.94.18 Reporter: Andrew Purtell Assignee: Andrew Purtell Attachments: HBASE-10884.patch The REST gateway pessimistically disables block caching when issuing Scans to the cluster, using Scan#setCacheBlocks(false) in ScannerResultGenerator. It does not do this when issuing Gets on behalf of HTTP clients in RowResultGenerator. This is an old idea now, the reasons for doing so lost sometime back in the era when HBase walked the earth with dinosaurs ( 0.20). We probably should not be penalizing REST scans in this way. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10952) [REST] Let the user turn off block caching if desired
[ https://issues.apache.org/jira/browse/HBASE-10952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965514#comment-13965514 ] Andrew Purtell commented on HBASE-10952: Patch soon. Ping [~stack] and [~lhofhansl], companion issue to HBASE-10884. [REST] Let the user turn off block caching if desired - Key: HBASE-10952 URL: https://issues.apache.org/jira/browse/HBASE-10952 Project: HBase Issue Type: Improvement Affects Versions: 0.98.1, 0.99.0, 0.94.19, 0.96.3 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor After HBASE-10884 the REST gateway will use scanner defaults with respect to block caching. Add support for a query parameter for hinting blocks for the query should not be cached. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10951) Use PBKDF2 to generate test encryption keys in the shell
[ https://issues.apache.org/jira/browse/HBASE-10951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10951: --- Affects Version/s: 0.99.0 0.98.1 Status: Patch Available (was: Open) Use PBKDF2 to generate test encryption keys in the shell Key: HBASE-10951 URL: https://issues.apache.org/jira/browse/HBASE-10951 Project: HBase Issue Type: Improvement Affects Versions: 0.98.1, 0.99.0 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Trivial Fix For: 0.99.0, 0.98.2 Attachments: HBASE-10951.patch We provide some support in the shell for setting the column family data encryption key, which enables some simple testing when kicking the tires. (CF data key management should be done using the Java API.) Despite the very modest goal there might be an objection to using a hash instead of a key derivation function, so just go ahead and do that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-10957) HBASE-10070: HMaster can abort with NPE in #rebuildUserRegions
Nicolas Liochon created HBASE-10957: --- Summary: HBASE-10070: HMaster can abort with NPE in #rebuildUserRegions Key: HBASE-10957 URL: https://issues.apache.org/jira/browse/HBASE-10957 Project: HBase Issue Type: Bug Components: master Affects Versions: hbase-10070 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: hbase-10070 Seen during tests. The fix is to test this condition as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10957) HBASE-10070: HMaster can abort with NPE in #rebuildUserRegions
[ https://issues.apache.org/jira/browse/HBASE-10957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-10957: Attachment: 10957.v1.patch HBASE-10070: HMaster can abort with NPE in #rebuildUserRegions --- Key: HBASE-10957 URL: https://issues.apache.org/jira/browse/HBASE-10957 Project: HBase Issue Type: Bug Components: master Affects Versions: hbase-10070 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: hbase-10070 Attachments: 10957.v1.patch Seen during tests. The fix is to test this condition as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10900) FULL table backup and restore
[
https://issues.apache.org/jira/browse/HBASE-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965518#comment-13965518
]
Ted Yu commented on HBASE-10900:
For the first question above, I think hbase-client module would be Okay.
I don't have strong opinion for second question.
FULL table backup and restore
-
Key: HBASE-10900
URL: https://issues.apache.org/jira/browse/HBASE-10900
Project: HBase
Issue Type: Task
Reporter: Demai Ni
Assignee: Demai Ni
Fix For: 1.0.0
Attachments: HBASE-10900-fullbackup-trunk-v1.patch
h2. Feature Description
This is a subtask of
[HBase-7912|https://issues.apache.org/jira/browse/HBASE-7912] to support FULL
backup/restore, and will complete the following function:
{code:title=Backup Restore example|borderStyle=solid}
/* backup from sourcecluster to targetcluster
*/
/* if no table name specified, all tables from source cluster will be
backuped */
[sourcecluster]$ hbase backup create full
hdfs://hostname.targetcluster.org:9000/userid/backupdir t1_dn,t2_dn,t3_dn
/* restore on targetcluser, this is a local restore
*/
/* backup_1396650096738 - backup image name
*/
/* t1_dn,etc are the original table names. All tables will be restored if not
specified */
/* t1_dn_restore, etc. are the restored table. if not specified, orginal
table name will be used*/
[targetcluster]$ hbase restore /userid/backupdir backup_1396650096738
t1_dn,t2_dn,t3_dn t1_dn_restore,t2_dn_restore,t3_dn_restore
/* restore from targetcluster back to source cluster, this is a remote restore
[sourcecluster]$ hbase restore
hdfs://hostname.targetcluster.org:9000/userid/backupdir backup_1396650096738
t1_dn,t2_dn,t3_dn t1_dn_restore,t2_dn_restore,t3_dn_restore
{code}
h2. Detail layout and frame work for the next jiras
The patch is a wrapper of the existing snapshot and exportSnapshot, and will
use as the base framework for the over-all solution of
[HBase-7912|https://issues.apache.org/jira/browse/HBASE-7912] as described
below:
* *bin/hbase* : end-user command line interface to invoke
BackupClient and RestoreClient
* *BackupClient.java* : 'main' entry for backup operations. This patch will
only support 'full' backup. In future jiras, will support:
** *create* incremental backup
** *cancel* an ongoing backup
** *delete* an exisitng backup image
** *describe* the detailed informaiton of backup image
** show *history* of all successful backups
** show the *status* of the latest backup request
** *convert* incremental backup WAL files into HFiles. either on-the-fly
during create or after create
** *merge* backup image
** *stop* backup a table of existing backup image
** *show* tables of a backup image
* *BackupCommands.java* : a place to keep all the command usages and options
* *BackupManager.java* : handle backup requests on server-side, create
BACKUP ZOOKEEPER nodes to keep track backup. The timestamps kept in zookeeper
will be used for future incremental backup (not included in this jira).
Create BackupContext and DispatchRequest.
* *BackupHandler.java* : in this patch, it is a wrapper of snapshot and
exportsnapshot. In future jiras,
** *timestamps* info will be recorded in ZK
** carry on *incremental* backup.
** update backup *progress*
** set flags of *status*
** build up *backupManifest* file(in this jira only limited info for
fullback. later on, timestamps and dependency of multipl backup images are
also recorded here)
** clean up after *failed* backup
** clean up after *cancelled* backup
** allow on-the-fly *convert* during incremental backup
* *BackupContext.java* : encapsulate backup information like backup ID, table
names, directory info, phase, TimeStamps of backup progress, size of data,
ancestor info, etc.
* *BackupCopier.java* : the copying operation. Later on, to support
progress report and mapper estimation; and extends DisCp for progress
updating to ZK during backup.
* *BackupExcpetion.java*: to handle exception from backup/restore
* *BackupManifest.java* : encapsulate all the backup image information. The
manifest info will be bundled as manifest file together with data. So that
each backup image will contain all the info needed for restore.
* *BackupStatus.java* : encapsulate backup status at table level during
backup progress
* *BackupUtil.java* : utility methods during backup process
* *RestoreClient.java* : 'main' entry for restore operations. This patch
will only support 'full' backup.
* *RestoreUtil.java*: utility methods during restore process
* *ExportSnapshot.java* : remove 'final' so that another
[jira] [Commented] (HBASE-10957) HBASE-10070: HMaster can abort with NPE in #rebuildUserRegions
[ https://issues.apache.org/jira/browse/HBASE-10957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965520#comment-13965520 ] Nicolas Liochon commented on HBASE-10957: - (for hbase-10070 only) HBASE-10070: HMaster can abort with NPE in #rebuildUserRegions --- Key: HBASE-10957 URL: https://issues.apache.org/jira/browse/HBASE-10957 Project: HBase Issue Type: Bug Components: master Affects Versions: hbase-10070 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: hbase-10070 Attachments: 10957.v1.patch Seen during tests. The fix is to test this condition as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10953) Remove unused @private SimpleBlockCache (and RandomSeek tool that uses it)
[ https://issues.apache.org/jira/browse/HBASE-10953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965531#comment-13965531 ] Nick Dimiduk commented on HBASE-10953: -- +1 Remove unused @private SimpleBlockCache (and RandomSeek tool that uses it) -- Key: HBASE-10953 URL: https://issues.apache.org/jira/browse/HBASE-10953 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Attachments: 10953.txt Tripped over these today. Remove. Not used. @Private -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10070) HBase read high-availability using eventually consistent region replicas
[ https://issues.apache.org/jira/browse/HBASE-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965533#comment-13965533 ] Tianying Chang commented on HBASE-10070: [~enis] I am interested in the feature as a whole. I can see Get a single row is already supported. How about the get(listGet)? I seems cannot find it. Will that be supported later together with scan in phase 1? HBase read high-availability using eventually consistent region replicas Key: HBASE-10070 URL: https://issues.apache.org/jira/browse/HBASE-10070 Project: HBase Issue Type: New Feature Reporter: Enis Soztutar Assignee: Enis Soztutar Attachments: HighAvailabilityDesignforreadsApachedoc.pdf In the present HBase architecture, it is hard, probably impossible, to satisfy constraints like 99th percentile of the reads will be served under 10 ms. One of the major factors that affects this is the MTTR for regions. There are three phases in the MTTR process - detection, assignment, and recovery. Of these, the detection is usually the longest and is presently in the order of 20-30 seconds. During this time, the clients would not be able to read the region data. However, some clients will be better served if regions will be available for reads during recovery for doing eventually consistent reads. This will help with satisfying low latency guarantees for some class of applications which can work with stale reads. For improving read availability, we propose a replicated read-only region serving design, also referred as secondary regions, or region shadows. Extending current model of a region being opened for reads and writes in a single region server, the region will be also opened for reading in region servers. The region server which hosts the region for reads and writes (as in current case) will be declared as PRIMARY, while 0 or more region servers might be hosting the region as SECONDARY. There may be more than one secondary (replica count 2). Will attach a design doc shortly which contains most of the details and some thoughts about development approaches. Reviews are more than welcome. We also have a proof of concept patch, which includes the master and regions server side of changes. Client side changes will be coming soon as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10951) Use PBKDF2 to generate test encryption keys in the shell
[ https://issues.apache.org/jira/browse/HBASE-10951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10951: --- Status: Open (was: Patch Available) Use PBKDF2 to generate test encryption keys in the shell Key: HBASE-10951 URL: https://issues.apache.org/jira/browse/HBASE-10951 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Trivial Fix For: 0.99.0, 0.98.2 Attachments: HBASE-10951.patch We provide some support in the shell for setting the column family data encryption key, which enables some simple testing when kicking the tires. (CF data key management should be done using the Java API.) Despite the very modest goal there might be an objection to using a hash instead of a key derivation function, so just go ahead and do that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10897) On master start, deadlock if refresh UI
[
https://issues.apache.org/jira/browse/HBASE-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965535#comment-13965535
]
Jimmy Xiang commented on HBASE-10897:
-
{quote}
Is this right?
+ if (master.getMasterCoprocessorHost() == null) {
What do CPs have to do w/ this?
{quote}
It doesn't look right. Let me do it in a different way.
{quote}
Has to be public?
+ public boolean isCatalogJanitorEnabled() {
{quote}
I will make it packaged protected.
On master start, deadlock if refresh UI
---
Key: HBASE-10897
URL: https://issues.apache.org/jira/browse/HBASE-10897
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: stack
Assignee: Jimmy Xiang
Fix For: 0.99.0
Attachments: hbase-10897.patch, hbase-10897_v2.patch,
hbase-10897_v3.patch
Playing w/ MTTR recovery on trunk, master starting up deadlocked:
Waiting to finish active master initialization:
{code}
ActiveMasterManager daemon prio=10 tid=0x7fafb5dc3800 nid=0x5fb5
waiting for monitor entry [0x7faf8f57d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1683)
- waiting to lock 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:53)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1029)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:989)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:830)
at
org.apache.hadoop.hbase.client.ConnectionAdapter.getRegionLocation(ConnectionAdapter.java:305)
at
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:77)
at
org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:118)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:101)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:169)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:164)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:107)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:766)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:539)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScanOfMeta(MetaReader.java:140)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.isMetaTableUpdated(MetaMigrationConvertingToPB.java:164)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.updateMetaIfNecessary(MetaMigrationConvertingToPB.java:131)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:567)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:147)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1242)
at java.lang.Thread.run(Thread.java:744)
{code}
... but the master servlet has the lock while trying to access master:
{code}
686004346@qtp-2101021459-0 daemon prio=10 tid=0x7fafb5d2a800 nid=0x5fb1
waiting on condition [0x7faf8f87f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1562)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1597)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1805)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.listTables(ConnectionManager.java:2481)
at
org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:321)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.__jamon_innerUnit__userTables(MasterStatusTmplImpl.java:530)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:255)
at
[jira] [Updated] (HBASE-10957) HBASE-10070: HMaster can abort with NPE in #rebuildUserRegions
[ https://issues.apache.org/jira/browse/HBASE-10957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-10957: Issue Type: Sub-task (was: Bug) Parent: HBASE-10070 HBASE-10070: HMaster can abort with NPE in #rebuildUserRegions --- Key: HBASE-10957 URL: https://issues.apache.org/jira/browse/HBASE-10957 Project: HBase Issue Type: Sub-task Components: master Affects Versions: hbase-10070 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: hbase-10070 Attachments: 10957.v1.patch Seen during tests. The fix is to test this condition as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-6618) Implement FuzzyRowFilter with ranges support
[ https://issues.apache.org/jira/browse/HBASE-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965545#comment-13965545 ] Alex Baranau commented on HBASE-6618: - totally agree on overloading ctor. Will add that. Also will see if Builder makes sense: it'd help with these lists of pairs/triples. Thank you for looking at the patch, [~kuzmiigo]! Implement FuzzyRowFilter with ranges support Key: HBASE-6618 URL: https://issues.apache.org/jira/browse/HBASE-6618 Project: HBase Issue Type: New Feature Components: Filters Reporter: Alex Baranau Assignee: Alex Baranau Priority: Minor Fix For: 0.99.0 Attachments: HBASE-6618-algo-desc-bits.png, HBASE-6618-algo.patch, HBASE-6618.patch, HBASE-6618_2.path, HBASE-6618_3.path, HBASE-6618_4.patch, HBASE-6618_5.patch Apart from current ability to specify fuzzy row filter e.g. for userId_actionId format as _0004 (where 0004 - actionId) it would be great to also have ability to specify the fuzzy range , e.g. _0004, ..., _0099. See initial discussion here: http://search-hadoop.com/m/WVLJdX0Z65 Note: currently it is possible to provide multiple fuzzy row rules to existing FuzzyRowFilter, but in case when the range is big (contains thousands of values) it is not efficient. Filter should perform efficient fast-forwarding during the scan (this is what distinguishes it from regex row filter). While such functionality may seem like a proper fit for custom filter (i.e. not including into standard filter set) it looks like the filter may be very re-useable. We may judge based on the implementation that will hopefully be added. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10070) HBase read high-availability using eventually consistent region replicas
[ https://issues.apache.org/jira/browse/HBASE-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965556#comment-13965556 ] Sergey Shelukhin commented on HBASE-10070: -- list get is also supported, although some changes need to be committed to address corner cases, like HBASE-10794 HBase read high-availability using eventually consistent region replicas Key: HBASE-10070 URL: https://issues.apache.org/jira/browse/HBASE-10070 Project: HBase Issue Type: New Feature Reporter: Enis Soztutar Assignee: Enis Soztutar Attachments: HighAvailabilityDesignforreadsApachedoc.pdf In the present HBase architecture, it is hard, probably impossible, to satisfy constraints like 99th percentile of the reads will be served under 10 ms. One of the major factors that affects this is the MTTR for regions. There are three phases in the MTTR process - detection, assignment, and recovery. Of these, the detection is usually the longest and is presently in the order of 20-30 seconds. During this time, the clients would not be able to read the region data. However, some clients will be better served if regions will be available for reads during recovery for doing eventually consistent reads. This will help with satisfying low latency guarantees for some class of applications which can work with stale reads. For improving read availability, we propose a replicated read-only region serving design, also referred as secondary regions, or region shadows. Extending current model of a region being opened for reads and writes in a single region server, the region will be also opened for reading in region servers. The region server which hosts the region for reads and writes (as in current case) will be declared as PRIMARY, while 0 or more region servers might be hosting the region as SECONDARY. There may be more than one secondary (replica count 2). Will attach a design doc shortly which contains most of the details and some thoughts about development approaches. Reviews are more than welcome. We also have a proof of concept patch, which includes the master and regions server side of changes. Client side changes will be coming soon as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10897) On master start, deadlock if refresh UI
[
https://issues.apache.org/jira/browse/HBASE-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-10897:
Status: Open (was: Patch Available)
On master start, deadlock if refresh UI
---
Key: HBASE-10897
URL: https://issues.apache.org/jira/browse/HBASE-10897
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: stack
Assignee: Jimmy Xiang
Fix For: 0.99.0
Attachments: hbase-10897.patch, hbase-10897_v2.patch,
hbase-10897_v3.patch
Playing w/ MTTR recovery on trunk, master starting up deadlocked:
Waiting to finish active master initialization:
{code}
ActiveMasterManager daemon prio=10 tid=0x7fafb5dc3800 nid=0x5fb5
waiting for monitor entry [0x7faf8f57d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1683)
- waiting to lock 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:53)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1029)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:989)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:830)
at
org.apache.hadoop.hbase.client.ConnectionAdapter.getRegionLocation(ConnectionAdapter.java:305)
at
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:77)
at
org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:118)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:101)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:169)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:164)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:107)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:766)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:539)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScanOfMeta(MetaReader.java:140)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.isMetaTableUpdated(MetaMigrationConvertingToPB.java:164)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.updateMetaIfNecessary(MetaMigrationConvertingToPB.java:131)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:567)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:147)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1242)
at java.lang.Thread.run(Thread.java:744)
{code}
... but the master servlet has the lock while trying to access master:
{code}
686004346@qtp-2101021459-0 daemon prio=10 tid=0x7fafb5d2a800 nid=0x5fb1
waiting on condition [0x7faf8f87f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1562)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1597)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1805)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.listTables(ConnectionManager.java:2481)
at
org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:321)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.__jamon_innerUnit__userTables(MasterStatusTmplImpl.java:530)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:255)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:382)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:372)
at
org.apache.hadoop.hbase.master.MasterStatusServlet.doGet(MasterStatusServlet.java:102)
...
{code}
--
[jira] [Updated] (HBASE-10897) On master start, deadlock if refresh UI
[
https://issues.apache.org/jira/browse/HBASE-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-10897:
Attachment: hbase-10897_v4.patch
Attached v4 that has the fixes mentioned right above.
On master start, deadlock if refresh UI
---
Key: HBASE-10897
URL: https://issues.apache.org/jira/browse/HBASE-10897
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: stack
Assignee: Jimmy Xiang
Fix For: 0.99.0
Attachments: hbase-10897.patch, hbase-10897_v2.patch,
hbase-10897_v3.patch, hbase-10897_v4.patch
Playing w/ MTTR recovery on trunk, master starting up deadlocked:
Waiting to finish active master initialization:
{code}
ActiveMasterManager daemon prio=10 tid=0x7fafb5dc3800 nid=0x5fb5
waiting for monitor entry [0x7faf8f57d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1683)
- waiting to lock 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:53)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1029)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:989)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:830)
at
org.apache.hadoop.hbase.client.ConnectionAdapter.getRegionLocation(ConnectionAdapter.java:305)
at
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:77)
at
org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:118)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:101)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:169)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:164)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:107)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:766)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:539)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScanOfMeta(MetaReader.java:140)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.isMetaTableUpdated(MetaMigrationConvertingToPB.java:164)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.updateMetaIfNecessary(MetaMigrationConvertingToPB.java:131)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:567)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:147)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1242)
at java.lang.Thread.run(Thread.java:744)
{code}
... but the master servlet has the lock while trying to access master:
{code}
686004346@qtp-2101021459-0 daemon prio=10 tid=0x7fafb5d2a800 nid=0x5fb1
waiting on condition [0x7faf8f87f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1562)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1597)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1805)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.listTables(ConnectionManager.java:2481)
at
org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:321)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.__jamon_innerUnit__userTables(MasterStatusTmplImpl.java:530)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:255)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:382)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:372)
at
[jira] [Updated] (HBASE-10897) On master start, deadlock if refresh UI
[
https://issues.apache.org/jira/browse/HBASE-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-10897:
Status: Patch Available (was: Open)
On master start, deadlock if refresh UI
---
Key: HBASE-10897
URL: https://issues.apache.org/jira/browse/HBASE-10897
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: stack
Assignee: Jimmy Xiang
Fix For: 0.99.0
Attachments: hbase-10897.patch, hbase-10897_v2.patch,
hbase-10897_v3.patch, hbase-10897_v4.patch
Playing w/ MTTR recovery on trunk, master starting up deadlocked:
Waiting to finish active master initialization:
{code}
ActiveMasterManager daemon prio=10 tid=0x7fafb5dc3800 nid=0x5fb5
waiting for monitor entry [0x7faf8f57d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1683)
- waiting to lock 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:53)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1029)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:989)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:830)
at
org.apache.hadoop.hbase.client.ConnectionAdapter.getRegionLocation(ConnectionAdapter.java:305)
at
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:77)
at
org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:118)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:101)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:169)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:164)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:107)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:766)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:539)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScanOfMeta(MetaReader.java:140)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.isMetaTableUpdated(MetaMigrationConvertingToPB.java:164)
at
org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.updateMetaIfNecessary(MetaMigrationConvertingToPB.java:131)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:567)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:147)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1242)
at java.lang.Thread.run(Thread.java:744)
{code}
... but the master servlet has the lock while trying to access master:
{code}
686004346@qtp-2101021459-0 daemon prio=10 tid=0x7fafb5d2a800 nid=0x5fb1
waiting on condition [0x7faf8f87f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1562)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1597)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1805)
- locked 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.listTables(ConnectionManager.java:2481)
at
org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:321)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.__jamon_innerUnit__userTables(MasterStatusTmplImpl.java:530)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:255)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:382)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:372)
at
[jira] [Commented] (HBASE-10957) HBASE-10070: HMaster can abort with NPE in #rebuildUserRegions
[ https://issues.apache.org/jira/browse/HBASE-10957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965563#comment-13965563 ] Devaraj Das commented on HBASE-10957: - [~nkeywal], as discussed offline, if we can come up with a sequence of actions that would lead to the HRI read from meta being not parseable (thereby leading to the method MetaReader.getHRegionInfo returning null), it'd be great. HBASE-10070: HMaster can abort with NPE in #rebuildUserRegions --- Key: HBASE-10957 URL: https://issues.apache.org/jira/browse/HBASE-10957 Project: HBase Issue Type: Sub-task Components: master Affects Versions: hbase-10070 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: hbase-10070 Attachments: 10957.v1.patch Seen during tests. The fix is to test this condition as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HBASE-10926) Use global procedure to flush table memstore cache
[ https://issues.apache.org/jira/browse/HBASE-10926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He reassigned HBASE-10926: Assignee: Jerry He Use global procedure to flush table memstore cache -- Key: HBASE-10926 URL: https://issues.apache.org/jira/browse/HBASE-10926 Project: HBase Issue Type: Improvement Components: Admin Affects Versions: 0.96.2, 0.98.1 Reporter: Jerry He Assignee: Jerry He Fix For: 0.99.0 Currently, user can trigger table flush through hbase shell or HBaseAdmin API. To flush the table cache, each region server hosting the regions is contacted and flushed sequentially, which is less efficient. In HBase snapshot global procedure is used to coordinate and flush the regions in a distributed way. Let's provide a distributed table flush for general use. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-10875) Remove GSon dependency
[ https://issues.apache.org/jira/browse/HBASE-10875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark resolved HBASE-10875. --- Resolution: Fixed Committed here: https://github.com/apache/hbase/commit/3fc5ce10e878ec706ee92cb01cf0c5252199e59e Remove GSon dependency -- Key: HBASE-10875 URL: https://issues.apache.org/jira/browse/HBASE-10875 Project: HBase Issue Type: Bug Components: build, Region Assignment Affects Versions: 0.89-fb Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.89-fb Remove gson from use and remove the pom dependency. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-10876) Remove Avro Connector
[ https://issues.apache.org/jira/browse/HBASE-10876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark resolved HBASE-10876. --- Resolution: Fixed Committed Here: https://github.com/apache/hbase/commit/fc07025a710a0013b0982b421a7010eda5f6ccb7 Remove Avro Connector - Key: HBASE-10876 URL: https://issues.apache.org/jira/browse/HBASE-10876 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.89-fb Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.89-fb Follow trunk and remove avro connector. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-10865) Store per table server assignment preference
[ https://issues.apache.org/jira/browse/HBASE-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark resolved HBASE-10865. --- Resolution: Fixed Committed here: * https://github.com/apache/hbase/commit/f5ce0d716c8b73eca28ae89797a255a8e594bddd * https://github.com/apache/hbase/commit/385f6c6f9bea737aa7ec6544301fe34817d8143f * https://github.com/apache/hbase/commit/bcf6023243701ebf902c3c42e882c476c18d2f3c Store per table server assignment preference Key: HBASE-10865 URL: https://issues.apache.org/jira/browse/HBASE-10865 Project: HBase Issue Type: Improvement Components: Admin, master, Region Assignment Affects Versions: 0.89-fb Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.89-fb Storing per table assignment preference in HTD will allow it to be used after table creation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-10843) Prepare HBase for java 8
[ https://issues.apache.org/jira/browse/HBASE-10843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark resolved HBASE-10843. --- Resolution: Fixed Committed Here: https://github.com/apache/hbase/commit/a5c7998feaa53f4aacdbc1218ec5f53175ebe604 Prepare HBase for java 8 Key: HBASE-10843 URL: https://issues.apache.org/jira/browse/HBASE-10843 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.89-fb Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.89-fb -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-10904) [89-FB] Add shell command support for creating tables on specific servers
[ https://issues.apache.org/jira/browse/HBASE-10904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark resolved HBASE-10904. --- Resolution: Fixed Committed Here: https://github.com/apache/hbase/commit/aab4a3bb23a578649f2de52a12ecd67867c148f6 [89-FB] Add shell command support for creating tables on specific servers - Key: HBASE-10904 URL: https://issues.apache.org/jira/browse/HBASE-10904 Project: HBase Issue Type: Bug Components: shell Affects Versions: 0.89-fb Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.89-fb -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10950) Add a configuration point for MaxVersion of Column Family
[ https://issues.apache.org/jira/browse/HBASE-10950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Demai Ni updated HBASE-10950: - Assignee: (was: Demai Ni) Add a configuration point for MaxVersion of Column Family -- Key: HBASE-10950 URL: https://issues.apache.org/jira/browse/HBASE-10950 Project: HBase Issue Type: Improvement Components: Admin Affects Versions: 0.98.0, 0.96.0 Reporter: Demai Ni Fix For: 0.99.0, 0.98.2, 0.96.3 Starting on 0.96.0. HColumnDescriptor.DEFAULT_VERSIONS change to 1 from 3. So a columnfamily will be default have 1 version of data. Currently a user can specifiy the maxVersion during create table time or alter the columnfam later. This feature will add a configuration point in hbase-sit.xml so that an admin can set the default globally. a small discussion in [HBASE-10941|https://issues.apache.org/jira/browse/HBASE-10941] lead to this jira -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10950) Add a configuration point for MaxVersion of Column Family
[ https://issues.apache.org/jira/browse/HBASE-10950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965617#comment-13965617 ] Demai Ni commented on HBASE-10950: -- [~enhs8920], thanks for working on this jira. [~ndimiduk], Enoch from my team will provide the patch for this jira. is there a way to assign the jira to him? thanks Add a configuration point for MaxVersion of Column Family -- Key: HBASE-10950 URL: https://issues.apache.org/jira/browse/HBASE-10950 Project: HBase Issue Type: Improvement Components: Admin Affects Versions: 0.98.0, 0.96.0 Reporter: Demai Ni Fix For: 0.99.0, 0.98.2, 0.96.3 Starting on 0.96.0. HColumnDescriptor.DEFAULT_VERSIONS change to 1 from 3. So a columnfamily will be default have 1 version of data. Currently a user can specifiy the maxVersion during create table time or alter the columnfam later. This feature will add a configuration point in hbase-sit.xml so that an admin can set the default globally. a small discussion in [HBASE-10941|https://issues.apache.org/jira/browse/HBASE-10941] lead to this jira -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (HBASE-10884) [REST] Do not disable block caching when scanning
[ https://issues.apache.org/jira/browse/HBASE-10884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965513#comment-13965513 ] Andrew Purtell edited comment on HBASE-10884 at 4/10/14 6:01 PM: - Test failure is not related, will commit soon. Ping [~stack] and [~lhofhansl], this is a trivial patch and perf improvement, but limited to REST use cases. However it is a behavioral change without recourse without HBASE-10952, and that introduces a new query parameter and field to the Scanner model in a backwards compatible way. was (Author: apurtell): Test failure is not related, will commit soon. Ping [~stack] and [~lhofhansl], I'm going to assume no objection to trivial patch and perf improvement, but limited to REST use cases. [REST] Do not disable block caching when scanning - Key: HBASE-10884 URL: https://issues.apache.org/jira/browse/HBASE-10884 Project: HBase Issue Type: Improvement Affects Versions: 0.98.1, 0.96.1.1, 0.94.18 Reporter: Andrew Purtell Assignee: Andrew Purtell Attachments: HBASE-10884.patch The REST gateway pessimistically disables block caching when issuing Scans to the cluster, using Scan#setCacheBlocks(false) in ScannerResultGenerator. It does not do this when issuing Gets on behalf of HTTP clients in RowResultGenerator. This is an old idea now, the reasons for doing so lost sometime back in the era when HBase walked the earth with dinosaurs ( 0.20). We probably should not be penalizing REST scans in this way. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10955) HBCK leaves the region in masters in-memory RegionStates if region hdfs dir is lost
[
https://issues.apache.org/jira/browse/HBASE-10955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-10955:
---
Status: Patch Available (was: Open)
HBCK leaves the region in masters in-memory RegionStates if region hdfs dir
is lost
---
Key: HBASE-10955
URL: https://issues.apache.org/jira/browse/HBASE-10955
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.99.0, 0.98.2
Attachments: hbase-10955.v1.patch
One of our tests removes the hdfs directory for the region, and invokes HBCK
to fix the issue. This test fails flakily because the region is removed from
meta and unassigned, but the region is not offlined from the masters
in-memory. This affects further LB runs and disable table, etc.
In case of {{inMeta !inHdfs isDeployed}}, we should not just close the
region from RS, but call master.unassign().
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10929) Change ScanQueryMatcher to use Cells instead of KeyValue.
[ https://issues.apache.org/jira/browse/HBASE-10929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10929: --- Attachment: HBASE-10929_2.patch Updated patch. Change ScanQueryMatcher to use Cells instead of KeyValue. - Key: HBASE-10929 URL: https://issues.apache.org/jira/browse/HBASE-10929 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10929.patch, HBASE-10929_1.patch, HBASE-10929_2.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10929) Change ScanQueryMatcher to use Cells instead of KeyValue.
[ https://issues.apache.org/jira/browse/HBASE-10929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10929: --- Status: Patch Available (was: Open) Change ScanQueryMatcher to use Cells instead of KeyValue. - Key: HBASE-10929 URL: https://issues.apache.org/jira/browse/HBASE-10929 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10929.patch, HBASE-10929_1.patch, HBASE-10929_2.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-10958) [dataloss] Bulk loading with seqids can prevent some log entries from being replayed
Jean-Daniel Cryans created HBASE-10958: -- Summary: [dataloss] Bulk loading with seqids can prevent some log entries from being replayed Key: HBASE-10958 URL: https://issues.apache.org/jira/browse/HBASE-10958 Project: HBase Issue Type: Bug Affects Versions: 0.94.18, 0.98.1, 0.96.2 Reporter: Jean-Daniel Cryans Priority: Blocker Fix For: 0.99.0, 0.94.19, 0.98.2, 0.96.3 We found an issue with bulk loads causing data loss when assigning sequence ids (HBASE-6630) that is triggered when replaying recovered edits. We're nicknaming this issue *Blindspot*. The problem is that the sequence id given to a bulk loaded file is higher than those of the edits in the region's memstore. When replaying recovered edits, the rule to skip some of them is that they have to be _lower than the highest sequence id_. In other words, the edits that have a sequence id lower than the highest one in the store files *should* have also been flushed. This is not the case with bulk loaded files since we now have an HFile with a sequence id higher than unflushed edits. The log recovery code takes this into account by simply skipping the bulk loaded files, but this bulk loaded status is *lost* on compaction. The edits in the logs that have a sequence id lower than the bulk loaded file that got compacted are put in a blind spot and are skipped during replay. Here's the easiest way to recreate this issue: - Create an empty table - Put one row in it (let's say it gets seqid 1) - Bulk load one file (it gets seqid 2). I used ImporTsv and set hbase.mapreduce.bulkload.assign.sequenceNumbers. - Bulk load a second file the same way (it gets seqid 3). - Major compact the table (the new file has seqid 3 and isn't considered bulk loaded). - Kill the region server that holds the table's region. - Scan the table once the region is made available again. The first row, at seqid 1, will be missing since the HFile with seqid 3 makes us believe that everything that came before it was flushed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10958) [dataloss] Bulk loading with seqids can prevent some log entries from being replayed
[ https://issues.apache.org/jira/browse/HBASE-10958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965642#comment-13965642 ] Jean-Daniel Cryans commented on HBASE-10958: One workaround we found is to completely disable compactions, then when you need to run them you have to force flush the regions that have bulk loaded file first and ensure that bulk loads aren't coming in at the same time. Workloads that are strictly doing incremental bulk loads aren't affected, you need a mix of bulk loaded files and normal Puts. A hacky solution could be to force flush when bulk loading with seqids and grab the next sequence id that comes after the memstore flush to go to the bulk loaded file. This means that bulk loading needs to initiate a flush, get the sequence id under the region write lock, then do the bulk load. We don't need to wait for the flush to happen... unless the possibility for the bulk loaded file to be compacted before the flush is done is high enough. [dataloss] Bulk loading with seqids can prevent some log entries from being replayed Key: HBASE-10958 URL: https://issues.apache.org/jira/browse/HBASE-10958 Project: HBase Issue Type: Bug Affects Versions: 0.96.2, 0.98.1, 0.94.18 Reporter: Jean-Daniel Cryans Priority: Blocker Fix For: 0.99.0, 0.94.19, 0.98.2, 0.96.3 We found an issue with bulk loads causing data loss when assigning sequence ids (HBASE-6630) that is triggered when replaying recovered edits. We're nicknaming this issue *Blindspot*. The problem is that the sequence id given to a bulk loaded file is higher than those of the edits in the region's memstore. When replaying recovered edits, the rule to skip some of them is that they have to be _lower than the highest sequence id_. In other words, the edits that have a sequence id lower than the highest one in the store files *should* have also been flushed. This is not the case with bulk loaded files since we now have an HFile with a sequence id higher than unflushed edits. The log recovery code takes this into account by simply skipping the bulk loaded files, but this bulk loaded status is *lost* on compaction. The edits in the logs that have a sequence id lower than the bulk loaded file that got compacted are put in a blind spot and are skipped during replay. Here's the easiest way to recreate this issue: - Create an empty table - Put one row in it (let's say it gets seqid 1) - Bulk load one file (it gets seqid 2). I used ImporTsv and set hbase.mapreduce.bulkload.assign.sequenceNumbers. - Bulk load a second file the same way (it gets seqid 3). - Major compact the table (the new file has seqid 3 and isn't considered bulk loaded). - Kill the region server that holds the table's region. - Scan the table once the region is made available again. The first row, at seqid 1, will be missing since the HFile with seqid 3 makes us believe that everything that came before it was flushed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10943) Backport HBASE-7329 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-10943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965662#comment-13965662 ] Lars Hofhansl commented on HBASE-10943: --- So this is a pretty radical change for 0.94, I think. [~liushaohui] do you absolutely need this in 0.94? Are we 100% that does not introduce any compatibility issues for rolling upgrades (i.e. some servers running old cold, some the new)? My gut feeling here is not to do this in 0.94. Backport HBASE-7329 to 0.94 --- Key: HBASE-10943 URL: https://issues.apache.org/jira/browse/HBASE-10943 Project: HBase Issue Type: Improvement Affects Versions: 0.94.18 Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Attachments: HBASE-10943-0.94-v1.diff See HBASE-7329 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10947) Allow subclassing of HTable and HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-10947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965666#comment-13965666 ] Lars Hofhansl commented on HBASE-10947: --- I think we should change the M/R code. The logic so far has been: * All table related API should be in HTableInterface. * Any implementation API that leaks details (like regions, etc) should only be in HTable. Maybe it is time to rethink that. If alternate HTable implementations are needed, we should extend HTableInterface to what we need. Allow subclassing of HTable and HBaseAdmin -- Key: HBASE-10947 URL: https://issues.apache.org/jira/browse/HBASE-10947 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan To extend functionality of HTable and HBaseAdmin we may need to subclass them. This JIRA allows to add a default constructor and probably remove the final variables in them so that we could subclass them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10952) [REST] Let the user turn off block caching if desired
[ https://issues.apache.org/jira/browse/HBASE-10952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10952: --- Attachment: HBASE-10952.patch [REST] Let the user turn off block caching if desired - Key: HBASE-10952 URL: https://issues.apache.org/jira/browse/HBASE-10952 Project: HBase Issue Type: Improvement Affects Versions: 0.98.1, 0.99.0, 0.94.19, 0.96.3 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: HBASE-10952.patch After HBASE-10884 the REST gateway will use scanner defaults with respect to block caching. Add support for a query parameter for hinting blocks for the query should not be cached. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10952) [REST] Let the user turn off block caching if desired
[ https://issues.apache.org/jira/browse/HBASE-10952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10952: --- Status: Patch Available (was: Open) [REST] Let the user turn off block caching if desired - Key: HBASE-10952 URL: https://issues.apache.org/jira/browse/HBASE-10952 Project: HBase Issue Type: Improvement Affects Versions: 0.98.1, 0.99.0, 0.94.19, 0.96.3 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: HBASE-10952.patch After HBASE-10884 the REST gateway will use scanner defaults with respect to block caching. Add support for a query parameter for hinting blocks for the query should not be cached. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10884) [REST] Do not disable block caching when scanning
[ https://issues.apache.org/jira/browse/HBASE-10884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965674#comment-13965674 ] Lars Hofhansl commented on HBASE-10884: --- Maybe in 0.94 we should invert the default (leave it at false and allow enabling this)...? I'm fine either way. In 0.94 we should limit the element of the surprise. But which of the following two is more suprising? # scanning via REST and the Java API use different defaults # in 0.94.18 scanning via REST did not cache, but in 0.94.19 it does by default. I guess I'm fine either way. Any opinions? [REST] Do not disable block caching when scanning - Key: HBASE-10884 URL: https://issues.apache.org/jira/browse/HBASE-10884 Project: HBase Issue Type: Improvement Affects Versions: 0.98.1, 0.96.1.1, 0.94.18 Reporter: Andrew Purtell Assignee: Andrew Purtell Attachments: HBASE-10884.patch The REST gateway pessimistically disables block caching when issuing Scans to the cluster, using Scan#setCacheBlocks(false) in ScannerResultGenerator. It does not do this when issuing Gets on behalf of HTTP clients in RowResultGenerator. This is an old idea now, the reasons for doing so lost sometime back in the era when HBase walked the earth with dinosaurs ( 0.20). We probably should not be penalizing REST scans in this way. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10952) [REST] Let the user turn off block caching if desired
[ https://issues.apache.org/jira/browse/HBASE-10952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965672#comment-13965672 ] Andrew Purtell commented on HBASE-10952: Attached patch introduces a new query parameter and field to the Scanner model in a backwards compatible way. [REST] Let the user turn off block caching if desired - Key: HBASE-10952 URL: https://issues.apache.org/jira/browse/HBASE-10952 Project: HBase Issue Type: Improvement Affects Versions: 0.98.1, 0.99.0, 0.94.19, 0.96.3 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: HBASE-10952.patch After HBASE-10884 the REST gateway will use scanner defaults with respect to block caching. Add support for a query parameter for hinting blocks for the query should not be cached. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10958) [dataloss] Bulk loading with seqids can prevent some log entries from being replayed
[ https://issues.apache.org/jira/browse/HBASE-10958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965678#comment-13965678 ] Lars Hofhansl commented on HBASE-10958: --- Seems fine to force a flush before bulk loading. [dataloss] Bulk loading with seqids can prevent some log entries from being replayed Key: HBASE-10958 URL: https://issues.apache.org/jira/browse/HBASE-10958 Project: HBase Issue Type: Bug Affects Versions: 0.96.2, 0.98.1, 0.94.18 Reporter: Jean-Daniel Cryans Priority: Blocker Fix For: 0.99.0, 0.94.19, 0.98.2, 0.96.3 We found an issue with bulk loads causing data loss when assigning sequence ids (HBASE-6630) that is triggered when replaying recovered edits. We're nicknaming this issue *Blindspot*. The problem is that the sequence id given to a bulk loaded file is higher than those of the edits in the region's memstore. When replaying recovered edits, the rule to skip some of them is that they have to be _lower than the highest sequence id_. In other words, the edits that have a sequence id lower than the highest one in the store files *should* have also been flushed. This is not the case with bulk loaded files since we now have an HFile with a sequence id higher than unflushed edits. The log recovery code takes this into account by simply skipping the bulk loaded files, but this bulk loaded status is *lost* on compaction. The edits in the logs that have a sequence id lower than the bulk loaded file that got compacted are put in a blind spot and are skipped during replay. Here's the easiest way to recreate this issue: - Create an empty table - Put one row in it (let's say it gets seqid 1) - Bulk load one file (it gets seqid 2). I used ImporTsv and set hbase.mapreduce.bulkload.assign.sequenceNumbers. - Bulk load a second file the same way (it gets seqid 3). - Major compact the table (the new file has seqid 3 and isn't considered bulk loaded). - Kill the region server that holds the table's region. - Scan the table once the region is made available again. The first row, at seqid 1, will be missing since the HFile with seqid 3 makes us believe that everything that came before it was flushed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10829) Flush is skipped after log replay if the last recovered edits file is skipped
[ https://issues.apache.org/jira/browse/HBASE-10829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965680#comment-13965680 ] Cosmin Lehene commented on HBASE-10829: --- I can't find this issue in the 0.98.1 release notes. Perhaps fix version should be 0.98.2? Flush is skipped after log replay if the last recovered edits file is skipped - Key: HBASE-10829 URL: https://issues.apache.org/jira/browse/HBASE-10829 Project: HBase Issue Type: Bug Reporter: Enis Soztutar Assignee: Enis Soztutar Priority: Critical Fix For: 0.98.1, 0.99.0, 0.96.3 Attachments: hbase-10829_v1.patch, hbase-10829_v2.patch, hbase-10829_v3.patch We caught this in an extended test run where IntegrationTestBigLinkedList failed with some missing keys. The problem is that HRegion.replayRecoveredEdits() would return -1 if all the edits in the log file is skipped, which is true for example if the log file only contains a single compaction record (HBASE-2231) or somehow the edits cannot be applied (column family deleted, etc). The callee, HRegion.replayRecoveredEditsIfAny() only looks for the last returned seqId to decide whether a flush is necessary or not before opening the region, and discarding replayed recovered edits files. Therefore, if the last recovered edits file is skipped but some edits from earlier recovered edits files are applied, the mandatory flush before opening the region is skipped. If the region server dies after this point before a flush, the edits are lost. This is important to fix, though the sequence of events are super rare for a production cluster. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10952) [REST] Let the user turn off block caching if desired
[ https://issues.apache.org/jira/browse/HBASE-10952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10952: --- Description: After HBASE-10884 the REST gateway will use scanner defaults with respect to block caching. Add support for a query parameter for hinting blocks for the query should not be cached. Enable block caching by default. (was: After HBASE-10884 the REST gateway will use scanner defaults with respect to block caching. Add support for a query parameter for hinting blocks for the query should not be cached.) [REST] Let the user turn off block caching if desired - Key: HBASE-10952 URL: https://issues.apache.org/jira/browse/HBASE-10952 Project: HBase Issue Type: Improvement Affects Versions: 0.98.1, 0.99.0, 0.94.19, 0.96.3 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: HBASE-10952.patch After HBASE-10884 the REST gateway will use scanner defaults with respect to block caching. Add support for a query parameter for hinting blocks for the query should not be cached. Enable block caching by default. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10902) Make Secure Bulk Load work across remote secure clusters
[
https://issues.apache.org/jira/browse/HBASE-10902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965691#comment-13965691
]
Jerry He commented on HBASE-10902:
--
Thank you [~yuzhih...@gmail.com] for the review.
[~apurtell], [~mbertozzi]: do you have comment?
Make Secure Bulk Load work across remote secure clusters
Key: HBASE-10902
URL: https://issues.apache.org/jira/browse/HBASE-10902
Project: HBase
Issue Type: Improvement
Affects Versions: 0.96.1
Reporter: Jerry He
Assignee: Jerry He
Fix For: 0.99.0
Attachments: HBASE-10902-v0-0.96.patch, HBASE-10902-v1-trunk.patch,
HBASE-10902-v2-trunk.patch
Two secure clusters, both with kerberos enabled.
Run bulk load on one cluster to load files from another cluster.
biadmin@hdtest249:~ hbase
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles
hdfs://bdvm197.svl.ibm.com:9000/user/biadmin/mybackups/TestTable/0709e79bb131af13ed088bf1afd5649c
TestTable_rr
Bulk load failed. In the region server log:
{code}
2014-04-02 20:04:56,361 ERROR
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint: Failed to
complete bulk load
java.lang.IllegalArgumentException: Wrong FS:
hdfs://bdvm197.svl.ibm.com:9000/user/biadmin/mybackups/TestTable/0709e79bb131af13ed088bf1afd5649c/info/6b44ca48aebf48d98cb3491f512c41a7,
expected: hdfs://hdtest249.svl.ibm.com:9000
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:651)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:181)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:92)
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1248)
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1244)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.setPermission(DistributedFileSystem.java:1244)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:233)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:223)
at
java.security.AccessController.doPrivileged(AccessController.java:300)
at javax.security.auth.Subject.doAs(Subject.java:494)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1482)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.secureBulkLoadHFiles(SecureBulkLoadEndpoint.java:223)
at
org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4631)
at
org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5088)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3219)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26933)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1854)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10902) Make Secure Bulk Load work across remote secure clusters
[
https://issues.apache.org/jira/browse/HBASE-10902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965698#comment-13965698
]
Matteo Bertozzi commented on HBASE-10902:
-
+1 looks good to me
Make Secure Bulk Load work across remote secure clusters
Key: HBASE-10902
URL: https://issues.apache.org/jira/browse/HBASE-10902
Project: HBase
Issue Type: Improvement
Affects Versions: 0.96.1
Reporter: Jerry He
Assignee: Jerry He
Fix For: 0.99.0
Attachments: HBASE-10902-v0-0.96.patch, HBASE-10902-v1-trunk.patch,
HBASE-10902-v2-trunk.patch
Two secure clusters, both with kerberos enabled.
Run bulk load on one cluster to load files from another cluster.
biadmin@hdtest249:~ hbase
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles
hdfs://bdvm197.svl.ibm.com:9000/user/biadmin/mybackups/TestTable/0709e79bb131af13ed088bf1afd5649c
TestTable_rr
Bulk load failed. In the region server log:
{code}
2014-04-02 20:04:56,361 ERROR
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint: Failed to
complete bulk load
java.lang.IllegalArgumentException: Wrong FS:
hdfs://bdvm197.svl.ibm.com:9000/user/biadmin/mybackups/TestTable/0709e79bb131af13ed088bf1afd5649c/info/6b44ca48aebf48d98cb3491f512c41a7,
expected: hdfs://hdtest249.svl.ibm.com:9000
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:651)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:181)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:92)
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1248)
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1244)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.setPermission(DistributedFileSystem.java:1244)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:233)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:223)
at
java.security.AccessController.doPrivileged(AccessController.java:300)
at javax.security.auth.Subject.doAs(Subject.java:494)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1482)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.secureBulkLoadHFiles(SecureBulkLoadEndpoint.java:223)
at
org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4631)
at
org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5088)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3219)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26933)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1854)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10949) Meta scan could hang
[ https://issues.apache.org/jira/browse/HBASE-10949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-10949: Priority: Blocker (was: Critical) Meta scan could hang Key: HBASE-10949 URL: https://issues.apache.org/jira/browse/HBASE-10949 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Priority: Blocker Fix For: 0.99.0 Attachments: master-jstack.log When I play with ITBLL (from trunk tip), sometimes, meta scan hangs when the cluster is rolling restarted. When this happens, the master takes about 1000% of CPU. It looks like there is an infinite loop somewhere. The logs show nothing interesting except some meta scanner RPC calls timed out. Jstask shows the 10 high QoS RPC handlers are busy with meta scanning. However, if I run it again without HBASE-10018, things are fine. I suspect there is something to do with the small/reverse scan. By the way, I see this problem even with log replay off and hfile version = 2. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10955) HBCK leaves the region in masters in-memory RegionStates if region hdfs dir is lost
[
https://issues.apache.org/jira/browse/HBASE-10955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965735#comment-13965735
]
Devaraj Das commented on HBASE-10955:
-
Other than the unneeded whitespace changes, looks good to me. Also, just as a
sanity check, do run the hbck unit tests on the hbase-10070 branch.
HBCK leaves the region in masters in-memory RegionStates if region hdfs dir
is lost
---
Key: HBASE-10955
URL: https://issues.apache.org/jira/browse/HBASE-10955
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.99.0, 0.98.2
Attachments: hbase-10955.v1.patch
One of our tests removes the hdfs directory for the region, and invokes HBCK
to fix the issue. This test fails flakily because the region is removed from
meta and unassigned, but the region is not offlined from the masters
in-memory. This affects further LB runs and disable table, etc.
In case of {{inMeta !inHdfs isDeployed}}, we should not just close the
region from RS, but call master.unassign().
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HBASE-10949) Meta scan could hang
[ https://issues.apache.org/jira/browse/HBASE-10949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang reassigned HBASE-10949: --- Assignee: Jimmy Xiang Meta scan could hang Key: HBASE-10949 URL: https://issues.apache.org/jira/browse/HBASE-10949 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Blocker Fix For: 0.99.0 Attachments: master-jstack.log When I play with ITBLL (from trunk tip), sometimes, meta scan hangs when the cluster is rolling restarted. When this happens, the master takes about 1000% of CPU. It looks like there is an infinite loop somewhere. The logs show nothing interesting except some meta scanner RPC calls timed out. Jstask shows the 10 high QoS RPC handlers are busy with meta scanning. However, if I run it again without HBASE-10018, things are fine. I suspect there is something to do with the small/reverse scan. By the way, I see this problem even with log replay off and hfile version = 2. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10949) Meta scan could hang
[ https://issues.apache.org/jira/browse/HBASE-10949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965737#comment-13965737 ] Jimmy Xiang commented on HBASE-10949: - Looked into it and found StoreFileScanner#backwardSeek always returns true in some case. I am testing a patch so that if seek() return false, don't return true. Meta scan could hang Key: HBASE-10949 URL: https://issues.apache.org/jira/browse/HBASE-10949 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Blocker Fix For: 0.99.0 Attachments: master-jstack.log When I play with ITBLL (from trunk tip), sometimes, meta scan hangs when the cluster is rolling restarted. When this happens, the master takes about 1000% of CPU. It looks like there is an infinite loop somewhere. The logs show nothing interesting except some meta scanner RPC calls timed out. Jstask shows the 10 high QoS RPC handlers are busy with meta scanning. However, if I run it again without HBASE-10018, things are fine. I suspect there is something to do with the small/reverse scan. By the way, I see this problem even with log replay off and hfile version = 2. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10897) On master start, deadlock if refresh UI
[
https://issues.apache.org/jira/browse/HBASE-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965752#comment-13965752
]
Hadoop QA commented on HBASE-10897:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639610/hbase-10897_v4.patch
against trunk revision .
ATTACHMENT ID: 12639610
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9248//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9248//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9248//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9248//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9248//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9248//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9248//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9248//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9248//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9248//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9248//console
This message is automatically generated.
On master start, deadlock if refresh UI
---
Key: HBASE-10897
URL: https://issues.apache.org/jira/browse/HBASE-10897
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: stack
Assignee: Jimmy Xiang
Fix For: 0.99.0
Attachments: hbase-10897.patch, hbase-10897_v2.patch,
hbase-10897_v3.patch, hbase-10897_v4.patch
Playing w/ MTTR recovery on trunk, master starting up deadlocked:
Waiting to finish active master initialization:
{code}
ActiveMasterManager daemon prio=10 tid=0x7fafb5dc3800 nid=0x5fb5
waiting for monitor entry [0x7faf8f57d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1683)
- waiting to lock 0x00064ab4b9a8 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:53)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1029)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:989)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:830)
at
org.apache.hadoop.hbase.client.ConnectionAdapter.getRegionLocation(ConnectionAdapter.java:305)
at
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:77)
at
org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:118)
at
[jira] [Updated] (HBASE-10949) Reversed scan could hang
[ https://issues.apache.org/jira/browse/HBASE-10949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-10949: Priority: Critical (was: Blocker) Reversed scan could hang Key: HBASE-10949 URL: https://issues.apache.org/jira/browse/HBASE-10949 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Critical Fix For: 0.99.0 Attachments: master-jstack.log When I play with ITBLL (from trunk tip), sometimes, meta scan hangs when the cluster is rolling restarted. When this happens, the master takes about 1000% of CPU. It looks like there is an infinite loop somewhere. The logs show nothing interesting except some meta scanner RPC calls timed out. Jstask shows the 10 high QoS RPC handlers are busy with meta scanning. However, if I run it again without HBASE-10018, things are fine. I suspect there is something to do with the small/reverse scan. By the way, I see this problem even with log replay off and hfile version = 2. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10949) Reversed scan could hang
[ https://issues.apache.org/jira/browse/HBASE-10949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-10949: Summary: Reversed scan could hang (was: Meta scan could hang) Reversed scan could hang Key: HBASE-10949 URL: https://issues.apache.org/jira/browse/HBASE-10949 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Blocker Fix For: 0.99.0 Attachments: master-jstack.log When I play with ITBLL (from trunk tip), sometimes, meta scan hangs when the cluster is rolling restarted. When this happens, the master takes about 1000% of CPU. It looks like there is an infinite loop somewhere. The logs show nothing interesting except some meta scanner RPC calls timed out. Jstask shows the 10 high QoS RPC handlers are busy with meta scanning. However, if I run it again without HBASE-10018, things are fine. I suspect there is something to do with the small/reverse scan. By the way, I see this problem even with log replay off and hfile version = 2. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10949) Reversed scan could hang
[ https://issues.apache.org/jira/browse/HBASE-10949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-10949: Affects Version/s: 0.98.0 Reversed scan could hang Key: HBASE-10949 URL: https://issues.apache.org/jira/browse/HBASE-10949 Project: HBase Issue Type: Bug Affects Versions: 0.98.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Critical Fix For: 0.99.0 Attachments: master-jstack.log When I play with ITBLL (from trunk tip), sometimes, meta scan hangs when the cluster is rolling restarted. When this happens, the master takes about 1000% of CPU. It looks like there is an infinite loop somewhere. The logs show nothing interesting except some meta scanner RPC calls timed out. Jstask shows the 10 high QoS RPC handlers are busy with meta scanning. However, if I run it again without HBASE-10018, things are fine. I suspect there is something to do with the small/reverse scan. By the way, I see this problem even with log replay off and hfile version = 2. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10070) HBase read high-availability using eventually consistent region replicas
[ https://issues.apache.org/jira/browse/HBASE-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965779#comment-13965779 ] Tianying Chang commented on HBASE-10070: I am using Enis git branch of 10070, it has commit until Jan 30. That is probably why I don't see it? I checked the 10070-working branch, it has up to Feb 13, seems still not update enough. Do you know which branch has the multi get support? HBase read high-availability using eventually consistent region replicas Key: HBASE-10070 URL: https://issues.apache.org/jira/browse/HBASE-10070 Project: HBase Issue Type: New Feature Reporter: Enis Soztutar Assignee: Enis Soztutar Attachments: HighAvailabilityDesignforreadsApachedoc.pdf In the present HBase architecture, it is hard, probably impossible, to satisfy constraints like 99th percentile of the reads will be served under 10 ms. One of the major factors that affects this is the MTTR for regions. There are three phases in the MTTR process - detection, assignment, and recovery. Of these, the detection is usually the longest and is presently in the order of 20-30 seconds. During this time, the clients would not be able to read the region data. However, some clients will be better served if regions will be available for reads during recovery for doing eventually consistent reads. This will help with satisfying low latency guarantees for some class of applications which can work with stale reads. For improving read availability, we propose a replicated read-only region serving design, also referred as secondary regions, or region shadows. Extending current model of a region being opened for reads and writes in a single region server, the region will be also opened for reading in region servers. The region server which hosts the region for reads and writes (as in current case) will be declared as PRIMARY, while 0 or more region servers might be hosting the region as SECONDARY. There may be more than one secondary (replica count 2). Will attach a design doc shortly which contains most of the details and some thoughts about development approaches. Reviews are more than welcome. We also have a proof of concept patch, which includes the master and regions server side of changes. Client side changes will be coming soon as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10070) HBase read high-availability using eventually consistent region replicas
[ https://issues.apache.org/jira/browse/HBASE-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965781#comment-13965781 ] Ted Yu commented on HBASE-10070: Tianying, here is the branch: https://svn.apache.org/repos/asf/hbase/branches/hbase-10070 HBase read high-availability using eventually consistent region replicas Key: HBASE-10070 URL: https://issues.apache.org/jira/browse/HBASE-10070 Project: HBase Issue Type: New Feature Reporter: Enis Soztutar Assignee: Enis Soztutar Attachments: HighAvailabilityDesignforreadsApachedoc.pdf In the present HBase architecture, it is hard, probably impossible, to satisfy constraints like 99th percentile of the reads will be served under 10 ms. One of the major factors that affects this is the MTTR for regions. There are three phases in the MTTR process - detection, assignment, and recovery. Of these, the detection is usually the longest and is presently in the order of 20-30 seconds. During this time, the clients would not be able to read the region data. However, some clients will be better served if regions will be available for reads during recovery for doing eventually consistent reads. This will help with satisfying low latency guarantees for some class of applications which can work with stale reads. For improving read availability, we propose a replicated read-only region serving design, also referred as secondary regions, or region shadows. Extending current model of a region being opened for reads and writes in a single region server, the region will be also opened for reading in region servers. The region server which hosts the region for reads and writes (as in current case) will be declared as PRIMARY, while 0 or more region servers might be hosting the region as SECONDARY. There may be more than one secondary (replica count 2). Will attach a design doc shortly which contains most of the details and some thoughts about development approaches. Reviews are more than welcome. We also have a proof of concept patch, which includes the master and regions server side of changes. Client side changes will be coming soon as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-10959) Packaging test sources jar
Manukranth Kolloju created HBASE-10959: -- Summary: Packaging test sources jar Key: HBASE-10959 URL: https://issues.apache.org/jira/browse/HBASE-10959 Project: HBase Issue Type: Test Components: test Affects Versions: 0.89-fb Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Priority: Trivial Fix For: 0.89-fb Packaging test sources jar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10794) multi-get should handle missing replica location from cache
[ https://issues.apache.org/jira/browse/HBASE-10794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-10794: - Attachment: HBASE-10794.03.patch Rebased patch, merged in the addendum. No substantial changes so I will commit to branch later today multi-get should handle missing replica location from cache --- Key: HBASE-10794 URL: https://issues.apache.org/jira/browse/HBASE-10794 Project: HBase Issue Type: Sub-task Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: hbase-10070 Attachments: HBASE-10794.01.patch, HBASE-10794.02.addendum.patch, HBASE-10794.02.patch, HBASE-10794.03.patch, HBASE-10794.patch, HBASE-10794.patch Currently the way cache works is that the meta row is stored together for all replicas of a region, so if some replicas are in recovery, getting locations for a region will still go to cache only and return null locations for these. Multi-get currently ignores such replicas. It should instead try to get location again from meta if any replica is null. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10915) Decouple region closing from ZooKeeper
[
https://issues.apache.org/jira/browse/HBASE-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965803#comment-13965803
]
Ted Yu commented on HBASE-10915:
This method in HRegionServer can be package private:
{code}
+ public ConsensusProvider getConsensus() {
{code}
{code}
+ /** Config for pluggable consensus providers */
+ public static final String HBASE_CONSENSUS_PROVIDER_CLASS =
hbase.consensus.provider.class;
{code}
nit: 'pluggable consensus providers' - 'pluggable consensus provider'
lgtm
Decouple region closing from ZooKeeper
--
Key: HBASE-10915
URL: https://issues.apache.org/jira/browse/HBASE-10915
Project: HBase
Issue Type: Sub-task
Components: Zookeeper
Reporter: Mikhail Antonov
Assignee: Mikhail Antonov
Attachments: HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch,
HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch
Decouple CloseRegionHandler class and code using it (HRegionServer,
ProtobufUtil) from ZK API.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10900) FULL table backup and restore
[
https://issues.apache.org/jira/browse/HBASE-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965804#comment-13965804
]
Demai Ni commented on HBASE-10900:
--
[~tedyu], thanks. I can move the code to hbase-client module and within a new
'backup' package there.
[~tedyu] and [~mbertozzi], many thanks for your input through review board. I
am going through them right now, and will incorporate in the next version of
the patch.
Demai
FULL table backup and restore
-
Key: HBASE-10900
URL: https://issues.apache.org/jira/browse/HBASE-10900
Project: HBase
Issue Type: Task
Reporter: Demai Ni
Assignee: Demai Ni
Fix For: 1.0.0
Attachments: HBASE-10900-fullbackup-trunk-v1.patch
h2. Feature Description
This is a subtask of
[HBase-7912|https://issues.apache.org/jira/browse/HBASE-7912] to support FULL
backup/restore, and will complete the following function:
{code:title=Backup Restore example|borderStyle=solid}
/* backup from sourcecluster to targetcluster
*/
/* if no table name specified, all tables from source cluster will be
backuped */
[sourcecluster]$ hbase backup create full
hdfs://hostname.targetcluster.org:9000/userid/backupdir t1_dn,t2_dn,t3_dn
/* restore on targetcluser, this is a local restore
*/
/* backup_1396650096738 - backup image name
*/
/* t1_dn,etc are the original table names. All tables will be restored if not
specified */
/* t1_dn_restore, etc. are the restored table. if not specified, orginal
table name will be used*/
[targetcluster]$ hbase restore /userid/backupdir backup_1396650096738
t1_dn,t2_dn,t3_dn t1_dn_restore,t2_dn_restore,t3_dn_restore
/* restore from targetcluster back to source cluster, this is a remote restore
[sourcecluster]$ hbase restore
hdfs://hostname.targetcluster.org:9000/userid/backupdir backup_1396650096738
t1_dn,t2_dn,t3_dn t1_dn_restore,t2_dn_restore,t3_dn_restore
{code}
h2. Detail layout and frame work for the next jiras
The patch is a wrapper of the existing snapshot and exportSnapshot, and will
use as the base framework for the over-all solution of
[HBase-7912|https://issues.apache.org/jira/browse/HBASE-7912] as described
below:
* *bin/hbase* : end-user command line interface to invoke
BackupClient and RestoreClient
* *BackupClient.java* : 'main' entry for backup operations. This patch will
only support 'full' backup. In future jiras, will support:
** *create* incremental backup
** *cancel* an ongoing backup
** *delete* an exisitng backup image
** *describe* the detailed informaiton of backup image
** show *history* of all successful backups
** show the *status* of the latest backup request
** *convert* incremental backup WAL files into HFiles. either on-the-fly
during create or after create
** *merge* backup image
** *stop* backup a table of existing backup image
** *show* tables of a backup image
* *BackupCommands.java* : a place to keep all the command usages and options
* *BackupManager.java* : handle backup requests on server-side, create
BACKUP ZOOKEEPER nodes to keep track backup. The timestamps kept in zookeeper
will be used for future incremental backup (not included in this jira).
Create BackupContext and DispatchRequest.
* *BackupHandler.java* : in this patch, it is a wrapper of snapshot and
exportsnapshot. In future jiras,
** *timestamps* info will be recorded in ZK
** carry on *incremental* backup.
** update backup *progress*
** set flags of *status*
** build up *backupManifest* file(in this jira only limited info for
fullback. later on, timestamps and dependency of multipl backup images are
also recorded here)
** clean up after *failed* backup
** clean up after *cancelled* backup
** allow on-the-fly *convert* during incremental backup
* *BackupContext.java* : encapsulate backup information like backup ID, table
names, directory info, phase, TimeStamps of backup progress, size of data,
ancestor info, etc.
* *BackupCopier.java* : the copying operation. Later on, to support
progress report and mapper estimation; and extends DisCp for progress
updating to ZK during backup.
* *BackupExcpetion.java*: to handle exception from backup/restore
* *BackupManifest.java* : encapsulate all the backup image information. The
manifest info will be bundled as manifest file together with data. So that
each backup image will contain all the info needed for restore.
* *BackupStatus.java* : encapsulate backup status at table level during
backup progress
* *BackupUtil.java* : utility methods during backup process
* *RestoreClient.java* : 'main' entry for restore operations. This
