[jira] [Commented] (HBASE-11144) Filter to support scan multiple row key ranges
[
https://issues.apache.org/jira/browse/HBASE-11144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14267462#comment-14267462
]
Hadoop QA commented on HBASE-11144:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690513/HBASE_11144_V9.patch
against master branch at commit 45fc2bdd863ff238491f4941b3da04a38731d7e1.
ATTACHMENT ID: 12690513
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
2085 checkstyle errors (more than the master's current 2084 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12338//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12338//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12338//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12338//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12338//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12338//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12338//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12338//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12338//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12338//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12338//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12338//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12338//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12338//console
This message is automatically generated.
Filter to support scan multiple row key ranges
--
Key: HBASE-11144
URL: https://issues.apache.org/jira/browse/HBASE-11144
Project: HBase
Issue Type: Improvement
Components: Filters
Reporter: Jiajia Li
Assignee: Jiajia Li
Attachments: HBASE_11144_4.patch, HBASE_11144_V5.patch,
HBASE_11144_V6.patch, HBASE_11144_V7.patch, HBASE_11144_V9.patch,
MultiRowRangeFilter.patch, MultiRowRangeFilter2.patch,
MultiRowRangeFilter3.patch, hbase_11144_V8.patch
HBase is quite efficient when scanning only one small row key range. If user
needs to specify multiple row key ranges in one scan, the typical solutions
are: 1. through FilterList which is a list of row key Filters, 2. using the
SQL layer over HBase to join with two table, such as hive, phoenix etc.
However, both solutions are inefficient. Both of them can’t utilize the range
info to perform fast forwarding during scan which is quite time consuming. If
the number of ranges are quite big (e.g. millions), join is a proper solution
though it is slow. However, there are cases that user wants to specify a
small number of ranges to scan (e.g. 1000 ranges). Both solutions can’t
provide satisfactory performance in such case.
We provide this filter
[jira] [Commented] (HBASE-8410) Basic quota support for namespaces
[
https://issues.apache.org/jira/browse/HBASE-8410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14267473#comment-14267473
]
Hadoop QA commented on HBASE-8410:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690516/HBASE-8410-master.7.patch
against master branch at commit 45fc2bdd863ff238491f4941b3da04a38731d7e1.
ATTACHMENT ID: 12690516
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
2087 checkstyle errors (more than the master's current 2084 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12339//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12339//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12339//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12339//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12339//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12339//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12339//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12339//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12339//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12339//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12339//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12339//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12339//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12339//console
This message is automatically generated.
Basic quota support for namespaces
--
Key: HBASE-8410
URL: https://issues.apache.org/jira/browse/HBASE-8410
Project: HBase
Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Vandana Ayyalasomayajula
Attachments: HBASE-8410-master.1.patch, HBASE-8410-master.4.patch,
HBASE-8410-master.5.patch, HBASE-8410-master.6.patch,
HBASE-8410-master.7.patch, HBASE-8410-master.patch, HBASE-8410.docx,
HBASE-8410_trunk_14.patch
This task involves creating an observer which provides basic quota support to
namespaces in terms of (1) number of tables and (2) number of regions. The
quota support can be enabled by setting:
property
namehbase.coprocessor.region.classes/name
valueorg.apache.hadoop.hbase.namespace.NamespaceController/value
/property
property
namehbase.coprocessor.master.classes/name
valueorg.apache.hadoop.hbase.namespace.NamespaceController/value
/property
in the hbase-site.xml.
To add quotas to namespace, while creating namespace properties need to be
added.
Examples:
1. namespace_create 'ns1', {'hbase.namespace.quota.maxregion'='10'}
2. 1. namespace_create 'ns2', {'hbase.namespace.quota.maxtables'='2'},
{'hbase.namespace.quota.maxregion'='5'}
The quotas can be modified/added to namespace
[jira] [Commented] (HBASE-5205) Delete handles deleteFamily incorrectly
[ https://issues.apache.org/jira/browse/HBASE-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14267481#comment-14267481 ] Cosmin Lehene commented on HBASE-5205: -- [~lhofhansl] would it make sense to set a fixVersion (1.0.1, 1.1.0?)? Delete handles deleteFamily incorrectly --- Key: HBASE-5205 URL: https://issues.apache.org/jira/browse/HBASE-5205 Project: HBase Issue Type: Bug Components: Client Reporter: Lars Hofhansl Priority: Minor Attachments: 5205.txt Delete.deleteFamily clears all other markers for the same family. That is not correct as some of these other markers might be for a later time. That logic should be removed. If (really) needed this can be slightly optimized by keeping track of the max TS so far for each family. If both the TS-so-far and the TS of a new deleteFamily request is LATEST_TIMESTAMP and the TS-so-far is deleteFamily marker, or if both the TS-so-far and the new TS equal LATEST_TIMESTAMP, then the previous delete markerz for that family could be removed. I think that might be overkill, as most deletes issued from clients are for LATEST_TIMESTAMP (which the server translates to the current time). I'll have a (one-line) patch soon. Unless folks insist on the optimization I mentioned above. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12728) buffered writes substantially less useful after removal of HTablePool
[
https://issues.apache.org/jira/browse/HBASE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14267482#comment-14267482
]
Carter commented on HBASE-12728:
Lots of great comments/questions. Solomon can dig into some of the grittier
implementation trade-offs, as this is largely his design. I can answer some
initial questions to keep the ball rolling:
{quote}
{{ExceptionListener}} should also get the original {{Put}}
{quote}
We were planning to return {{RetriesExhaustedWithDetailsException}} whenever
possible, which contains the Put. That’s more consistent with calling Put
synchronously.
{quote}
all these API's should accept Mutation as the argument
{quote}
Figuring that’s where we should head eventually, but doesn’t seem we need to be
there immediately, right? Probably makes sense to follow Stack’s idea and make
a more general name, e.g. s/AsyncPutter/AsyncMutator/
{quote}
AsyncPutter looks a lot like HTable's internal AsyncProcess
{quote}
Yeah, worth making sure we’re not reinventing the wheel, but also don’t want to
inherit unnecessary complexity. I'll leave this one to Solomon, and maybe it
can be debated further in a patch review.
{quote}
does BufferedTable.flush() block the calling thread?
{quote}
Absolutely. It’s a way to guarantee that previous writes are persisted, when
such confidence is needed.
{quote}
Isn't PutBuffer just an implementation detail…
{quote}
PutBuffer is an implementation detail, and it’s probably confusing that I
surfaced it in the last straw man. Just disregard it and focus on the
AsyncPutter, which is the long-lived owner of the buffered mutation mechanism
and all that entails.
{quote}
In the event of a BufferedTable that does not share it's PutBuffer with any
other instances, can it just defer back to the current implementation in HTable?
{quote}
If it makes sense to have a different AsyncPutter for single-threaded
situations than multi-threaded, then I’d suggest another implementation rather
than trying to make it guess for itself. But why do you think we should have
multiple implementations? Synchronization locks should be insignificant
compared to even very fast wire latencies.
{quote}
On AsyncPutter, will we ever want to buffer Increments or Appends or Deletes?
{quote}
I think deletes make sense. Appends make me nervous because we’re already
breaking sequence guarantees. I think if we do a solution which supports puts,
and can be extended to any other mutation, then we can decide at another time.
Using {[AsyncMutator}} as a name would hint at such extensibility.
{quote}
do we even need to expose BufferedTable?
{quote}
Yes, need it for flush(), which I described more above.
Re: stack’s concern on ExceptionListener, returning
RetriesExhaustedWithDetailsException should provide enough info, no?
{quote}
BufferedTable shouldn't have a close
{quote}
I agree, but if it implements Table, then it has to have a nop close.
We’re generally still in alignment with Lars’ comments. Table implementations
become lightweight. BufferedConnection might be a bit of a misnomer, since
it’s more a factory than a “buffered connection”. But it seems easier for a
developer to understand rather than minting a new factory concept.
{quote}
Just add a Connection.getBufferedTable
{quote}
But Connection is an interface. BufferedConnection is intended as an
implementation of that interface, something like:
{code:java}
public class BufferedConnection implements Connection {
public BufferedConnection(Connection conn, ExceptionListener l);
}
{code}
The idea is that given any implementation of conn, BufferedConnection will wrap
the encapsulated tables that are created with buffering logic. If we put it in
Connection, then we make buffering first-class functionality again, rather than
extended functionality.
{quote}
Because in the proposal, the BufferedTable is not the owner of the put buffer
{quote}
BufferedTable#flush would be a synchronous pass-through to AsyncPutter#flush.
It doesn't need to own it to do so.
Some more comments...
Was not planning to have BufferedTable to be heavy or thread safe. Doing that
makes it less interchangeable with HTable — which is not thread safe, and would
be lighter after this refactor.
Basically the lifecycle revolves around the {{AsyncMutator}}, which is the only
heavy thing here. BufferedConnection constructs and owns it, and injects the
mutator into new BufferedTable objects, which will be cheap to construct.
Calling BufferedConnection#close would close AsyncMutator, which in turn would
flush its buffers and shutdown its worker threads. (And throw
IllegalStateException if any other operations come through, except for another
close, which should be idempotent.)
There was also a question about the need for two Connection objects. There
only needs to be one Connection object, which BufferedConnection
[jira] [Created] (HBASE-12816) GC logs are lost upon Region Server restart if GCLogFileRotation is enabled
Abhishek Singh Chouhan created HBASE-12816: -- Summary: GC logs are lost upon Region Server restart if GCLogFileRotation is enabled Key: HBASE-12816 URL: https://issues.apache.org/jira/browse/HBASE-12816 Project: HBase Issue Type: Bug Components: scripts Reporter: Abhishek Singh Chouhan Priority: Minor When -XX:+UseGCLogFileRotation is used gc log files end with .gc.0 instead of .gc. hbase_rotate_log () in hbase-daemon.sh does not handle this correctly and hence when a RS is restarted old gc logs are lost(overwritten). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-8410) Basic quota support for namespaces
[
https://issues.apache.org/jira/browse/HBASE-8410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vandana Ayyalasomayajula updated HBASE-8410:
Attachment: HBASE-8410-master.7.patch
Patch to fix unit tests.
Basic quota support for namespaces
--
Key: HBASE-8410
URL: https://issues.apache.org/jira/browse/HBASE-8410
Project: HBase
Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Vandana Ayyalasomayajula
Attachments: HBASE-8410-master.1.patch, HBASE-8410-master.4.patch,
HBASE-8410-master.5.patch, HBASE-8410-master.6.patch,
HBASE-8410-master.7.patch, HBASE-8410-master.patch, HBASE-8410.docx,
HBASE-8410_trunk_14.patch
This task involves creating an observer which provides basic quota support to
namespaces in terms of (1) number of tables and (2) number of regions. The
quota support can be enabled by setting:
property
namehbase.coprocessor.region.classes/name
valueorg.apache.hadoop.hbase.namespace.NamespaceController/value
/property
property
namehbase.coprocessor.master.classes/name
valueorg.apache.hadoop.hbase.namespace.NamespaceController/value
/property
in the hbase-site.xml.
To add quotas to namespace, while creating namespace properties need to be
added.
Examples:
1. namespace_create 'ns1', {'hbase.namespace.quota.maxregion'='10'}
2. 1. namespace_create 'ns2', {'hbase.namespace.quota.maxtables'='2'},
{'hbase.namespace.quota.maxregion'='5'}
The quotas can be modified/added to namespace at any point of time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12817) Data missing while scanning using PREFIX_TREE DATA-BLOCK-ENCODING
zhangduo created HBASE-12817:
Summary: Data missing while scanning using PREFIX_TREE
DATA-BLOCK-ENCODING
Key: HBASE-12817
URL: https://issues.apache.org/jira/browse/HBASE-12817
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.98.9
Reporter: zhangduo
Assignee: zhangduo
write a testcase like this
{code}
@Test
public void test() throws IOException {
for (int i = 0; i 100; i++) {
region.put(new Put(Bytes.toBytes(obj + (2900 + i))).add(fam, qual1,
Bytes.toBytes(i)));
}
region.put(new Put(Bytes.toBytes(obj299)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj29)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj2)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj3)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.flushcache(true);
Scan scan = new Scan(Bytes.toBytes(obj29995));
RegionScanner scanner = region.getScanner(scan);
ListCell cells = new ArrayListCell();
assertFalse(scanner.next(cells));
assertArrayEquals(Bytes.toBytes(obj3), Result.create(cells).getRow());
}
{code}
use obj29995 to scan should return obj3, but obj2990 is returned.
Seems a bug introduced by the fix of HBASE-11728.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-5402) PerformanceEvaluation creates the wrong number of rows in randomWrite
[
https://issues.apache.org/jira/browse/HBASE-5402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cosmin Lehene updated HBASE-5402:
-
Labels: beginner (was: delete)
PerformanceEvaluation creates the wrong number of rows in randomWrite
-
Key: HBASE-5402
URL: https://issues.apache.org/jira/browse/HBASE-5402
Project: HBase
Issue Type: Bug
Components: test
Reporter: Oliver Meyn
Labels: beginner
The command line 'hbase org.apache.hadoop.hbase.PerformanceEvaluation
randomWrite 10' should result in a table with 10 * (1024 * 1024) rows (so
10485760). Instead what happens is that the randomWrite job reports writing
that many rows (exactly) but running rowcounter against the table reveals
only e.g 6549899 rows. A second attempt to build the table produced slightly
different results (e.g. 6627689). I see a similar discrepancy when using 50
instead of 10 clients (~35% smaller than expected).
Further experimentation reveals that the problem is key collision - by
removing the % totalRows in getRandomRow I saw a reduction in collisions
(table was ~8M rows instead of 6.6M). Replacing the random row key with
UUIDs instead of Integers solved the problem and produced exactly 10485760
rows. But that makes the key size 16 bytes instead of the current 10, so I'm
not sure that's an acceptable solution.
Here's the UUID code I used:
public static byte[] format(final UUID uuid) {
long msb = uuid.getMostSignificantBits();
long lsb = uuid.getLeastSignificantBits();
byte[] buffer = new byte[16];
for (int i = 0; i 8; i++) {
buffer[i] = (byte) (msb 8 * (7 - i));
}
for (int i = 8; i 16; i++) {
buffer[i] = (byte) (lsb 8 * (7 - i));
}
return buffer;
}
which is invoked within getRandomRow with
return format(UUID.randomUUID());
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-5402) PerformanceEvaluation creates the wrong number of rows in randomWrite
[
https://issues.apache.org/jira/browse/HBASE-5402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cosmin Lehene updated HBASE-5402:
-
Labels: delete (was: )
PerformanceEvaluation creates the wrong number of rows in randomWrite
-
Key: HBASE-5402
URL: https://issues.apache.org/jira/browse/HBASE-5402
Project: HBase
Issue Type: Bug
Components: test
Reporter: Oliver Meyn
Labels: beginner
The command line 'hbase org.apache.hadoop.hbase.PerformanceEvaluation
randomWrite 10' should result in a table with 10 * (1024 * 1024) rows (so
10485760). Instead what happens is that the randomWrite job reports writing
that many rows (exactly) but running rowcounter against the table reveals
only e.g 6549899 rows. A second attempt to build the table produced slightly
different results (e.g. 6627689). I see a similar discrepancy when using 50
instead of 10 clients (~35% smaller than expected).
Further experimentation reveals that the problem is key collision - by
removing the % totalRows in getRandomRow I saw a reduction in collisions
(table was ~8M rows instead of 6.6M). Replacing the random row key with
UUIDs instead of Integers solved the problem and produced exactly 10485760
rows. But that makes the key size 16 bytes instead of the current 10, so I'm
not sure that's an acceptable solution.
Here's the UUID code I used:
public static byte[] format(final UUID uuid) {
long msb = uuid.getMostSignificantBits();
long lsb = uuid.getLeastSignificantBits();
byte[] buffer = new byte[16];
for (int i = 0; i 8; i++) {
buffer[i] = (byte) (msb 8 * (7 - i));
}
for (int i = 8; i 16; i++) {
buffer[i] = (byte) (lsb 8 * (7 - i));
}
return buffer;
}
which is invoked within getRandomRow with
return format(UUID.randomUUID());
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12811) [AccessController] NPE while scan a table with user not having READ permission on the namespace
[
https://issues.apache.org/jira/browse/HBASE-12811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Singhi updated HBASE-12811:
--
Description:
Steps to reproduce
1) Grant a user permission(other than READ) on a namespace
2) Scan a table in that namespace from that user
we get the following exception.
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hbase.security.access.TablePermission.implies(TablePermission.java:215)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:340)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:332)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorizeGroup(TableAuthManager.java:473)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:490)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:500)
at
org.apache.hadoop.hbase.security.access.AccessController.permissionGranted(AccessController.java:415)
at
org.apache.hadoop.hbase.security.access.AccessController.permissionGranted(AccessController.java:484)
at
org.apache.hadoop.hbase.security.access.AccessController.internalPreRead(AccessController.java:1504)
at
org.apache.hadoop.hbase.security.access.AccessController.preScannerOpen(AccessController.java:2027)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1987)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3102)
{noformat}
*Note:* Line numbers may not match.
was:
A user is associated with two groups.
{noformat}
/hbase/bin groups ashish_test
ashish_test : defaultgroup ashish_test_1420524824527
{noformat}
One of its group is granted permission on a table as shown by user_permission
command.
{noformat}
hbase(main):005:0 user_permission 't1'
User
Table,Family,Qualifier:Permission
@ashish_test_1420524824527 t1,,: [Permission:
actions=EXEC,WRITE,CREATE]
@ashish_test_1420524824527 t1,d,: [Permission:
actions=EXEC,WRITE,CREATE]
hbase t1,,: [Permission:
actions=READ,WRITE,EXEC,CREATE,ADMIN]
3 row(s) in 0.3710 seconds
{noformat}
Now when this user try the scan the table, we get the following exception.
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hbase.security.access.TablePermission.implies(TablePermission.java:215)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:340)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:332)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorizeGroup(TableAuthManager.java:473)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:490)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:500)
at
org.apache.hadoop.hbase.security.access.AccessController.permissionGranted(AccessController.java:415)
at
org.apache.hadoop.hbase.security.access.AccessController.permissionGranted(AccessController.java:484)
at
org.apache.hadoop.hbase.security.access.AccessController.internalPreRead(AccessController.java:1504)
at
org.apache.hadoop.hbase.security.access.AccessController.preScannerOpen(AccessController.java:2027)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1987)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3102)
{noformat}
*Note:* Line numbers may not match.
Exception is coming because the other group of same user which has not been
granted permission on the table will have the TablePermission's table(name) as
null.
Summary: [AccessController] NPE while scan a table with user not having
READ permission on the namespace (was: [AccessController] NPE while scan a
table with user associated with multiple groups.)
[AccessController] NPE while scan a table with user not having READ
permission on the namespace
---
Key: HBASE-12811
URL: https://issues.apache.org/jira/browse/HBASE-12811
Project: HBase
Issue Type: Bug
Components: security
Affects Versions: 0.98.9
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Steps to reproduce
1) Grant a user permission(other than READ) on a namespace
2) Scan a table in that namespace from
[jira] [Commented] (HBASE-5528) Change retrying splitting log forever if throws IOException to numbered times, and abort master when retries exhausted
[ https://issues.apache.org/jira/browse/HBASE-5528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14267632#comment-14267632 ] Cosmin Lehene commented on HBASE-5528: -- [~zjushch], [~apurtell] is this still valid? Change retrying splitting log forever if throws IOException to numbered times, and abort master when retries exhausted --- Key: HBASE-5528 URL: https://issues.apache.org/jira/browse/HBASE-5528 Project: HBase Issue Type: Bug Reporter: chunhui shen Assignee: chunhui shen Labels: delete Attachments: hbase-5528.patch, hbase-5528v2.patch, hbase-5528v3.patch In current log-splitting retry logic, it will retry forever if throws IOException, I think we'd better change it to numbered times, and abort master when retries exhausted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5532) get NPE during MajorCompactionChecker
[ https://issues.apache.org/jira/browse/HBASE-5532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14267612#comment-14267612 ] Cosmin Lehene commented on HBASE-5532: -- [~terry_zhang], [~apurtell] this may not apply anymore. The code has changed a bit, and it seems we check for null readers. I'm not sure about the locking semantics. The logic has moved to StoreFile.java and there's related logic in StripeStoreFileManager isMajorCompaction() is in HStore.java get NPE during MajorCompactionChecker -- Key: HBASE-5532 URL: https://issues.apache.org/jira/browse/HBASE-5532 Project: HBase Issue Type: Bug Components: regionserver Reporter: terry zhang Labels: delete Attachments: HBASE-5532-v2.patch, HBASE-5532.patch We found error log (NullPointerException) below on our online cluster: 2012-03-05 00:17:09,592 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker: Caught exception java.lang.NullPointerException at org.apache.hadoop.hbase.regionserver.Store.isMajorCompaction(Store.java:878) at org.apache.hadoop.hbase.regionserver.Store.isMajorCompaction(Store.java:857) at org.apache.hadoop.hbase.regionserver.HRegion.isMajorCompaction(HRegion.java:3017) at org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker.chore(HRegionServer.java:1172) at org.apache.hadoop.hbase.Chore.run(Chore.java:66) After Check the code we found although it already check whether store files has null reader at the begin of the function(isMajorCompaction), but it still has some possibility the reader is closed before it return(eg mini compaction). So we need to check store file reader before we use it to avoid this NPE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5761) [Thrift2] TDelete.deleteType defaults to TDeleteType.DELETE_COLUMNS, but the docs suggest otherwise
[
https://issues.apache.org/jira/browse/HBASE-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cosmin Lehene updated HBASE-5761:
-
Priority: Trivial (was: Major)
[Thrift2] TDelete.deleteType defaults to TDeleteType.DELETE_COLUMNS, but the
docs suggest otherwise
---
Key: HBASE-5761
URL: https://issues.apache.org/jira/browse/HBASE-5761
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Wouter Bolsterlee
Priority: Trivial
Labels: beginner
It seems to me there is an inconsistency (or error) in the Thrift2
{{TDelete}} struct and its documentation. The docs for the {{TDelete}} struct
state:
{quote}
If no timestamp is specified the most recent version will be deleted. To
delete all previous versions, specify the DELETE_COLUMNS TDeleteType.
{quote}
...which implies that the default is {{TDeleteType.DELETE_COLUMN}}
(singular), not {{TDeleteType.DELETE_COLUMNS}} (plural).
However, the {{deleteType}} field in the {{TDelete}} struct defaults to the
value {{1}}, which is {{TDeleteType.DELETE_COLUMNS}} (plural) in
{{/src/main/resources/org/apache/hadoop/hbase/thrift2/hbase.thrift}}. The
field is currently (r1239241) defined as follows:
{{4: optional TDeleteType deleteType = 1,}}
I'd suggest that the default for this optional field is changed to
{{TDeleteType.DELETE_COLUMN}} (singular). The line above from the {{TDelete}}
struct would then become:
{{4: optional TDeleteType deleteType = 0,}}
Since this change just involves changing a {{1}} into a {{0}}, I'll leave the
trivial patch to someone who can also commit it in one go. Thanks in advance.
:)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-5761) [Thrift2] TDelete.deleteType defaults to TDeleteType.DELETE_COLUMNS, but the docs suggest otherwise
[
https://issues.apache.org/jira/browse/HBASE-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cosmin Lehene updated HBASE-5761:
-
Component/s: documentation
[Thrift2] TDelete.deleteType defaults to TDeleteType.DELETE_COLUMNS, but the
docs suggest otherwise
---
Key: HBASE-5761
URL: https://issues.apache.org/jira/browse/HBASE-5761
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Wouter Bolsterlee
Labels: beginner
It seems to me there is an inconsistency (or error) in the Thrift2
{{TDelete}} struct and its documentation. The docs for the {{TDelete}} struct
state:
{quote}
If no timestamp is specified the most recent version will be deleted. To
delete all previous versions, specify the DELETE_COLUMNS TDeleteType.
{quote}
...which implies that the default is {{TDeleteType.DELETE_COLUMN}}
(singular), not {{TDeleteType.DELETE_COLUMNS}} (plural).
However, the {{deleteType}} field in the {{TDelete}} struct defaults to the
value {{1}}, which is {{TDeleteType.DELETE_COLUMNS}} (plural) in
{{/src/main/resources/org/apache/hadoop/hbase/thrift2/hbase.thrift}}. The
field is currently (r1239241) defined as follows:
{{4: optional TDeleteType deleteType = 1,}}
I'd suggest that the default for this optional field is changed to
{{TDeleteType.DELETE_COLUMN}} (singular). The line above from the {{TDelete}}
struct would then become:
{{4: optional TDeleteType deleteType = 0,}}
Since this change just involves changing a {{1}} into a {{0}}, I'll leave the
trivial patch to someone who can also commit it in one go. Thanks in advance.
:)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-5761) [Thrift2] TDelete.deleteType defaults to TDeleteType.DELETE_COLUMNS, but the docs suggest otherwise
[
https://issues.apache.org/jira/browse/HBASE-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cosmin Lehene updated HBASE-5761:
-
Labels: beginner (was: delete)
[Thrift2] TDelete.deleteType defaults to TDeleteType.DELETE_COLUMNS, but the
docs suggest otherwise
---
Key: HBASE-5761
URL: https://issues.apache.org/jira/browse/HBASE-5761
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Wouter Bolsterlee
Priority: Trivial
Labels: beginner
It seems to me there is an inconsistency (or error) in the Thrift2
{{TDelete}} struct and its documentation. The docs for the {{TDelete}} struct
state:
{quote}
If no timestamp is specified the most recent version will be deleted. To
delete all previous versions, specify the DELETE_COLUMNS TDeleteType.
{quote}
...which implies that the default is {{TDeleteType.DELETE_COLUMN}}
(singular), not {{TDeleteType.DELETE_COLUMNS}} (plural).
However, the {{deleteType}} field in the {{TDelete}} struct defaults to the
value {{1}}, which is {{TDeleteType.DELETE_COLUMNS}} (plural) in
{{/src/main/resources/org/apache/hadoop/hbase/thrift2/hbase.thrift}}. The
field is currently (r1239241) defined as follows:
{{4: optional TDeleteType deleteType = 1,}}
I'd suggest that the default for this optional field is changed to
{{TDeleteType.DELETE_COLUMN}} (singular). The line above from the {{TDelete}}
struct would then become:
{{4: optional TDeleteType deleteType = 0,}}
Since this change just involves changing a {{1}} into a {{0}}, I'll leave the
trivial patch to someone who can also commit it in one go. Thanks in advance.
:)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-5949) We needn't add splitParent region to assignment map when rebuildUserRegions
[ https://issues.apache.org/jira/browse/HBASE-5949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene resolved HBASE-5949. -- Resolution: Duplicate We needn't add splitParent region to assignment map when rebuildUserRegions --- Key: HBASE-5949 URL: https://issues.apache.org/jira/browse/HBASE-5949 Project: HBase Issue Type: Bug Components: master Reporter: chunhui shen Assignee: chunhui shen Attachments: HBASE-5949.patch In current AssignmentManager#rebuildUserRegions, we will add splitParent region to assignment map. When doing balance, the splitParent region will be always in RIT because no server is carrying it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12728) buffered writes substantially less useful after removal of HTablePool
[
https://issues.apache.org/jira/browse/HBASE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14267678#comment-14267678
]
Solomon Duskis commented on HBASE-12728:
I like having a separate interface for bulk writing that's accessible from a
new method on Connection.
At this point, I'm rethinking the AsyncPutter / BufferedTable approach. Bulk
writing asynchronously is geared to a couple of very specific cases. Table
currently has 37 methods on it, most of which will not be implemented any
differently in the asynchronous use cases. Given those two complexities, I
would think that a Separation of Concerns and Keep It Simple might be best.
A BulkWriter (or BulkMutator) interface with a limited number of methods on it
might work better than extending Table. Perhaps a simplified API like this
might work:
{code}
public interface BulkWriter {
void put(Put p);
void delete(Delete);
flush();
close();
}
public interface Connection {
...
BulkWriter getBulkWriter(int maxBufferSize [, some other configuration
parameters]);
}
{code}
Thoughts?
buffered writes substantially less useful after removal of HTablePool
-
Key: HBASE-12728
URL: https://issues.apache.org/jira/browse/HBASE-12728
Project: HBase
Issue Type: Bug
Components: hbase
Affects Versions: 0.98.0
Reporter: Aaron Beppu
Assignee: Solomon Duskis
Priority: Blocker
Fix For: 1.0.0, 2.0.0, 1.1.0
In previous versions of HBase, when use of HTablePool was encouraged, HTable
instances were long-lived in that pool, and for that reason, if autoFlush was
set to false, the table instance could accumulate a full buffer of writes
before a flush was triggered. Writes from the client to the cluster could
then be substantially larger and less frequent than without buffering.
However, when HTablePool was deprecated, the primary justification seems to
have been that creating HTable instances is cheap, so long as the connection
and executor service being passed to it are pre-provided. A use pattern was
encouraged where users should create a new HTable instance for every
operation, using an existing connection and executor service, and then close
the table. In this pattern, buffered writes are substantially less useful;
writes are as small and as frequent as they would have been with
autoflush=true, except the synchronous write is moved from the operation
itself to the table close call which immediately follows.
More concretely :
```
// Given these two helpers ...
private HTableInterface getAutoFlushTable(String tableName) throws
IOException {
// (autoflush is true by default)
return storedConnection.getTable(tableName, executorService);
}
private HTableInterface getBufferedTable(String tableName) throws IOException
{
HTableInterface table = getAutoFlushTable(tableName);
table.setAutoFlush(false);
return table;
}
// it's my contention that these two methods would behave almost identically,
// except the first will hit a synchronous flush during the put call,
and the second will
// flush during the (hidden) close call on table.
private void writeAutoFlushed(Put somePut) throws IOException {
try (HTableInterface table = getAutoFlushTable(tableName)) {
table.put(somePut); // will do synchronous flush
}
}
private void writeBuffered(Put somePut) throws IOException {
try (HTableInterface table = getBufferedTable(tableName)) {
table.put(somePut);
} // auto-close will trigger synchronous flush
}
```
For buffered writes to actually provide a performance benefit to users, one
of two things must happen:
- The writeBuffer itself shouldn't live, flush and die with the lifecycle of
it's HTableInstance. If the writeBuffer were managed elsewhere and had a long
lifespan, this could cease to be an issue. However, if the same writeBuffer
is appended to by multiple tables, then some additional concurrency control
will be needed around it.
- Alternatively, there should be some pattern for having long-lived HTable
instances. However, since HTable is not thread-safe, we'd need multiple
instances, and a mechanism for leasing them out safely -- which sure sounds a
lot like the old HTablePool to me.
See discussion on mailing list here :
http://mail-archives.apache.org/mod_mbox/hbase-user/201412.mbox/%3CCAPdJLkEzmUQZ_kvD%3D8mrxi4V%3DhCmUp3g9MUZsddD%2Bmon%2BAvNtg%40mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12817) Data missing while scanning using PREFIX_TREE DATA-BLOCK-ENCODING
[
https://issues.apache.org/jira/browse/HBASE-12817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14267551#comment-14267551
]
ramkrishna.s.vasudevan commented on HBASE-12817:
Thanks for filing this [~Apache9]. Are you working on a patch for this?
Data missing while scanning using PREFIX_TREE DATA-BLOCK-ENCODING
-
Key: HBASE-12817
URL: https://issues.apache.org/jira/browse/HBASE-12817
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.98.9
Reporter: zhangduo
Assignee: zhangduo
write a testcase like this
{code}
@Test
public void test() throws IOException {
for (int i = 0; i 100; i++) {
region.put(new Put(Bytes.toBytes(obj + (2900 + i))).add(fam, qual1,
Bytes.toBytes(i)));
}
region.put(new Put(Bytes.toBytes(obj299)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj29)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj2)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj3)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.flushcache(true);
Scan scan = new Scan(Bytes.toBytes(obj29995));
RegionScanner scanner = region.getScanner(scan);
ListCell cells = new ArrayListCell();
assertFalse(scanner.next(cells));
assertArrayEquals(Bytes.toBytes(obj3), Result.create(cells).getRow());
}
{code}
use obj29995 to scan should return obj3, but obj2990 is returned.
Seems a bug introduced by the fix of HBASE-11728.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-5402) PerformanceEvaluation creates the wrong number of rows in randomWrite
[
https://issues.apache.org/jira/browse/HBASE-5402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cosmin Lehene updated HBASE-5402:
-
Issue Type: Improvement (was: Bug)
PerformanceEvaluation creates the wrong number of rows in randomWrite
-
Key: HBASE-5402
URL: https://issues.apache.org/jira/browse/HBASE-5402
Project: HBase
Issue Type: Improvement
Components: test
Reporter: Oliver Meyn
Labels: beginner
The command line 'hbase org.apache.hadoop.hbase.PerformanceEvaluation
randomWrite 10' should result in a table with 10 * (1024 * 1024) rows (so
10485760). Instead what happens is that the randomWrite job reports writing
that many rows (exactly) but running rowcounter against the table reveals
only e.g 6549899 rows. A second attempt to build the table produced slightly
different results (e.g. 6627689). I see a similar discrepancy when using 50
instead of 10 clients (~35% smaller than expected).
Further experimentation reveals that the problem is key collision - by
removing the % totalRows in getRandomRow I saw a reduction in collisions
(table was ~8M rows instead of 6.6M). Replacing the random row key with
UUIDs instead of Integers solved the problem and produced exactly 10485760
rows. But that makes the key size 16 bytes instead of the current 10, so I'm
not sure that's an acceptable solution.
Here's the UUID code I used:
public static byte[] format(final UUID uuid) {
long msb = uuid.getMostSignificantBits();
long lsb = uuid.getLeastSignificantBits();
byte[] buffer = new byte[16];
for (int i = 0; i 8; i++) {
buffer[i] = (byte) (msb 8 * (7 - i));
}
for (int i = 8; i 16; i++) {
buffer[i] = (byte) (lsb 8 * (7 - i));
}
return buffer;
}
which is invoked within getRandomRow with
return format(UUID.randomUUID());
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-5532) get NPE during MajorCompactionChecker
[ https://issues.apache.org/jira/browse/HBASE-5532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene updated HBASE-5532: - Labels: delete (was: ) get NPE during MajorCompactionChecker -- Key: HBASE-5532 URL: https://issues.apache.org/jira/browse/HBASE-5532 Project: HBase Issue Type: Bug Components: regionserver Reporter: terry zhang Labels: delete Attachments: HBASE-5532-v2.patch, HBASE-5532.patch We found error log (NullPointerException) below on our online cluster: 2012-03-05 00:17:09,592 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker: Caught exception java.lang.NullPointerException at org.apache.hadoop.hbase.regionserver.Store.isMajorCompaction(Store.java:878) at org.apache.hadoop.hbase.regionserver.Store.isMajorCompaction(Store.java:857) at org.apache.hadoop.hbase.regionserver.HRegion.isMajorCompaction(HRegion.java:3017) at org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker.chore(HRegionServer.java:1172) at org.apache.hadoop.hbase.Chore.run(Chore.java:66) After Check the code we found although it already check whether store files has null reader at the begin of the function(isMajorCompaction), but it still has some possibility the reader is closed before it return(eg mini compaction). So we need to check store file reader before we use it to avoid this NPE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5528) Change retrying splitting log forever if throws IOException to numbered times, and abort master when retries exhausted
[ https://issues.apache.org/jira/browse/HBASE-5528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene updated HBASE-5528: - Labels: delete (was: ) Change retrying splitting log forever if throws IOException to numbered times, and abort master when retries exhausted --- Key: HBASE-5528 URL: https://issues.apache.org/jira/browse/HBASE-5528 Project: HBase Issue Type: Bug Reporter: chunhui shen Assignee: chunhui shen Labels: delete Attachments: hbase-5528.patch, hbase-5528v2.patch, hbase-5528v3.patch In current log-splitting retry logic, it will retry forever if throws IOException, I think we'd better change it to numbered times, and abort master when retries exhausted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5703) Bound the number of threads in HRegionThriftServer
[ https://issues.apache.org/jira/browse/HBASE-5703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene updated HBASE-5703: - Labels: delete (was: ) Bound the number of threads in HRegionThriftServer -- Key: HBASE-5703 URL: https://issues.apache.org/jira/browse/HBASE-5703 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Labels: delete We need to bound the number of threads spawned in HRegionThriftServer, similarly to what was done in HBASE-4863 to the standalone Thrift gateway. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5703) Bound the number of threads in HRegionThriftServer
[ https://issues.apache.org/jira/browse/HBASE-5703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14267644#comment-14267644 ] Cosmin Lehene commented on HBASE-5703: -- [~mikhail], [~apurtell] Is this still valid? There's no HRegionThriftServer anymore.. Can we close? Bound the number of threads in HRegionThriftServer -- Key: HBASE-5703 URL: https://issues.apache.org/jira/browse/HBASE-5703 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Labels: delete We need to bound the number of threads spawned in HRegionThriftServer, similarly to what was done in HBASE-4863 to the standalone Thrift gateway. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5868) jmx bean layout makes no sense
[ https://issues.apache.org/jira/browse/HBASE-5868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14267657#comment-14267657 ] Cosmin Lehene commented on HBASE-5868: -- [~saint@gmail.com] is this still valid? jmx bean layout makes no sense -- Key: HBASE-5868 URL: https://issues.apache.org/jira/browse/HBASE-5868 Project: HBase Issue Type: Bug Reporter: stack Priority: Critical Labels: delete Top level is 'hadoop' Under 'hadoop', there is 'HBase' MBean. BESIDE this MBean is one named Master and another named RegionServer. It makes no sense. Top level should be org.apache.hbase. Inside there should be an MBean per running server. It should be the server's ServerName, not 'Master' or 'RegionServer'. Under RegionServer there is a RegionServer bean [sic], then beside it a RegionServerStatistics and a RegionServerDynamicStatistics. I'd think that as they are, they are unusable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5868) jmx bean layout makes no sense
[ https://issues.apache.org/jira/browse/HBASE-5868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene updated HBASE-5868: - Labels: delete (was: ) jmx bean layout makes no sense -- Key: HBASE-5868 URL: https://issues.apache.org/jira/browse/HBASE-5868 Project: HBase Issue Type: Bug Reporter: stack Priority: Critical Labels: delete Top level is 'hadoop' Under 'hadoop', there is 'HBase' MBean. BESIDE this MBean is one named Master and another named RegionServer. It makes no sense. Top level should be org.apache.hbase. Inside there should be an MBean per running server. It should be the server's ServerName, not 'Master' or 'RegionServer'. Under RegionServer there is a RegionServer bean [sic], then beside it a RegionServerStatistics and a RegionServerDynamicStatistics. I'd think that as they are, they are unusable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-5828) TestLogRolling fails in 0.90 branch killing the test suite up on jenkins
[ https://issues.apache.org/jira/browse/HBASE-5828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene resolved HBASE-5828. -- Resolution: Cannot Reproduce TestLogRolling fails in 0.90 branch killing the test suite up on jenkins Key: HBASE-5828 URL: https://issues.apache.org/jira/browse/HBASE-5828 Project: HBase Issue Type: Bug Reporter: stack Assignee: xufeng Attachments: HBASE-5828_90_v1.patch, HBASE-5828_90_v1_100times_test_result.txt, TestLogRolling.java, org.apache.hadoop.hbase.regionserver.wal.TestLogRolling-output.rar See recent 0.90 builds up on jenkins: https://builds.apache.org/view/G-L/view/HBase/job/hbase-0.90/471/console -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12817) Data missing while scanning using PREFIX_TREE DATA-BLOCK-ENCODING
[
https://issues.apache.org/jira/browse/HBASE-12817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangduo updated HBASE-12817:
-
Status: Patch Available (was: Open)
Data missing while scanning using PREFIX_TREE DATA-BLOCK-ENCODING
-
Key: HBASE-12817
URL: https://issues.apache.org/jira/browse/HBASE-12817
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.98.9
Reporter: zhangduo
Assignee: zhangduo
Attachments: HBASE-12817.patch
write a testcase like this
{code}
@Test
public void test() throws IOException {
for (int i = 0; i 100; i++) {
region.put(new Put(Bytes.toBytes(obj + (2900 + i))).add(fam, qual1,
Bytes.toBytes(i)));
}
region.put(new Put(Bytes.toBytes(obj299)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj29)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj2)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj3)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.flushcache(true);
Scan scan = new Scan(Bytes.toBytes(obj29995));
RegionScanner scanner = region.getScanner(scan);
ListCell cells = new ArrayListCell();
assertFalse(scanner.next(cells));
assertArrayEquals(Bytes.toBytes(obj3), Result.create(cells).getRow());
}
{code}
use obj29995 to scan should return obj3, but obj2990 is returned.
Seems a bug introduced by the fix of HBASE-11728.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12817) Data missing while scanning using PREFIX_TREE DATA-BLOCK-ENCODING
[
https://issues.apache.org/jira/browse/HBASE-12817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangduo updated HBASE-12817:
-
Attachment: HBASE-12817.patch
Data missing while scanning using PREFIX_TREE DATA-BLOCK-ENCODING
-
Key: HBASE-12817
URL: https://issues.apache.org/jira/browse/HBASE-12817
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.98.9
Reporter: zhangduo
Assignee: zhangduo
Attachments: HBASE-12817.patch
write a testcase like this
{code}
@Test
public void test() throws IOException {
for (int i = 0; i 100; i++) {
region.put(new Put(Bytes.toBytes(obj + (2900 + i))).add(fam, qual1,
Bytes.toBytes(i)));
}
region.put(new Put(Bytes.toBytes(obj299)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj29)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj2)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj3)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.flushcache(true);
Scan scan = new Scan(Bytes.toBytes(obj29995));
RegionScanner scanner = region.getScanner(scan);
ListCell cells = new ArrayListCell();
assertFalse(scanner.next(cells));
assertArrayEquals(Bytes.toBytes(obj3), Result.create(cells).getRow());
}
{code}
use obj29995 to scan should return obj3, but obj2990 is returned.
Seems a bug introduced by the fix of HBASE-11728.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12811) [AccessController] NPE while scan a table with user not having READ permission on the namespace
[
https://issues.apache.org/jira/browse/HBASE-12811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Singhi updated HBASE-12811:
--
Description:
Steps to reproduce
1) Grant a group permission(other than READ) on a namespace
2) Scan a table in that namespace from a user belonging to that group
we get the following exception.
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hbase.security.access.TablePermission.implies(TablePermission.java:215)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:340)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:332)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorizeGroup(TableAuthManager.java:473)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:490)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:500)
at
org.apache.hadoop.hbase.security.access.AccessController.permissionGranted(AccessController.java:415)
at
org.apache.hadoop.hbase.security.access.AccessController.permissionGranted(AccessController.java:484)
at
org.apache.hadoop.hbase.security.access.AccessController.internalPreRead(AccessController.java:1504)
at
org.apache.hadoop.hbase.security.access.AccessController.preScannerOpen(AccessController.java:2027)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1987)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3102)
{noformat}
*Note:* Line numbers may not match.
was:
Steps to reproduce
1) Grant a user permission(other than READ) on a namespace
2) Scan a table in that namespace from that user
we get the following exception.
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hbase.security.access.TablePermission.implies(TablePermission.java:215)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:340)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:332)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorizeGroup(TableAuthManager.java:473)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:490)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:500)
at
org.apache.hadoop.hbase.security.access.AccessController.permissionGranted(AccessController.java:415)
at
org.apache.hadoop.hbase.security.access.AccessController.permissionGranted(AccessController.java:484)
at
org.apache.hadoop.hbase.security.access.AccessController.internalPreRead(AccessController.java:1504)
at
org.apache.hadoop.hbase.security.access.AccessController.preScannerOpen(AccessController.java:2027)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1987)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3102)
{noformat}
*Note:* Line numbers may not match.
[AccessController] NPE while scan a table with user not having READ
permission on the namespace
---
Key: HBASE-12811
URL: https://issues.apache.org/jira/browse/HBASE-12811
Project: HBase
Issue Type: Bug
Components: security
Affects Versions: 0.98.9
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12811.patch
Steps to reproduce
1) Grant a group permission(other than READ) on a namespace
2) Scan a table in that namespace from a user belonging to that group
we get the following exception.
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hbase.security.access.TablePermission.implies(TablePermission.java:215)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:340)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:332)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorizeGroup(TableAuthManager.java:473)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:490)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:500)
at
org.apache.hadoop.hbase.security.access.AccessController.permissionGranted(AccessController.java:415)
at
[jira] [Updated] (HBASE-12811) [AccessController] NPE while scan a table with user not having READ permission on the namespace
[
https://issues.apache.org/jira/browse/HBASE-12811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Singhi updated HBASE-12811:
--
Status: Patch Available (was: Open)
[AccessController] NPE while scan a table with user not having READ
permission on the namespace
---
Key: HBASE-12811
URL: https://issues.apache.org/jira/browse/HBASE-12811
Project: HBase
Issue Type: Bug
Components: security
Affects Versions: 0.98.9
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12811.patch
Steps to reproduce
1) Grant a group permission(other than READ) on a namespace
2) Scan a table in that namespace from a user belonging to that group
we get the following exception.
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hbase.security.access.TablePermission.implies(TablePermission.java:215)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:340)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:332)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorizeGroup(TableAuthManager.java:473)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:490)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:500)
at
org.apache.hadoop.hbase.security.access.AccessController.permissionGranted(AccessController.java:415)
at
org.apache.hadoop.hbase.security.access.AccessController.permissionGranted(AccessController.java:484)
at
org.apache.hadoop.hbase.security.access.AccessController.internalPreRead(AccessController.java:1504)
at
org.apache.hadoop.hbase.security.access.AccessController.preScannerOpen(AccessController.java:2027)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1987)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3102)
{noformat}
*Note:* Line numbers may not match.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-4924) More issues running hbase on hadoop 0.23
[
https://issues.apache.org/jira/browse/HBASE-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cosmin Lehene updated HBASE-4924:
-
Labels: delete (was: )
More issues running hbase on hadoop 0.23
Key: HBASE-4924
URL: https://issues.apache.org/jira/browse/HBASE-4924
Project: HBase
Issue Type: Bug
Reporter: stack
Labels: delete
From Roman:
{code}
3. This is a little bit more sinister and it is not clear what
the resolution path here is. When compiled against .23
the MiniMRCluster dependency latches onto artifact
hadoop-mapreduce/test. There are 2 problems with that:
1. this works only accidentally (but as long as it does
may be it is fine) since we don't have an explicit
dependency on hadoop-mapreduce test artifact (nor
should we, I think!).
2. MiniMRCluster from there is soon to be deprecated
(MAPREDUCE-3169) ans HBaseTestingUtility should
really transition onto using MiniMRClientClusterFactory
to get a MiniMRCluster.
{code}
I think there already an issue for #2 above (courtesy of Andrew IIRC). We
should fix #1 too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-4924) More issues running hbase on hadoop 0.23
[
https://issues.apache.org/jira/browse/HBASE-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14267642#comment-14267642
]
Cosmin Lehene commented on HBASE-4924:
--
[~saint@gmail.com] , [~apurtell] can we close this one?
More issues running hbase on hadoop 0.23
Key: HBASE-4924
URL: https://issues.apache.org/jira/browse/HBASE-4924
Project: HBase
Issue Type: Bug
Reporter: stack
Labels: delete
From Roman:
{code}
3. This is a little bit more sinister and it is not clear what
the resolution path here is. When compiled against .23
the MiniMRCluster dependency latches onto artifact
hadoop-mapreduce/test. There are 2 problems with that:
1. this works only accidentally (but as long as it does
may be it is fine) since we don't have an explicit
dependency on hadoop-mapreduce test artifact (nor
should we, I think!).
2. MiniMRCluster from there is soon to be deprecated
(MAPREDUCE-3169) ans HBaseTestingUtility should
really transition onto using MiniMRClientClusterFactory
to get a MiniMRCluster.
{code}
I think there already an issue for #2 above (courtesy of Andrew IIRC). We
should fix #1 too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-5761) [Thrift2] TDelete.deleteType defaults to TDeleteType.DELETE_COLUMNS, but the docs suggest otherwise
[
https://issues.apache.org/jira/browse/HBASE-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cosmin Lehene updated HBASE-5761:
-
Fix Version/s: 1.0.0
[Thrift2] TDelete.deleteType defaults to TDeleteType.DELETE_COLUMNS, but the
docs suggest otherwise
---
Key: HBASE-5761
URL: https://issues.apache.org/jira/browse/HBASE-5761
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Wouter Bolsterlee
Priority: Trivial
Labels: beginner
Fix For: 1.0.0
It seems to me there is an inconsistency (or error) in the Thrift2
{{TDelete}} struct and its documentation. The docs for the {{TDelete}} struct
state:
{quote}
If no timestamp is specified the most recent version will be deleted. To
delete all previous versions, specify the DELETE_COLUMNS TDeleteType.
{quote}
...which implies that the default is {{TDeleteType.DELETE_COLUMN}}
(singular), not {{TDeleteType.DELETE_COLUMNS}} (plural).
However, the {{deleteType}} field in the {{TDelete}} struct defaults to the
value {{1}}, which is {{TDeleteType.DELETE_COLUMNS}} (plural) in
{{/src/main/resources/org/apache/hadoop/hbase/thrift2/hbase.thrift}}. The
field is currently (r1239241) defined as follows:
{{4: optional TDeleteType deleteType = 1,}}
I'd suggest that the default for this optional field is changed to
{{TDeleteType.DELETE_COLUMN}} (singular). The line above from the {{TDelete}}
struct would then become:
{{4: optional TDeleteType deleteType = 0,}}
Since this change just involves changing a {{1}} into a {{0}}, I'll leave the
trivial patch to someone who can also commit it in one go. Thanks in advance.
:)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5761) [Thrift2] TDelete.deleteType defaults to TDeleteType.DELETE_COLUMNS, but the docs suggest otherwise
[
https://issues.apache.org/jira/browse/HBASE-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14267653#comment-14267653
]
Cosmin Lehene commented on HBASE-5761:
--
[~uws], [~saint@gmail.com], [~apurtell] wouldn't it make more sense to
change the documentation for TDelete?
[Thrift2] TDelete.deleteType defaults to TDeleteType.DELETE_COLUMNS, but the
docs suggest otherwise
---
Key: HBASE-5761
URL: https://issues.apache.org/jira/browse/HBASE-5761
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Wouter Bolsterlee
Priority: Trivial
Labels: beginner
Fix For: 1.0.0
It seems to me there is an inconsistency (or error) in the Thrift2
{{TDelete}} struct and its documentation. The docs for the {{TDelete}} struct
state:
{quote}
If no timestamp is specified the most recent version will be deleted. To
delete all previous versions, specify the DELETE_COLUMNS TDeleteType.
{quote}
...which implies that the default is {{TDeleteType.DELETE_COLUMN}}
(singular), not {{TDeleteType.DELETE_COLUMNS}} (plural).
However, the {{deleteType}} field in the {{TDelete}} struct defaults to the
value {{1}}, which is {{TDeleteType.DELETE_COLUMNS}} (plural) in
{{/src/main/resources/org/apache/hadoop/hbase/thrift2/hbase.thrift}}. The
field is currently (r1239241) defined as follows:
{{4: optional TDeleteType deleteType = 1,}}
I'd suggest that the default for this optional field is changed to
{{TDeleteType.DELETE_COLUMN}} (singular). The line above from the {{TDelete}}
struct would then become:
{{4: optional TDeleteType deleteType = 0,}}
Since this change just involves changing a {{1}} into a {{0}}, I'll leave the
trivial patch to someone who can also commit it in one go. Thanks in advance.
:)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-5756) we can change defalult File Appender to RFA instead of DRFA.
[ https://issues.apache.org/jira/browse/HBASE-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene resolved HBASE-5756. -- Resolution: Invalid Closing at invalid since it doesn't apply for any of the current versions. we can change defalult File Appender to RFA instead of DRFA. Key: HBASE-5756 URL: https://issues.apache.org/jira/browse/HBASE-5756 Project: HBase Issue Type: Improvement Reporter: Rohith Priority: Minor This can be a point of concern when on a certain day the logging happens more because of more and more activity. In that case the log file for that day can grow huge. These logs can not be opened for analysis since size is more. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12817) Data missing while scanning using PREFIX_TREE data block encoding
[
https://issues.apache.org/jira/browse/HBASE-12817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangduo updated HBASE-12817:
-
Attachment: HBASE-12817-branch-1.patch
patch for branch-1
Data missing while scanning using PREFIX_TREE data block encoding
-
Key: HBASE-12817
URL: https://issues.apache.org/jira/browse/HBASE-12817
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.98.9
Reporter: zhangduo
Assignee: zhangduo
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12817-branch-1.patch, HBASE-12817.patch
write a testcase like this
{code}
@Test
public void test() throws IOException {
for (int i = 0; i 100; i++) {
region.put(new Put(Bytes.toBytes(obj + (2900 + i))).add(fam, qual1,
Bytes.toBytes(i)));
}
region.put(new Put(Bytes.toBytes(obj299)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj29)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj2)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj3)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.flushcache(true);
Scan scan = new Scan(Bytes.toBytes(obj29995));
RegionScanner scanner = region.getScanner(scan);
ListCell cells = new ArrayListCell();
assertFalse(scanner.next(cells));
assertArrayEquals(Bytes.toBytes(obj3), Result.create(cells).getRow());
}
{code}
use obj29995 to scan should return obj3, but obj2990 is returned.
Seems a bug introduced by the fix of HBASE-11728.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12817) Data missing while scanning using PREFIX_TREE data block encoding
[
https://issues.apache.org/jira/browse/HBASE-12817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangduo updated HBASE-12817:
-
Attachment: (was: HBASE-12817-branch-1.patch)
Data missing while scanning using PREFIX_TREE data block encoding
-
Key: HBASE-12817
URL: https://issues.apache.org/jira/browse/HBASE-12817
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.98.9
Reporter: zhangduo
Assignee: zhangduo
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12817.patch
write a testcase like this
{code}
@Test
public void test() throws IOException {
for (int i = 0; i 100; i++) {
region.put(new Put(Bytes.toBytes(obj + (2900 + i))).add(fam, qual1,
Bytes.toBytes(i)));
}
region.put(new Put(Bytes.toBytes(obj299)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj29)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj2)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj3)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.flushcache(true);
Scan scan = new Scan(Bytes.toBytes(obj29995));
RegionScanner scanner = region.getScanner(scan);
ListCell cells = new ArrayListCell();
assertFalse(scanner.next(cells));
assertArrayEquals(Bytes.toBytes(obj3), Result.create(cells).getRow());
}
{code}
use obj29995 to scan should return obj3, but obj2990 is returned.
Seems a bug introduced by the fix of HBASE-11728.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-11295) Long running scan produces OutOfOrderScannerNextException
[ https://issues.apache.org/jira/browse/HBASE-11295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268908#comment-14268908 ] Lars Hofhansl edited comment on HBASE-11295 at 1/8/15 7:04 AM: --- We actually ran into the same issue. Our filter was a timerange specified on the scan object, that filtered a large percentage of the data. The client would time out because it did not hear back from the server, retries, and boom we get this exception. With the patch that we came up with above this no longer happens. [~anoop.hbase], could you have a quick look and confirm that this will not break anything new? was (Author: lhofhansl): We actually ran into the same issue. Out filter was a timerange specified on the scan object, that filtered a large percentage of the data. The client would time because it did not hear back from the server, retries, and boom we get this exception. With the patch that we came up with above this no longer happens. [~anoop.hbase], could you have a quick look and confirm that this will not break anything new? Long running scan produces OutOfOrderScannerNextException - Key: HBASE-11295 URL: https://issues.apache.org/jira/browse/HBASE-11295 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.96.0 Reporter: Jeff Cunningham Assignee: Andrew Purtell Priority: Critical Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: OutOfOrderScannerNextException.tar.gz Attached Files: HRegionServer.java - instramented from 0.96.1.1-cdh5.0.0 HBaseLeaseTimeoutIT.java - reproducing JUnit 4 test WaitFilter.java - Scan filter (extends FilterBase) that overrides filterRowKey() to sleep during invocation SpliceFilter.proto - Protobuf defintiion for WaitFilter.java OutOfOrderScann_InstramentedServer.log - instramented server log Steps.txt - this note Set up: In HBaseLeaseTimeoutIT, create a scan, set the given filter (which sleeps in overridden filterRowKey() method) and set it on the scan, and scan the table. This is done in test client_0x0_server_15x10(). Here's what I'm seeing (see also attached log): A new request comes into server (ID 1940798815214593802 - RpcServer.handler=96) and a RegionScanner is created for it, cached by ID, immediately looked up again and cached RegionScannerHolder's nextCallSeq incremeted (now at 1). The RegionScan thread goes to sleep in WaitFilter#filterRowKey(). A short (variable) period later, another request comes into the server (ID 8946109289649235722 - RpcServer.handler=98) and the same series of events happen to this request. At this point both RegionScanner threads are sleeping in WaitFilter.filterRowKey(). After another period, the client retries another scan request which thinks its next_call_seq is 0. However, HRegionServer's cached RegionScannerHolder thinks the matching RegionScanner's nextCallSeq should be 1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11144) Filter to support scan multiple row key ranges
[
https://issues.apache.org/jira/browse/HBASE-11144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268946#comment-14268946
]
Hadoop QA commented on HBASE-11144:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690724/HBASE_11144_V11.patch
against master branch at commit 645fbd7d87450b31b67b1e535cdb9c1cf50ffd16.
ATTACHMENT ID: 12690724
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12357//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12357//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12357//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12357//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12357//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12357//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12357//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12357//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12357//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12357//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12357//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12357//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12357//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12357//console
This message is automatically generated.
Filter to support scan multiple row key ranges
--
Key: HBASE-11144
URL: https://issues.apache.org/jira/browse/HBASE-11144
Project: HBase
Issue Type: Improvement
Components: Filters
Reporter: Jiajia Li
Assignee: Jiajia Li
Attachments: HBASE_11144_4.patch, HBASE_11144_V10.patch,
HBASE_11144_V11.patch, HBASE_11144_V5.patch, HBASE_11144_V6.patch,
HBASE_11144_V7.patch, HBASE_11144_V9.patch, MultiRowRangeFilter.patch,
MultiRowRangeFilter2.patch, MultiRowRangeFilter3.patch, hbase_11144_V8.patch
HBase is quite efficient when scanning only one small row key range. If user
needs to specify multiple row key ranges in one scan, the typical solutions
are: 1. through FilterList which is a list of row key Filters, 2. using the
SQL layer over HBase to join with two table, such as hive, phoenix etc.
However, both solutions are inefficient. Both of them can’t utilize the range
info to perform fast forwarding during scan which is quite time consuming. If
the number of ranges are quite big (e.g. millions), join is a proper solution
though it is slow. However, there are cases that user wants to specify a
small number of ranges to scan (e.g. 1000 ranges). Both solutions can’t
provide satisfactory performance in such case.
We provide this filter
[jira] [Resolved] (HBASE-6753) Potential bug in Put object toString()
[ https://issues.apache.org/jira/browse/HBASE-6753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene resolved HBASE-6753. -- Resolution: Cannot Reproduce I'm suspecting a side effect of something else. Resolving as cannot reproduce [~wal] In case you think this is still happening please update accordingly. Potential bug in Put object toString() -- Key: HBASE-6753 URL: https://issues.apache.org/jira/browse/HBASE-6753 Project: HBase Issue Type: Bug Components: Coprocessors Environment: Cloudera CDH 4.0.1 with hbase 0.92.1-cdh4.0.1 Reporter: Walter Tietze Priority: Minor I'm a newbie to HBase. I implemented a coprocessor which is pretty nice with Cloudera version 4.0.1. Testing my copressor evolved a problem, because everytime I inserted logging into my prePut-method, the Put object was not stored anymore into HBase. I analyzed the code and could reduce the problem to the fact, that calling the toString-method on the Put object alone, is the reason for this behaviour. There seems to be a problem with the serialization of the object. Serialization seems to modifiy the object with the result, that it is not inserted in HBase anymore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-6403) RegionCoprocessorHost provides empty config when loading a coprocessor
[ https://issues.apache.org/jira/browse/HBASE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene updated HBASE-6403: - Labels: delete (was: ) RegionCoprocessorHost provides empty config when loading a coprocessor -- Key: HBASE-6403 URL: https://issues.apache.org/jira/browse/HBASE-6403 Project: HBase Issue Type: Bug Components: regionserver Reporter: Eric Newton Priority: Minor Labels: delete I started playing with Giraffa. I am running it against Hadoop 2.0.0-alpha, and current HBase trunk. On line 159 of RegionCoprocessorHost, the server's configuration is copied... or at least an attempt is made to copy it. However, the server's configuration object, a CompoundConfiguration, does not store the data in the same way as the base Configuration object, and so nothing is copied. This leaves the coprocessor without access to configuration values, like the fs.defaultFS, which Giraffa is looking for. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6403) RegionCoprocessorHost provides empty config when loading a coprocessor
[ https://issues.apache.org/jira/browse/HBASE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268943#comment-14268943 ] Cosmin Lehene commented on HBASE-6403: -- I'm not sure if this still the case. I'm assuming this was happening in the constructor. [~ecn], [~te...@apache.org] can you look at this and if still the case add the actual method to the description or close otherwise? RegionCoprocessorHost provides empty config when loading a coprocessor -- Key: HBASE-6403 URL: https://issues.apache.org/jira/browse/HBASE-6403 Project: HBase Issue Type: Bug Components: regionserver Reporter: Eric Newton Priority: Minor Labels: delete I started playing with Giraffa. I am running it against Hadoop 2.0.0-alpha, and current HBase trunk. On line 159 of RegionCoprocessorHost, the server's configuration is copied... or at least an attempt is made to copy it. However, the server's configuration object, a CompoundConfiguration, does not store the data in the same way as the base Configuration object, and so nothing is copied. This leaves the coprocessor without access to configuration values, like the fs.defaultFS, which Giraffa is looking for. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12782) ITBLL fails for me if generator does anything but 5M per maptask
[ https://issues.apache.org/jira/browse/HBASE-12782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268766#comment-14268766 ] stack commented on HBASE-12782: --- Sorry. This one is taking me a while. Nothing obviously wrong in the processing but definetly missing rows. Trying to figure if writing or WAL replay. ITBLL fails for me if generator does anything but 5M per maptask Key: HBASE-12782 URL: https://issues.apache.org/jira/browse/HBASE-12782 Project: HBase Issue Type: Bug Components: integration tests Affects Versions: 1.0.0 Reporter: stack Priority: Critical Fix For: 1.0.0 Anyone else seeing this? If I do an ITBLL with generator doing 5M rows per maptask, all is good -- verify passes. I've been running 5 servers and had one splot per server. So below works: HADOOP_CLASSPATH=/home/stack/conf_hbase:`/home/stack/hbase/bin/hbase classpath` ./hadoop/bin/hadoop --config ~/conf_hadoop org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList --monkey serverKilling Generator 5 500 g1.tmp or if I double the map tasks, it works: HADOOP_CLASSPATH=/home/stack/conf_hbase:`/home/stack/hbase/bin/hbase classpath` ./hadoop/bin/hadoop --config ~/conf_hadoop org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList --monkey serverKilling Generator 10 500 g2.tmp ...but if I change the 5M to 50M or 25M, Verify fails. Looking into it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12791) HBase does not attempt to clean up an aborted split when the regionserver shutting down
[
https://issues.apache.org/jira/browse/HBASE-12791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajeshbabu Chintaguntla updated HBASE-12791:
Attachment: HBASE-12791_v4.patch
bq. We are doing some action without a parameter check (-fixAssignments,
-fixMeta, etc). Hbck running without any parameters should never do any
destructive action. Can we do this by adding a parameter, smt like
-fixFailedSplitAttempts.
It's in shouldFixMeta case [~enis]. So it won't be a problem.
bq. Can we save the results after we sort the regions for the table
In the current patch storing the regions derived from meta in TableInfo.
HBase does not attempt to clean up an aborted split when the regionserver
shutting down
---
Key: HBASE-12791
URL: https://issues.apache.org/jira/browse/HBASE-12791
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.0
Reporter: Rajeshbabu Chintaguntla
Assignee: Rajeshbabu Chintaguntla
Priority: Critical
Fix For: 2.0.0, 0.98.10, 1.0.1
Attachments: HBASE-12791.patch, HBASE-12791_98.patch,
HBASE-12791_branch1.patch, HBASE-12791_v2.patch, HBASE-12791_v3.patch,
HBASE-12791_v4.patch
HBase not cleaning the daughter region directories from HDFS if region
server shut down after creating the daughter region directories during the
split.
Here the logs.
- RS shutdown after creating the daughter regions.
{code}
2014-12-31 09:05:41,406 DEBUG [regionserver60020-splits-1419996941385]
zookeeper.ZKAssign: regionserver:60020-0x14a9701e53100d1,
quorum=localhost:2181, baseZNode=/hbase Transitioned node
80c665138d4fa32da4d792d8ed13206f from RS_ZK_REQUEST_REGION_SPLIT to
RS_ZK_REQUEST_REGION_SPLIT
2014-12-31 09:05:41,514 DEBUG [regionserver60020-splits-1419996941385]
regionserver.HRegion: Closing
t,,1419996880699.80c665138d4fa32da4d792d8ed13206f.: disabling compactions
flushes
2014-12-31 09:05:41,514 DEBUG [regionserver60020-splits-1419996941385]
regionserver.HRegion: Updates disabled for region
t,,1419996880699.80c665138d4fa32da4d792d8ed13206f.
2014-12-31 09:05:41,516 INFO
[StoreCloserThread-t,,1419996880699.80c665138d4fa32da4d792d8ed13206f.-1]
regionserver.HStore: Closed f
2014-12-31 09:05:41,518 INFO [regionserver60020-splits-1419996941385]
regionserver.HRegion: Closed
t,,1419996880699.80c665138d4fa32da4d792d8ed13206f.
2014-12-31 09:05:49,922 DEBUG [regionserver60020-splits-1419996941385]
regionserver.MetricsRegionSourceImpl: Creating new MetricsRegionSourceImpl
for table t dd9731ee43b104da565257ca1539aa8c
2014-12-31 09:05:49,922 DEBUG [regionserver60020-splits-1419996941385]
regionserver.HRegion: Instantiated
t,,1419996941401.dd9731ee43b104da565257ca1539aa8c.
2014-12-31 09:05:49,929 DEBUG [regionserver60020-splits-1419996941385]
regionserver.MetricsRegionSourceImpl: Creating new MetricsRegionSourceImpl
for table t 2e40a44511c0e187d357d651f13a1dab
2014-12-31 09:05:49,929 DEBUG [regionserver60020-splits-1419996941385]
regionserver.HRegion: Instantiated
t,row2,1419996941401.2e40a44511c0e187d357d651f13a1dab.
Wed Dec 31 09:06:30 IST 2014 Terminating regionserver
2014-12-31 09:06:30,465 INFO [Thread-8] regionserver.ShutdownHook: Shutdown
hook starting; hbase.shutdown.hook=true;
fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@42d2282e
{code}
- Skipping rollback if RS stopped or stopping so we end up in dirty daughter
regions in HDFS.
{code}
2014-12-31 09:07:49,547 INFO [regionserver60020-splits-1419996941385]
regionserver.SplitRequest: Skip rollback/cleanup of failed split of
t,,1419996880699.80c665138d4fa32da4d792d8ed13206f. because server is stopped
java.io.InterruptedIOException: Interrupted after 0 tries on 350
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:156)
{code}
Because of this hbck always showing inconsistencies.
{code}
ERROR: Region { meta = null, hdfs =
hdfs://localhost:9000/hbase/data/default/t/2e40a44511c0e187d357d651f13a1dab,
deployed = } on HDFS, but not listed in hbase:meta or deployed on any
region server
ERROR: Region { meta = null, hdfs =
hdfs://localhost:9000/hbase/data/default/t/dd9731ee43b104da565257ca1539aa8c,
deployed = } on HDFS, but not listed in hbase:meta or deployed on any
region server
{code}
If we try to repair then we end up in overlap regions in hbase:meta. and both
daughter regions and parent are online.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12817) Data missing while scanning using PREFIX_TREE data block encoding
[
https://issues.apache.org/jira/browse/HBASE-12817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-12817:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks for the quick turn around.
Data missing while scanning using PREFIX_TREE data block encoding
-
Key: HBASE-12817
URL: https://issues.apache.org/jira/browse/HBASE-12817
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.98.9
Reporter: zhangduo
Assignee: zhangduo
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12817-0.98.patch, HBASE-12817-branch-1.patch,
HBASE-12817.patch, HBASE-12817_1.patch
write a testcase like this
{code}
@Test
public void test() throws IOException {
for (int i = 0; i 100; i++) {
region.put(new Put(Bytes.toBytes(obj + (2900 + i))).add(fam, qual1,
Bytes.toBytes(i)));
}
region.put(new Put(Bytes.toBytes(obj299)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj29)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj2)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj3)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.flushcache(true);
Scan scan = new Scan(Bytes.toBytes(obj29995));
RegionScanner scanner = region.getScanner(scan);
ListCell cells = new ArrayListCell();
assertFalse(scanner.next(cells));
assertArrayEquals(Bytes.toBytes(obj3), Result.create(cells).getRow());
}
{code}
use obj29995 to scan should return obj3, but obj2990 is returned.
Seems a bug introduced by the fix of HBASE-11728.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11144) Filter to support scan multiple row key ranges
[ https://issues.apache.org/jira/browse/HBASE-11144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268852#comment-14268852 ] ramkrishna.s.vasudevan commented on HBASE-11144: Make the '-1' as constant - something like ROW_BEFORE_FIRST_RANGE (may be some better name). One more review from others would be great before we commit this. Filter to support scan multiple row key ranges -- Key: HBASE-11144 URL: https://issues.apache.org/jira/browse/HBASE-11144 Project: HBase Issue Type: Improvement Components: Filters Reporter: Jiajia Li Assignee: Jiajia Li Attachments: HBASE_11144_4.patch, HBASE_11144_V10.patch, HBASE_11144_V5.patch, HBASE_11144_V6.patch, HBASE_11144_V7.patch, HBASE_11144_V9.patch, MultiRowRangeFilter.patch, MultiRowRangeFilter2.patch, MultiRowRangeFilter3.patch, hbase_11144_V8.patch HBase is quite efficient when scanning only one small row key range. If user needs to specify multiple row key ranges in one scan, the typical solutions are: 1. through FilterList which is a list of row key Filters, 2. using the SQL layer over HBase to join with two table, such as hive, phoenix etc. However, both solutions are inefficient. Both of them can’t utilize the range info to perform fast forwarding during scan which is quite time consuming. If the number of ranges are quite big (e.g. millions), join is a proper solution though it is slow. However, there are cases that user wants to specify a small number of ranges to scan (e.g. 1000 ranges). Both solutions can’t provide satisfactory performance in such case. We provide this filter (MultiRowRangeFilter) to support such use case (scan multiple row key ranges), which can construct the row key ranges from user specified list and perform fast-forwarding during scan. Thus, the scan will be quite efficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12817) Data missing while scanning using PREFIX_TREE data block encoding
[
https://issues.apache.org/jira/browse/HBASE-12817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268893#comment-14268893
]
Hudson commented on HBASE-12817:
FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #745 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/745/])
HBASE-12817 Data missing while scanning using PREFIX_TREE data block encoding
(Duo Zhang) (tedyu: rev 6ef814e6d0bae2ba081a2c7a396f523689e7a15a)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestPrefixTree.java
*
hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/decode/PrefixTreeArrayScanner.java
*
hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/decode/PrefixTreeArrayReversibleScanner.java
*
hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/PrefixTreeSeeker.java
Data missing while scanning using PREFIX_TREE data block encoding
-
Key: HBASE-12817
URL: https://issues.apache.org/jira/browse/HBASE-12817
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.98.9
Reporter: zhangduo
Assignee: zhangduo
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12817-0.98.patch, HBASE-12817-branch-1.patch,
HBASE-12817.patch, HBASE-12817_1.patch
write a testcase like this
{code}
@Test
public void test() throws IOException {
for (int i = 0; i 100; i++) {
region.put(new Put(Bytes.toBytes(obj + (2900 + i))).add(fam, qual1,
Bytes.toBytes(i)));
}
region.put(new Put(Bytes.toBytes(obj299)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj29)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj2)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj3)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.flushcache(true);
Scan scan = new Scan(Bytes.toBytes(obj29995));
RegionScanner scanner = region.getScanner(scan);
ListCell cells = new ArrayListCell();
assertFalse(scanner.next(cells));
assertArrayEquals(Bytes.toBytes(obj3), Result.create(cells).getRow());
}
{code}
use obj29995 to scan should return obj3, but obj2990 is returned.
Seems a bug introduced by the fix of HBASE-11728.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11295) Long running scan produces OutOfOrderScannerNextException
[ https://issues.apache.org/jira/browse/HBASE-11295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11295: --- Fix Version/s: 1.1.0 0.98.10 2.0.0 1.0.0 Assignee: Andrew Purtell Long running scan produces OutOfOrderScannerNextException - Key: HBASE-11295 URL: https://issues.apache.org/jira/browse/HBASE-11295 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.96.0 Reporter: Jeff Cunningham Assignee: Andrew Purtell Priority: Critical Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: OutOfOrderScannerNextException.tar.gz Attached Files: HRegionServer.java - instramented from 0.96.1.1-cdh5.0.0 HBaseLeaseTimeoutIT.java - reproducing JUnit 4 test WaitFilter.java - Scan filter (extends FilterBase) that overrides filterRowKey() to sleep during invocation SpliceFilter.proto - Protobuf defintiion for WaitFilter.java OutOfOrderScann_InstramentedServer.log - instramented server log Steps.txt - this note Set up: In HBaseLeaseTimeoutIT, create a scan, set the given filter (which sleeps in overridden filterRowKey() method) and set it on the scan, and scan the table. This is done in test client_0x0_server_15x10(). Here's what I'm seeing (see also attached log): A new request comes into server (ID 1940798815214593802 - RpcServer.handler=96) and a RegionScanner is created for it, cached by ID, immediately looked up again and cached RegionScannerHolder's nextCallSeq incremeted (now at 1). The RegionScan thread goes to sleep in WaitFilter#filterRowKey(). A short (variable) period later, another request comes into the server (ID 8946109289649235722 - RpcServer.handler=98) and the same series of events happen to this request. At this point both RegionScanner threads are sleeping in WaitFilter.filterRowKey(). After another period, the client retries another scan request which thinks its next_call_seq is 0. However, HRegionServer's cached RegionScannerHolder thinks the matching RegionScanner's nextCallSeq should be 1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12821) Describe on table doesn't show table attributes on hbase shell
Amit Kabra created HBASE-12821:
--
Summary: Describe on table doesn't show table attributes on hbase
shell
Key: HBASE-12821
URL: https://issues.apache.org/jira/browse/HBASE-12821
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.8
Reporter: Amit Kabra
Priority: Minor
1) hbase(main):003:0 create 'test','CF'
2) hbase(main):006:0 alter 'test', METADATA = {'TEST_PROPERTY' =
'TEST_VALUE'}
3) hbase(main):007:0 describe 'test'
{NAME = 'CF', DATA_BLOCK_ENCODING = 'NONE', BLOOMFILTER = 'ROW',
REPLICATION_SCOPE = '0', VERSIONS = '1', COMPRESSION = 'NONE', MIN_VERSIONS
= '0', TTL = 'FOREVER', KEEP_DELETED_CELLS = 'FALSE', BLOCKSIZE = '65536',
IN_MEMORY = 'false', BLOCKCACHE = 'true'}
Issue : The added property , table attribute, isn't getting displayed.
Note : If we check the table description from master page, we can see the
changed property.
'test', {TABLE_ATTRIBUTES = {METADATA = {'TEST_PROPERTY' = 'TEST_VALUE'}},
{NAME = 'CF'}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HBASE-12816) GC logs are lost upon Region Server restart if GCLogFileRotation is enabled
[ https://issues.apache.org/jira/browse/HBASE-12816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-12816 started by Abhishek Singh Chouhan. -- GC logs are lost upon Region Server restart if GCLogFileRotation is enabled --- Key: HBASE-12816 URL: https://issues.apache.org/jira/browse/HBASE-12816 Project: HBase Issue Type: Bug Components: scripts Reporter: Abhishek Singh Chouhan Assignee: Abhishek Singh Chouhan Priority: Minor When -XX:+UseGCLogFileRotation is used gc log files end with .gc.0 instead of .gc. hbase_rotate_log () in hbase-daemon.sh does not handle this correctly and hence when a RS is restarted old gc logs are lost(overwritten). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12817) Data missing while scanning using PREFIX_TREE data block encoding
[
https://issues.apache.org/jira/browse/HBASE-12817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268819#comment-14268819
]
Hudson commented on HBASE-12817:
FAILURE: Integrated in HBase-1.0 #641 (See
[https://builds.apache.org/job/HBase-1.0/641/])
HBASE-12817 Data missing while scanning using PREFIX_TREE data block encoding
(Duo Zhang) (tedyu: rev 4648433fdc5490c98d5d1f963835e282d9f79c2c)
*
hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/decode/PrefixTreeArrayScanner.java
*
hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/PrefixTreeSeeker.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestPrefixTree.java
*
hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/decode/PrefixTreeArrayReversibleScanner.java
Data missing while scanning using PREFIX_TREE data block encoding
-
Key: HBASE-12817
URL: https://issues.apache.org/jira/browse/HBASE-12817
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.98.9
Reporter: zhangduo
Assignee: zhangduo
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12817-0.98.patch, HBASE-12817-branch-1.patch,
HBASE-12817.patch, HBASE-12817_1.patch
write a testcase like this
{code}
@Test
public void test() throws IOException {
for (int i = 0; i 100; i++) {
region.put(new Put(Bytes.toBytes(obj + (2900 + i))).add(fam, qual1,
Bytes.toBytes(i)));
}
region.put(new Put(Bytes.toBytes(obj299)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj29)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj2)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj3)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.flushcache(true);
Scan scan = new Scan(Bytes.toBytes(obj29995));
RegionScanner scanner = region.getScanner(scan);
ListCell cells = new ArrayListCell();
assertFalse(scanner.next(cells));
assertArrayEquals(Bytes.toBytes(obj3), Result.create(cells).getRow());
}
{code}
use obj29995 to scan should return obj3, but obj2990 is returned.
Seems a bug introduced by the fix of HBASE-11728.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11144) Filter to support scan multiple row key ranges
[ https://issues.apache.org/jira/browse/HBASE-11144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiajia Li updated HBASE-11144: -- Attachment: HBASE_11144_V11.patch Filter to support scan multiple row key ranges -- Key: HBASE-11144 URL: https://issues.apache.org/jira/browse/HBASE-11144 Project: HBase Issue Type: Improvement Components: Filters Reporter: Jiajia Li Assignee: Jiajia Li Attachments: HBASE_11144_4.patch, HBASE_11144_V10.patch, HBASE_11144_V11.patch, HBASE_11144_V5.patch, HBASE_11144_V6.patch, HBASE_11144_V7.patch, HBASE_11144_V9.patch, MultiRowRangeFilter.patch, MultiRowRangeFilter2.patch, MultiRowRangeFilter3.patch, hbase_11144_V8.patch HBase is quite efficient when scanning only one small row key range. If user needs to specify multiple row key ranges in one scan, the typical solutions are: 1. through FilterList which is a list of row key Filters, 2. using the SQL layer over HBase to join with two table, such as hive, phoenix etc. However, both solutions are inefficient. Both of them can’t utilize the range info to perform fast forwarding during scan which is quite time consuming. If the number of ranges are quite big (e.g. millions), join is a proper solution though it is slow. However, there are cases that user wants to specify a small number of ranges to scan (e.g. 1000 ranges). Both solutions can’t provide satisfactory performance in such case. We provide this filter (MultiRowRangeFilter) to support such use case (scan multiple row key ranges), which can construct the row key ranges from user specified list and perform fast-forwarding during scan. Thus, the scan will be quite efficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Singhi updated HBASE-5878:
-
Attachment: HBASE-5878.patch
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Fix For: 1.0.0
Attachments: HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12817) Data missing while scanning using PREFIX_TREE data block encoding
[
https://issues.apache.org/jira/browse/HBASE-12817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268770#comment-14268770
]
Hadoop QA commented on HBASE-12817:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690704/HBASE-12817-branch-1.patch
against master branch at commit 645fbd7d87450b31b67b1e535cdb9c1cf50ffd16.
ATTACHMENT ID: 12690704
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12354//console
This message is automatically generated.
Data missing while scanning using PREFIX_TREE data block encoding
-
Key: HBASE-12817
URL: https://issues.apache.org/jira/browse/HBASE-12817
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.98.9
Reporter: zhangduo
Assignee: zhangduo
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12817.patch
write a testcase like this
{code}
@Test
public void test() throws IOException {
for (int i = 0; i 100; i++) {
region.put(new Put(Bytes.toBytes(obj + (2900 + i))).add(fam, qual1,
Bytes.toBytes(i)));
}
region.put(new Put(Bytes.toBytes(obj299)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj29)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj2)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj3)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.flushcache(true);
Scan scan = new Scan(Bytes.toBytes(obj29995));
RegionScanner scanner = region.getScanner(scan);
ListCell cells = new ArrayListCell();
assertFalse(scanner.next(cells));
assertArrayEquals(Bytes.toBytes(obj3), Result.create(cells).getRow());
}
{code}
use obj29995 to scan should return obj3, but obj2990 is returned.
Seems a bug introduced by the fix of HBASE-11728.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11144) Filter to support scan multiple row key ranges
[
https://issues.apache.org/jira/browse/HBASE-11144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268784#comment-14268784
]
Hadoop QA commented on HBASE-11144:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690692/HBASE_11144_V10.patch
against master branch at commit 7dce0d5c4549a1d5986d3bc82734924219d8bb25.
ATTACHMENT ID: 12690692
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12353//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12353//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12353//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12353//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12353//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12353//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12353//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12353//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12353//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12353//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12353//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12353//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12353//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12353//console
This message is automatically generated.
Filter to support scan multiple row key ranges
--
Key: HBASE-11144
URL: https://issues.apache.org/jira/browse/HBASE-11144
Project: HBase
Issue Type: Improvement
Components: Filters
Reporter: Jiajia Li
Assignee: Jiajia Li
Attachments: HBASE_11144_4.patch, HBASE_11144_V10.patch,
HBASE_11144_V5.patch, HBASE_11144_V6.patch, HBASE_11144_V7.patch,
HBASE_11144_V9.patch, MultiRowRangeFilter.patch, MultiRowRangeFilter2.patch,
MultiRowRangeFilter3.patch, hbase_11144_V8.patch
HBase is quite efficient when scanning only one small row key range. If user
needs to specify multiple row key ranges in one scan, the typical solutions
are: 1. through FilterList which is a list of row key Filters, 2. using the
SQL layer over HBase to join with two table, such as hive, phoenix etc.
However, both solutions are inefficient. Both of them can’t utilize the range
info to perform fast forwarding during scan which is quite time consuming. If
the number of ranges are quite big (e.g. millions), join is a proper solution
though it is slow. However, there are cases that user wants to specify a
small number of ranges to scan (e.g. 1000 ranges). Both solutions can’t
provide satisfactory performance in such case.
We provide this filter (MultiRowRangeFilter) to
[jira] [Commented] (HBASE-12817) Data missing while scanning using PREFIX_TREE data block encoding
[
https://issues.apache.org/jira/browse/HBASE-12817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268799#comment-14268799
]
Hudson commented on HBASE-12817:
FAILURE: Integrated in HBase-TRUNK #6002 (See
[https://builds.apache.org/job/HBase-TRUNK/6002/])
HBASE-12817 Data missing while scanning using PREFIX_TREE data block encoding
(Duo Zhang) (tedyu: rev 645fbd7d87450b31b67b1e535cdb9c1cf50ffd16)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestPrefixTree.java
*
hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/decode/PrefixTreeArrayScanner.java
*
hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/PrefixTreeSeeker.java
*
hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/decode/PrefixTreeArrayReversibleScanner.java
Data missing while scanning using PREFIX_TREE data block encoding
-
Key: HBASE-12817
URL: https://issues.apache.org/jira/browse/HBASE-12817
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.98.9
Reporter: zhangduo
Assignee: zhangduo
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12817-0.98.patch, HBASE-12817-branch-1.patch,
HBASE-12817.patch, HBASE-12817_1.patch
write a testcase like this
{code}
@Test
public void test() throws IOException {
for (int i = 0; i 100; i++) {
region.put(new Put(Bytes.toBytes(obj + (2900 + i))).add(fam, qual1,
Bytes.toBytes(i)));
}
region.put(new Put(Bytes.toBytes(obj299)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj29)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj2)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj3)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.flushcache(true);
Scan scan = new Scan(Bytes.toBytes(obj29995));
RegionScanner scanner = region.getScanner(scan);
ListCell cells = new ArrayListCell();
assertFalse(scanner.next(cells));
assertArrayEquals(Bytes.toBytes(obj3), Result.create(cells).getRow());
}
{code}
use obj29995 to scan should return obj3, but obj2990 is returned.
Seems a bug introduced by the fix of HBASE-11728.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12817) Data missing while scanning using PREFIX_TREE data block encoding
[
https://issues.apache.org/jira/browse/HBASE-12817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268860#comment-14268860
]
Hudson commented on HBASE-12817:
SUCCESS: Integrated in HBase-1.1 #66 (See
[https://builds.apache.org/job/HBase-1.1/66/])
HBASE-12817 Data missing while scanning using PREFIX_TREE data block encoding
(Duo Zhang) (tedyu: rev 29ec882a5e0269ce453cf806727551a4f28020a5)
*
hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/decode/PrefixTreeArrayScanner.java
*
hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/PrefixTreeSeeker.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestPrefixTree.java
*
hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/decode/PrefixTreeArrayReversibleScanner.java
Data missing while scanning using PREFIX_TREE data block encoding
-
Key: HBASE-12817
URL: https://issues.apache.org/jira/browse/HBASE-12817
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.98.9
Reporter: zhangduo
Assignee: zhangduo
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12817-0.98.patch, HBASE-12817-branch-1.patch,
HBASE-12817.patch, HBASE-12817_1.patch
write a testcase like this
{code}
@Test
public void test() throws IOException {
for (int i = 0; i 100; i++) {
region.put(new Put(Bytes.toBytes(obj + (2900 + i))).add(fam, qual1,
Bytes.toBytes(i)));
}
region.put(new Put(Bytes.toBytes(obj299)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj29)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj2)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj3)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.flushcache(true);
Scan scan = new Scan(Bytes.toBytes(obj29995));
RegionScanner scanner = region.getScanner(scan);
ListCell cells = new ArrayListCell();
assertFalse(scanner.next(cells));
assertArrayEquals(Bytes.toBytes(obj3), Result.create(cells).getRow());
}
{code}
use obj29995 to scan should return obj3, but obj2990 is returned.
Seems a bug introduced by the fix of HBASE-11728.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Singhi updated HBASE-5878:
-
Fix Version/s: 2.0.0
Assignee: Ashish Singhi (was: Uma Maheswara Rao G)
Status: Patch Available (was: Open)
Patch for master branch.
Please review and share your thoughts.
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 1.0.0, 2.0.0
Attachments: HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12745) Visibility Labels: support visibility labels for user groups.
[
https://issues.apache.org/jira/browse/HBASE-12745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268906#comment-14268906
]
Anoop Sam John commented on HBASE-12745:
One suggestion
{code}
- public boolean havingSystemAuth(byte[] user) throws IOException {
+ public boolean havingSystemAuth(User user) throws IOException {
// A super user has 'system' auth.
if (isSystemOrSuperUser(user)) {
return true;
}
// A user can also be explicitly granted 'system' auth.
-ListString auths = this.getAuths(user, true);
+SetString auths = new HashSetString();
+auths.addAll(this.getUserAuths(Bytes.toBytes(user.getShortName()), true));
+auths.addAll(this.getGroupAuths(user.getGroupNames(), true));
if (LOG.isTraceEnabled()) {
- LOG.trace(The auths for user + Bytes.toString(user) + are +
auths);
+ LOG.trace(The auths for user + user.getShortName() + are + auths);
}
return auths.contains(SYSTEM_LABEL);
}
{code}
Better do early check for SYSTEM_LABEL for user auths and early out. Then go
with group auths
Else looks good..
Visibility Labels: support visibility labels for user groups.
--
Key: HBASE-12745
URL: https://issues.apache.org/jira/browse/HBASE-12745
Project: HBase
Issue Type: Improvement
Components: security
Affects Versions: 1.0.0, 0.98.9, 0.99.2
Reporter: Jerry He
Assignee: Jerry He
Fix For: 2.0.0
Attachments: HBASE-12745-master-v1.patch,
HBASE-12745-master-v2.patch, HBASE-12745-master-v3.patch
The thinking is that we should support visibility labels to be associated
with user groups.
We will then be able grant visibility labels to a group in addition to
individual users, which provides convenience and usability.
We will use '@group' to denote a group name, as similarly done in
AcccessController.
For example,
{code}
set_auths '@group1', ['SECRET','PRIVATE']
{code}
{code}
get_auth '@group1'
{code}
A user belonging to 'group1' will have all the visibility labels granted to
'group1'
We'll also support super user groups as specified in hbase-site.xml.
The code update will mainly be on the server side VisibilityLabelService
implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12817) Data missing while scanning using PREFIX_TREE data block encoding
[
https://issues.apache.org/jira/browse/HBASE-12817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangduo updated HBASE-12817:
-
Attachment: HBASE-12817-0.98.patch
HBASE-12817-branch-1.patch
HBASE-12817_1.patch
The previous patch contains System.out used for debugging, sorry.
Use the new patch please, thanks.
I think the patch for branch-1 could also be applied to branch-1.0, if not
please let me know and I will prepare a patch for branch-1.0.
Thanks.
Data missing while scanning using PREFIX_TREE data block encoding
-
Key: HBASE-12817
URL: https://issues.apache.org/jira/browse/HBASE-12817
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.98.9
Reporter: zhangduo
Assignee: zhangduo
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12817-0.98.patch, HBASE-12817-branch-1.patch,
HBASE-12817.patch, HBASE-12817_1.patch
write a testcase like this
{code}
@Test
public void test() throws IOException {
for (int i = 0; i 100; i++) {
region.put(new Put(Bytes.toBytes(obj + (2900 + i))).add(fam, qual1,
Bytes.toBytes(i)));
}
region.put(new Put(Bytes.toBytes(obj299)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj29)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj2)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj3)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.flushcache(true);
Scan scan = new Scan(Bytes.toBytes(obj29995));
RegionScanner scanner = region.getScanner(scan);
ListCell cells = new ArrayListCell();
assertFalse(scanner.next(cells));
assertArrayEquals(Bytes.toBytes(obj3), Result.create(cells).getRow());
}
{code}
use obj29995 to scan should return obj3, but obj2990 is returned.
Seems a bug introduced by the fix of HBASE-11728.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12817) Data missing while scanning using PREFIX_TREE data block encoding
[
https://issues.apache.org/jira/browse/HBASE-12817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268780#comment-14268780
]
Hadoop QA commented on HBASE-12817:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690711/HBASE-12817-0.98.patch
against master branch at commit 645fbd7d87450b31b67b1e535cdb9c1cf50ffd16.
ATTACHMENT ID: 12690711
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12356//console
This message is automatically generated.
Data missing while scanning using PREFIX_TREE data block encoding
-
Key: HBASE-12817
URL: https://issues.apache.org/jira/browse/HBASE-12817
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.98.9
Reporter: zhangduo
Assignee: zhangduo
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12817-0.98.patch, HBASE-12817-branch-1.patch,
HBASE-12817.patch, HBASE-12817_1.patch
write a testcase like this
{code}
@Test
public void test() throws IOException {
for (int i = 0; i 100; i++) {
region.put(new Put(Bytes.toBytes(obj + (2900 + i))).add(fam, qual1,
Bytes.toBytes(i)));
}
region.put(new Put(Bytes.toBytes(obj299)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj29)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj2)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj3)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.flushcache(true);
Scan scan = new Scan(Bytes.toBytes(obj29995));
RegionScanner scanner = region.getScanner(scan);
ListCell cells = new ArrayListCell();
assertFalse(scanner.next(cells));
assertArrayEquals(Bytes.toBytes(obj3), Result.create(cells).getRow());
}
{code}
use obj29995 to scan should return obj3, but obj2990 is returned.
Seems a bug introduced by the fix of HBASE-11728.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12791) HBase does not attempt to clean up an aborted split when the regionserver shutting down
[
https://issues.apache.org/jira/browse/HBASE-12791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268802#comment-14268802
]
Hadoop QA commented on HBASE-12791:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690707/HBASE-12791_v4.patch
against master branch at commit 645fbd7d87450b31b67b1e535cdb9c1cf50ffd16.
ATTACHMENT ID: 12690707
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
2085 checkstyle errors (more than the master's current 2084 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12355//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12355//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12355//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12355//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12355//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12355//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12355//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12355//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12355//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12355//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12355//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12355//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12355//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12355//console
This message is automatically generated.
HBase does not attempt to clean up an aborted split when the regionserver
shutting down
---
Key: HBASE-12791
URL: https://issues.apache.org/jira/browse/HBASE-12791
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.0
Reporter: Rajeshbabu Chintaguntla
Assignee: Rajeshbabu Chintaguntla
Priority: Critical
Fix For: 2.0.0, 0.98.10, 1.0.1
Attachments: HBASE-12791.patch, HBASE-12791_98.patch,
HBASE-12791_branch1.patch, HBASE-12791_v2.patch, HBASE-12791_v3.patch,
HBASE-12791_v4.patch
HBase not cleaning the daughter region directories from HDFS if region
server shut down after creating the daughter region directories during the
split.
Here the logs.
- RS shutdown after creating the daughter regions.
{code}
2014-12-31 09:05:41,406 DEBUG [regionserver60020-splits-1419996941385]
zookeeper.ZKAssign: regionserver:60020-0x14a9701e53100d1,
quorum=localhost:2181, baseZNode=/hbase Transitioned node
80c665138d4fa32da4d792d8ed13206f from RS_ZK_REQUEST_REGION_SPLIT to
RS_ZK_REQUEST_REGION_SPLIT
2014-12-31 09:05:41,514 DEBUG [regionserver60020-splits-1419996941385]
regionserver.HRegion: Closing
[jira] [Commented] (HBASE-12817) Data missing while scanning using PREFIX_TREE data block encoding
[
https://issues.apache.org/jira/browse/HBASE-12817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268854#comment-14268854
]
Hudson commented on HBASE-12817:
FAILURE: Integrated in HBase-0.98 #780 (See
[https://builds.apache.org/job/HBase-0.98/780/])
HBASE-12817 Data missing while scanning using PREFIX_TREE data block encoding
(Duo Zhang) (tedyu: rev 6ef814e6d0bae2ba081a2c7a396f523689e7a15a)
*
hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/PrefixTreeSeeker.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestPrefixTree.java
*
hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/decode/PrefixTreeArrayReversibleScanner.java
*
hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/decode/PrefixTreeArrayScanner.java
Data missing while scanning using PREFIX_TREE data block encoding
-
Key: HBASE-12817
URL: https://issues.apache.org/jira/browse/HBASE-12817
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.98.9
Reporter: zhangduo
Assignee: zhangduo
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12817-0.98.patch, HBASE-12817-branch-1.patch,
HBASE-12817.patch, HBASE-12817_1.patch
write a testcase like this
{code}
@Test
public void test() throws IOException {
for (int i = 0; i 100; i++) {
region.put(new Put(Bytes.toBytes(obj + (2900 + i))).add(fam, qual1,
Bytes.toBytes(i)));
}
region.put(new Put(Bytes.toBytes(obj299)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj29)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj2)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.put(new Put(Bytes.toBytes(obj3)).add(fam, qual1,
Bytes.toBytes(whatever)));
region.flushcache(true);
Scan scan = new Scan(Bytes.toBytes(obj29995));
RegionScanner scanner = region.getScanner(scan);
ListCell cells = new ArrayListCell();
assertFalse(scanner.next(cells));
assertArrayEquals(Bytes.toBytes(obj3), Result.create(cells).getRow());
}
{code}
use obj29995 to scan should return obj3, but obj2990 is returned.
Seems a bug introduced by the fix of HBASE-11728.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11144) Filter to support scan multiple row key ranges
[ https://issues.apache.org/jira/browse/HBASE-11144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268867#comment-14268867 ] Jiajia Li commented on HBASE-11144: --- [~ram_krish] thanks for your review [~tedyu] [~anoopsamjohn], can you take some time to look at the latest patch? The review link is https://reviews.apache.org/r/21370/ Filter to support scan multiple row key ranges -- Key: HBASE-11144 URL: https://issues.apache.org/jira/browse/HBASE-11144 Project: HBase Issue Type: Improvement Components: Filters Reporter: Jiajia Li Assignee: Jiajia Li Attachments: HBASE_11144_4.patch, HBASE_11144_V10.patch, HBASE_11144_V11.patch, HBASE_11144_V5.patch, HBASE_11144_V6.patch, HBASE_11144_V7.patch, HBASE_11144_V9.patch, MultiRowRangeFilter.patch, MultiRowRangeFilter2.patch, MultiRowRangeFilter3.patch, hbase_11144_V8.patch HBase is quite efficient when scanning only one small row key range. If user needs to specify multiple row key ranges in one scan, the typical solutions are: 1. through FilterList which is a list of row key Filters, 2. using the SQL layer over HBase to join with two table, such as hive, phoenix etc. However, both solutions are inefficient. Both of them can’t utilize the range info to perform fast forwarding during scan which is quite time consuming. If the number of ranges are quite big (e.g. millions), join is a proper solution though it is slow. However, there are cases that user wants to specify a small number of ranges to scan (e.g. 1000 ranges). Both solutions can’t provide satisfactory performance in such case. We provide this filter (MultiRowRangeFilter) to support such use case (scan multiple row key ranges), which can construct the row key ranges from user specified list and perform fast-forwarding during scan. Thus, the scan will be quite efficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12820) Use table lock instead of MobZookeeper
Jingcheng Du created HBASE-12820: Summary: Use table lock instead of MobZookeeper Key: HBASE-12820 URL: https://issues.apache.org/jira/browse/HBASE-12820 Project: HBase Issue Type: Sub-task Affects Versions: hbase-11339 Reporter: Jingcheng Du Assignee: Jingcheng Du We had a lock to synchronize the major compaction, and sweep tool. Now we will have MR-less mob compaction in the HBase, and need the lock as well. And the table lock is a better choice. In this JIRA, clean the MobZookeeper code and use TableLockManager instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11295) Long running scan produces OutOfOrderScannerNextException
[ https://issues.apache.org/jira/browse/HBASE-11295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268908#comment-14268908 ] Lars Hofhansl commented on HBASE-11295: --- We actually ran into the same issue. Out filter was a timerange specified on the scan object, that filtered a large percentage of the data. The client would time because it did not hear back from the server, retries, and boom we get this exception. With the patch that we came up with above this no longer happens. [~anoop.hbase], could you have a quick look and confirm that this will not break anything new? Long running scan produces OutOfOrderScannerNextException - Key: HBASE-11295 URL: https://issues.apache.org/jira/browse/HBASE-11295 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.96.0 Reporter: Jeff Cunningham Attachments: OutOfOrderScannerNextException.tar.gz Attached Files: HRegionServer.java - instramented from 0.96.1.1-cdh5.0.0 HBaseLeaseTimeoutIT.java - reproducing JUnit 4 test WaitFilter.java - Scan filter (extends FilterBase) that overrides filterRowKey() to sleep during invocation SpliceFilter.proto - Protobuf defintiion for WaitFilter.java OutOfOrderScann_InstramentedServer.log - instramented server log Steps.txt - this note Set up: In HBaseLeaseTimeoutIT, create a scan, set the given filter (which sleeps in overridden filterRowKey() method) and set it on the scan, and scan the table. This is done in test client_0x0_server_15x10(). Here's what I'm seeing (see also attached log): A new request comes into server (ID 1940798815214593802 - RpcServer.handler=96) and a RegionScanner is created for it, cached by ID, immediately looked up again and cached RegionScannerHolder's nextCallSeq incremeted (now at 1). The RegionScan thread goes to sleep in WaitFilter#filterRowKey(). A short (variable) period later, another request comes into the server (ID 8946109289649235722 - RpcServer.handler=98) and the same series of events happen to this request. At this point both RegionScanner threads are sleeping in WaitFilter.filterRowKey(). After another period, the client retries another scan request which thinks its next_call_seq is 0. However, HRegionServer's cached RegionScannerHolder thinks the matching RegionScanner's nextCallSeq should be 1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11295) Long running scan produces OutOfOrderScannerNextException
[ https://issues.apache.org/jira/browse/HBASE-11295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-11295: -- Priority: Critical (was: Major) Long running scan produces OutOfOrderScannerNextException - Key: HBASE-11295 URL: https://issues.apache.org/jira/browse/HBASE-11295 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.96.0 Reporter: Jeff Cunningham Priority: Critical Attachments: OutOfOrderScannerNextException.tar.gz Attached Files: HRegionServer.java - instramented from 0.96.1.1-cdh5.0.0 HBaseLeaseTimeoutIT.java - reproducing JUnit 4 test WaitFilter.java - Scan filter (extends FilterBase) that overrides filterRowKey() to sleep during invocation SpliceFilter.proto - Protobuf defintiion for WaitFilter.java OutOfOrderScann_InstramentedServer.log - instramented server log Steps.txt - this note Set up: In HBaseLeaseTimeoutIT, create a scan, set the given filter (which sleeps in overridden filterRowKey() method) and set it on the scan, and scan the table. This is done in test client_0x0_server_15x10(). Here's what I'm seeing (see also attached log): A new request comes into server (ID 1940798815214593802 - RpcServer.handler=96) and a RegionScanner is created for it, cached by ID, immediately looked up again and cached RegionScannerHolder's nextCallSeq incremeted (now at 1). The RegionScan thread goes to sleep in WaitFilter#filterRowKey(). A short (variable) period later, another request comes into the server (ID 8946109289649235722 - RpcServer.handler=98) and the same series of events happen to this request. At this point both RegionScanner threads are sleeping in WaitFilter.filterRowKey(). After another period, the client retries another scan request which thinks its next_call_seq is 0. However, HRegionServer's cached RegionScannerHolder thinks the matching RegionScanner's nextCallSeq should be 1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11295) Long running scan produces OutOfOrderScannerNextException
[ https://issues.apache.org/jira/browse/HBASE-11295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268914#comment-14268914 ] Andrew Purtell commented on HBASE-11295: Set fix versions. l will put up patches for target branches tomorrow Long running scan produces OutOfOrderScannerNextException - Key: HBASE-11295 URL: https://issues.apache.org/jira/browse/HBASE-11295 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.96.0 Reporter: Jeff Cunningham Assignee: Andrew Purtell Priority: Critical Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: OutOfOrderScannerNextException.tar.gz Attached Files: HRegionServer.java - instramented from 0.96.1.1-cdh5.0.0 HBaseLeaseTimeoutIT.java - reproducing JUnit 4 test WaitFilter.java - Scan filter (extends FilterBase) that overrides filterRowKey() to sleep during invocation SpliceFilter.proto - Protobuf defintiion for WaitFilter.java OutOfOrderScann_InstramentedServer.log - instramented server log Steps.txt - this note Set up: In HBaseLeaseTimeoutIT, create a scan, set the given filter (which sleeps in overridden filterRowKey() method) and set it on the scan, and scan the table. This is done in test client_0x0_server_15x10(). Here's what I'm seeing (see also attached log): A new request comes into server (ID 1940798815214593802 - RpcServer.handler=96) and a RegionScanner is created for it, cached by ID, immediately looked up again and cached RegionScannerHolder's nextCallSeq incremeted (now at 1). The RegionScan thread goes to sleep in WaitFilter#filterRowKey(). A short (variable) period later, another request comes into the server (ID 8946109289649235722 - RpcServer.handler=98) and the same series of events happen to this request. At this point both RegionScanner threads are sleeping in WaitFilter.filterRowKey(). After another period, the client retries another scan request which thinks its next_call_seq is 0. However, HRegionServer's cached RegionScannerHolder thinks the matching RegionScanner's nextCallSeq should be 1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6486) Enhance load test to print throughput measurements
[ https://issues.apache.org/jira/browse/HBASE-6486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268433#comment-14268433 ] Cosmin Lehene commented on HBASE-6486: -- [~karthik.ranga], [~aurickq], [~apurtell] marked as improvement. Is this still the case/desired? Enhance load test to print throughput measurements -- Key: HBASE-6486 URL: https://issues.apache.org/jira/browse/HBASE-6486 Project: HBase Issue Type: Improvement Reporter: Karthik Ranganathan Assignee: Aurick Qiao Idea is to know how many MB/sec of throughput we are able to get by writing into HBase using a simple tool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-6578) Make HDFS block size configurable for HBase WAL
[ https://issues.apache.org/jira/browse/HBASE-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene updated HBASE-6578: - Issue Type: Improvement (was: Bug) Make HDFS block size configurable for HBase WAL --- Key: HBASE-6578 URL: https://issues.apache.org/jira/browse/HBASE-6578 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Karthik Ranganathan Assignee: Li Pi Right now, because sync-on-block-close is enabled, HLog causes the disk to stall out on large writes (esp when we cross block boundary). We currently use 256MB blocks. The idea is that if we use smaller block sizes, we should be able to spray the data across more disks (because of round robin scheduling) and this would cause more uniform disk usage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-6486) Enhance load test to print throughput measurements
[ https://issues.apache.org/jira/browse/HBASE-6486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene updated HBASE-6486: - Issue Type: Improvement (was: Bug) Enhance load test to print throughput measurements -- Key: HBASE-6486 URL: https://issues.apache.org/jira/browse/HBASE-6486 Project: HBase Issue Type: Improvement Reporter: Karthik Ranganathan Assignee: Aurick Qiao Idea is to know how many MB/sec of throughput we are able to get by writing into HBase using a simple tool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6578) Make HDFS block size configurable for HBase WAL
[ https://issues.apache.org/jira/browse/HBASE-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268441#comment-14268441 ] Cosmin Lehene commented on HBASE-6578: -- [~kramachandran], [~li], [~apurtell] is this still wanted? Refresh? Make HDFS block size configurable for HBase WAL --- Key: HBASE-6578 URL: https://issues.apache.org/jira/browse/HBASE-6578 Project: HBase Issue Type: Improvement Components: Performance, regionserver Reporter: Karthik Ranganathan Assignee: Li Pi Right now, because sync-on-block-close is enabled, HLog causes the disk to stall out on large writes (esp when we cross block boundary). We currently use 256MB blocks. The idea is that if we use smaller block sizes, we should be able to spray the data across more disks (because of round robin scheduling) and this would cause more uniform disk usage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-6578) Make HDFS block size configurable for HBase WAL
[ https://issues.apache.org/jira/browse/HBASE-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene updated HBASE-6578: - Component/s: Performance Make HDFS block size configurable for HBase WAL --- Key: HBASE-6578 URL: https://issues.apache.org/jira/browse/HBASE-6578 Project: HBase Issue Type: Improvement Components: Performance, regionserver Reporter: Karthik Ranganathan Assignee: Li Pi Right now, because sync-on-block-close is enabled, HLog causes the disk to stall out on large writes (esp when we cross block boundary). We currently use 256MB blocks. The idea is that if we use smaller block sizes, we should be able to spray the data across more disks (because of round robin scheduling) and this would cause more uniform disk usage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12819) ExportSnapshot doesn't close FileSystem instances
[ https://issues.apache.org/jira/browse/HBASE-12819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12819: --- Attachment: (was: 12819-v2.patch) ExportSnapshot doesn't close FileSystem instances - Key: HBASE-12819 URL: https://issues.apache.org/jira/browse/HBASE-12819 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu Attachments: 12819-v1.patch, 12819-v2.patch While troubleshooting an out of memory error exporting snapshot to s3a, I found that ExportSnapshot doesn't close FileSystem instances it uses. An implementation (s3a e.g.) may depend on close() call so that resources are released. ExportSnapshot should close the FileSystem instances upon exit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12819) ExportSnapshot doesn't close FileSystem instances
[ https://issues.apache.org/jira/browse/HBASE-12819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12819: --- Attachment: 12819-v2.patch ExportSnapshot doesn't close FileSystem instances - Key: HBASE-12819 URL: https://issues.apache.org/jira/browse/HBASE-12819 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu Attachments: 12819-v1.patch, 12819-v2.patch While troubleshooting an out of memory error exporting snapshot to s3a, I found that ExportSnapshot doesn't close FileSystem instances it uses. An implementation (s3a e.g.) may depend on close() call so that resources are released. ExportSnapshot should close the FileSystem instances upon exit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-6128) Use weights instead of block counts in DoubleBlockCache with CacheBuilder
[ https://issues.apache.org/jira/browse/HBASE-6128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene updated HBASE-6128: - Issue Type: Improvement (was: Bug) Use weights instead of block counts in DoubleBlockCache with CacheBuilder - Key: HBASE-6128 URL: https://issues.apache.org/jira/browse/HBASE-6128 Project: HBase Issue Type: Improvement Reporter: Jean-Daniel Cryans Priority: Minor Thanks to HBASE-5739 we can now specify a max weight instead of a maximum number of blocks in the DoubleBlockCache. This will give a more accurate control over its memory usage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6664) Filter compound comparators nor comparator subclasses are compared properly by test code
[ https://issues.apache.org/jira/browse/HBASE-6664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268468#comment-14268468 ] Cosmin Lehene commented on HBASE-6664: -- [~saint@gmail.com] can this be closed as invalid? Filter compound comparators nor comparator subclasses are compared properly by test code Key: HBASE-6664 URL: https://issues.apache.org/jira/browse/HBASE-6664 Project: HBase Issue Type: Bug Components: test Reporter: stack Labels: delete Gregory over in https://reviews.apache.org/r/6670/ explains the issue. See the comment here at https://reviews.apache.org/r/6670/diff/1/?file=142078#file142078line88 Its about when tests call areSerializedFieldsEqual to figure if objects are equal. When its comparators, and the comparator has been subclassed or is made of multiple sub comparators, then we'll not compare right. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-6664) Filter compound comparators nor comparator subclasses are compared properly by test code
[ https://issues.apache.org/jira/browse/HBASE-6664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene updated HBASE-6664: - Labels: delete (was: ) Filter compound comparators nor comparator subclasses are compared properly by test code Key: HBASE-6664 URL: https://issues.apache.org/jira/browse/HBASE-6664 Project: HBase Issue Type: Bug Components: test Reporter: stack Labels: delete Gregory over in https://reviews.apache.org/r/6670/ explains the issue. See the comment here at https://reviews.apache.org/r/6670/diff/1/?file=142078#file142078line88 Its about when tests call areSerializedFieldsEqual to figure if objects are equal. When its comparators, and the comparator has been subclassed or is made of multiple sub comparators, then we'll not compare right. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6128) Use weights instead of block counts in DoubleBlockCache with CacheBuilder
[ https://issues.apache.org/jira/browse/HBASE-6128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268471#comment-14268471 ] Cosmin Lehene commented on HBASE-6128: -- Marking as improvement. [~jdcryans], [~ndimiduk] is this still valid? Use weights instead of block counts in DoubleBlockCache with CacheBuilder - Key: HBASE-6128 URL: https://issues.apache.org/jira/browse/HBASE-6128 Project: HBase Issue Type: Improvement Reporter: Jean-Daniel Cryans Priority: Minor Thanks to HBASE-5739 we can now specify a max weight instead of a maximum number of blocks in the DoubleBlockCache. This will give a more accurate control over its memory usage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5401) PerformanceEvaluation generates 10x the number of expected mappers
[
https://issues.apache.org/jira/browse/HBASE-5401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268484#comment-14268484
]
Cosmin Lehene commented on HBASE-5401:
--
The outer loop is still there, but based on how it looks it doesn't seem it
should be removed :)
[~stack], [~ndimiduk] can you comment?
{code}
/**
* Per client, how many tasks will we run? We divide number of rows by this
number and have the
* client do the resulting count in a map task.
*/
static int TASKS_PER_CLIENT = 10;
{code}
{code}
for (int i = 0; i TASKS_PER_CLIENT; i++) {
for (int j = 0; j opts.numClientThreads; j++) {
TestOptions next = new TestOptions(opts);
next.startRow = (j * perClientRows) + (i * (perClientRows/10));
next.perClientRunRows = perClientRows / 10;
String s = MAPPER.writeValueAsString(next);
LOG.info(Client= + j + , maptask= + i + , input= + s);
int hash = h.hash(Bytes.toBytes(s));
m.put(hash, s);
}
}
{code}
PerformanceEvaluation generates 10x the number of expected mappers
--
Key: HBASE-5401
URL: https://issues.apache.org/jira/browse/HBASE-5401
Project: HBase
Issue Type: Bug
Components: test
Reporter: Oliver Meyn
With a command line like 'hbase org.apache.hadoop.hbase.PerformanceEvaluation
randomWrite 10' there are 100 mappers spawned, rather than the expected 10.
The culprit appears to be the outer loop in writeInputFile which sets up 10
splits for every asked-for client. I think the fix is just to remove that
outer loop.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-6705) Refactor duplicate CF parsing code in ThriftServerRunner
[ https://issues.apache.org/jira/browse/HBASE-6705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene updated HBASE-6705: - Issue Type: Improvement (was: Bug) Refactor duplicate CF parsing code in ThriftServerRunner - Key: HBASE-6705 URL: https://issues.apache.org/jira/browse/HBASE-6705 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Priority: Minor We have a lot of repetitive code conditional on famAndQf.length = 1. Refactoring that into a function. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6705) Refactor duplicate CF parsing code in ThriftServerRunner
[ https://issues.apache.org/jira/browse/HBASE-6705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268493#comment-14268493 ] Cosmin Lehene commented on HBASE-6705: -- [~apurtell], [~stack] there's probably more stuff that could be improved in the thrift server. Given that there's also thrift 2, are these improvements still on the table? Refactor duplicate CF parsing code in ThriftServerRunner - Key: HBASE-6705 URL: https://issues.apache.org/jira/browse/HBASE-6705 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Priority: Minor We have a lot of repetitive code conditional on famAndQf.length = 1. Refactoring that into a function. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-4201) Fix delayed RPC crash
[
https://issues.apache.org/jira/browse/HBASE-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268503#comment-14268503
]
Cosmin Lehene commented on HBASE-4201:
--
[~ddvlad], [~stack], [~apurtell] this logic seems to be in RpcServer.endDelay.
I wonder if it has other implications or we could just have an adapted patch?
This said, I'm a bit puzzled as the code seems to be called only from tests..
{code}
@Override
public synchronized void endDelay(Object result) throws IOException {
assert this.delayResponse;
assert this.delayReturnValue || result == null;
this.delayResponse = false;
delayedCalls.decrementAndGet();
if (this.delayReturnValue) {
this.setResponse(result, null, null, null);
}
this.responder.doRespond(this);
}
{code}
Fix delayed RPC crash
-
Key: HBASE-4201
URL: https://issues.apache.org/jira/browse/HBASE-4201
Project: HBase
Issue Type: Bug
Components: IPC/RPC
Reporter: Vlad Dogaru
Assignee: Vlad Dogaru
Priority: Minor
Attachments: 4201.txt, HBASE-4201-v1.patch
Delayed RPC crashes if return value is not delayed and endDelay is called
before the actual function returns a value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-6741) Detect duplicate assignment more accurately
[ https://issues.apache.org/jira/browse/HBASE-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene updated HBASE-6741: - Labels: delete (was: ) Detect duplicate assignment more accurately --- Key: HBASE-6741 URL: https://issues.apache.org/jira/browse/HBASE-6741 Project: HBase Issue Type: Bug Reporter: Amitanand Aiyer Priority: Minor Labels: delete Currently, the duplicate detection logic does not differentiate if the two duplicate updates are from the same rs or different RS. Have seen issues where a duplicate notification from the same RS, causes the assignment to be considered a duplicate assignment. ... and the region ends up being assigned to no one. Keep track of where the regions are opened, and make the duplicate detection more accurate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11533) AsciiDoctor POC
[ https://issues.apache.org/jira/browse/HBASE-11533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268539#comment-14268539 ] Misty Stanley-Jones commented on HBASE-11533: - Oh and [~saint@gmail.com], I feel like Asciidoc has enough big adopters now to be a safe bet. O'Reilly has embraced it, as well as Github itself, and the ReadTheDocs initiative. Here are some links to Asciidoc-rendered content: * http://git-scm.com/book/ (src https://github.com/progit/progit2) * http://readthedocs.org/ (src https://github.com/rtfd) * http://guide.couchdb.org/editions/1/en/index.html (src not public) Also check out this article: https://medium.com/@chacon/living-the-future-of-technical-writing-2f368bd0a272 AsciiDoctor POC --- Key: HBASE-11533 URL: https://issues.apache.org/jira/browse/HBASE-11533 Project: HBase Issue Type: Sub-task Components: build, documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Priority: Minor Fix For: 2.0.0 Convert or create a subset of documentation in Asciidoc and integrate Asciidoctor into the Maven build. http://asciidoctor.org/ Get some folks to play around with it afterward. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11533) AsciiDoctor POC
[ https://issues.apache.org/jira/browse/HBASE-11533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268537#comment-14268537 ] stack commented on HBASE-11533: --- That is a nice feature that the pages are rendered as markdown and mostly works. AsciiDoctor POC --- Key: HBASE-11533 URL: https://issues.apache.org/jira/browse/HBASE-11533 Project: HBase Issue Type: Sub-task Components: build, documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Priority: Minor Fix For: 2.0.0 Convert or create a subset of documentation in Asciidoc and integrate Asciidoctor into the Maven build. http://asciidoctor.org/ Get some folks to play around with it afterward. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6448) Add RecoverableZooKeeper#setACL which handles transient zookeeper exception
[ https://issues.apache.org/jira/browse/HBASE-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268431#comment-14268431 ] Cosmin Lehene commented on HBASE-6448: -- [~te...@apache.org] is this still desired? Is it worth a refresh? Add RecoverableZooKeeper#setACL which handles transient zookeeper exception --- Key: HBASE-6448 URL: https://issues.apache.org/jira/browse/HBASE-6448 Project: HBase Issue Type: Bug Reporter: Ted Yu In HBASE-6447, Stack added retry logic for calling ZooKeeper#setACL. The retry logic should be encapsulated in a new method: RecoverableZooKeeper#setACL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5401) PerformanceEvaluation generates 10x the number of expected mappers
[ https://issues.apache.org/jira/browse/HBASE-5401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268474#comment-14268474 ] Cosmin Lehene commented on HBASE-5401: -- [~ivarley], [~oliver_meyn] is this still an issue? PerformanceEvaluation generates 10x the number of expected mappers -- Key: HBASE-5401 URL: https://issues.apache.org/jira/browse/HBASE-5401 Project: HBase Issue Type: Bug Components: test Reporter: Oliver Meyn With a command line like 'hbase org.apache.hadoop.hbase.PerformanceEvaluation randomWrite 10' there are 100 mappers spawned, rather than the expected 10. The culprit appears to be the outer loop in writeInputFile which sets up 10 splits for every asked-for client. I think the fix is just to remove that outer loop. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12405) WAL accounting by Store
[ https://issues.apache.org/jira/browse/HBASE-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268473#comment-14268473 ] Esteban Gutierrez commented on HBASE-12405: --- StoreSequenceId is gone in Zookeeper.proto how RegionServers that haven't been upgraded will handle the new sequence id? WAL accounting by Store --- Key: HBASE-12405 URL: https://issues.apache.org/jira/browse/HBASE-12405 Project: HBase Issue Type: Improvement Components: wal Affects Versions: 2.0.0, 1.1.0 Reporter: zhangduo Assignee: zhangduo Fix For: 2.0.0, 1.1.0 Attachments: HBASE-12405.patch, HBASE-12405_1.patch, HBASE-12405_2.patch HBASE-10201 has made flush decisions per Store, but has not done enough work on HLog, so there are two problems: 1. We record minSeqId both in HRegion and FSHLog, which is a duplication. 2. There maybe holes in WAL accounting. For example, assume family A with sequence id 1 and 3, family B with seqId 2. If we flush family A, we can only record that WAL before sequence id 1 can be removed safely. If we do a replay at this point, sequence id 3 will also be replayed which is unnecessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12405) WAL accounting by Store
[ https://issues.apache.org/jira/browse/HBASE-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268478#comment-14268478 ] Esteban Gutierrez commented on HBASE-12405: --- Ok, probably it won't make any difference since the message should be the same but now is handled by ClusterStatus. WAL accounting by Store --- Key: HBASE-12405 URL: https://issues.apache.org/jira/browse/HBASE-12405 Project: HBase Issue Type: Improvement Components: wal Affects Versions: 2.0.0, 1.1.0 Reporter: zhangduo Assignee: zhangduo Fix For: 2.0.0, 1.1.0 Attachments: HBASE-12405.patch, HBASE-12405_1.patch, HBASE-12405_2.patch HBASE-10201 has made flush decisions per Store, but has not done enough work on HLog, so there are two problems: 1. We record minSeqId both in HRegion and FSHLog, which is a duplication. 2. There maybe holes in WAL accounting. For example, assume family A with sequence id 1 and 3, family B with seqId 2. If we flush family A, we can only record that WAL before sequence id 1 can be removed safely. If we do a replay at this point, sequence id 3 will also be replayed which is unnecessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12728) buffered writes substantially less useful after removal of HTablePool
[
https://issues.apache.org/jira/browse/HBASE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268488#comment-14268488
]
stack commented on HBASE-12728:
---
I like the BulkWriter/BulkMutator suggestion. Seems cleanest suggestion so far
(You'd need to pass tablename when getting BulkWriter since we need one per
Table I believer -- see how [~ndimiduk] does it in his patch).
On the [~ndimiduk] direction (thanks for working up a patch boss), a few
comments:
+ In the multithreaded case, which thread calls the BufferedTable#flush? If all
do, no buffering is going on. Is flush then called after a 'big' put? How's
that going to work when many threads? Better if flush is done internally at
size/time/shutdown thresholds. It doesn't seem like a function that belongs in
the BufferedTable instance.
+ Ditto ExceptionListener in BufferedTable. Its awkward, right? If I register
a listener on BufferedTable, internally I'll need to pass this interest to the
exception handler and then remove interest when the BufferedTable goes away.
+ If you buy the above two points, then need for a special BufferedTable
Interface goes awaybut then how to listen on exceptions and how to flush?
(See the last [~sduskis] suggestion?)
+ I like AsyncMutator being an internal 'detail'.
+ We need a BufferedConnection? Can't we just add the few methods to
Connection? It is new in 1.0 so we'd be breaking no one (though there is the
issue of being able to specify an executor, at least optionally, which I see
you doing in BufferedConnection constructor in the factory).
+ The flush on the BufferedConnection is a bit odd, yeah. Flushes all tables?
Or there'd be an override that allowed passing which table to flush?
buffered writes substantially less useful after removal of HTablePool
-
Key: HBASE-12728
URL: https://issues.apache.org/jira/browse/HBASE-12728
Project: HBase
Issue Type: Bug
Components: hbase
Affects Versions: 0.98.0
Reporter: Aaron Beppu
Assignee: Solomon Duskis
Priority: Blocker
Fix For: 1.0.0, 2.0.0, 1.1.0
Attachments: 12728.connection-owns-buffers.example.branch-1.0.patch
In previous versions of HBase, when use of HTablePool was encouraged, HTable
instances were long-lived in that pool, and for that reason, if autoFlush was
set to false, the table instance could accumulate a full buffer of writes
before a flush was triggered. Writes from the client to the cluster could
then be substantially larger and less frequent than without buffering.
However, when HTablePool was deprecated, the primary justification seems to
have been that creating HTable instances is cheap, so long as the connection
and executor service being passed to it are pre-provided. A use pattern was
encouraged where users should create a new HTable instance for every
operation, using an existing connection and executor service, and then close
the table. In this pattern, buffered writes are substantially less useful;
writes are as small and as frequent as they would have been with
autoflush=true, except the synchronous write is moved from the operation
itself to the table close call which immediately follows.
More concretely :
```
// Given these two helpers ...
private HTableInterface getAutoFlushTable(String tableName) throws
IOException {
// (autoflush is true by default)
return storedConnection.getTable(tableName, executorService);
}
private HTableInterface getBufferedTable(String tableName) throws IOException
{
HTableInterface table = getAutoFlushTable(tableName);
table.setAutoFlush(false);
return table;
}
// it's my contention that these two methods would behave almost identically,
// except the first will hit a synchronous flush during the put call,
and the second will
// flush during the (hidden) close call on table.
private void writeAutoFlushed(Put somePut) throws IOException {
try (HTableInterface table = getAutoFlushTable(tableName)) {
table.put(somePut); // will do synchronous flush
}
}
private void writeBuffered(Put somePut) throws IOException {
try (HTableInterface table = getBufferedTable(tableName)) {
table.put(somePut);
} // auto-close will trigger synchronous flush
}
```
For buffered writes to actually provide a performance benefit to users, one
of two things must happen:
- The writeBuffer itself shouldn't live, flush and die with the lifecycle of
it's HTableInstance. If the writeBuffer were managed elsewhere and had a long
lifespan, this could cease to be an issue. However, if the same writeBuffer
is appended to by multiple tables, then some additional concurrency control
will be needed around it.
- Alternatively, there should be some pattern for
[jira] [Updated] (HBASE-12819) ExportSnapshot doesn't close FileSystem instances
[ https://issues.apache.org/jira/browse/HBASE-12819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12819: --- Attachment: 12819-v3.patch ExportSnapshot doesn't close FileSystem instances - Key: HBASE-12819 URL: https://issues.apache.org/jira/browse/HBASE-12819 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu Attachments: 12819-v1.patch, 12819-v2.patch, 12819-v3.patch While troubleshooting an out of memory error exporting snapshot to s3a, I found that ExportSnapshot doesn't close FileSystem instances it uses. An implementation (s3a e.g.) may depend on close() call so that resources are released. ExportSnapshot should close the FileSystem instances upon exit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6741) Detect duplicate assignment more accurately
[ https://issues.apache.org/jira/browse/HBASE-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268509#comment-14268509 ] Cosmin Lehene commented on HBASE-6741: -- [~amitanand] is this still an issue? If so can you provide a more elaborate description (methods involved, example, etc.)? [~apurtell] ping Detect duplicate assignment more accurately --- Key: HBASE-6741 URL: https://issues.apache.org/jira/browse/HBASE-6741 Project: HBase Issue Type: Bug Reporter: Amitanand Aiyer Priority: Minor Labels: delete Currently, the duplicate detection logic does not differentiate if the two duplicate updates are from the same rs or different RS. Have seen issues where a duplicate notification from the same RS, causes the assignment to be considered a duplicate assignment. ... and the region ends up being assigned to no one. Keep track of where the regions are opened, and make the duplicate detection more accurate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-4201) Fix delayed RPC crash
[
https://issues.apache.org/jira/browse/HBASE-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268510#comment-14268510
]
Hadoop QA commented on HBASE-4201:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12543410/4201.txt
against master branch at commit 7dce0d5c4549a1d5986d3bc82734924219d8bb25.
ATTACHMENT ID: 12543410
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12351//console
This message is automatically generated.
Fix delayed RPC crash
-
Key: HBASE-4201
URL: https://issues.apache.org/jira/browse/HBASE-4201
Project: HBase
Issue Type: Bug
Components: IPC/RPC
Reporter: Vlad Dogaru
Assignee: Vlad Dogaru
Priority: Minor
Attachments: 4201.txt, HBASE-4201-v1.patch
Delayed RPC crashes if return value is not delayed and endDelay is called
before the actual function returns a value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11533) AsciiDoctor POC
[ https://issues.apache.org/jira/browse/HBASE-11533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268519#comment-14268519 ] Misty Stanley-Jones commented on HBASE-11533: - Check out how the Asciidoc renders on Github without you even having to build the book. Click any file to see its chapter: https://github.com/apache/hbase/tree/HBASE-11533/src/main/asciidoc The only limitation there is that Github doesn't render the includes, so the book.adoc looks rather boring for instance. Also currently images don't show up, but I think I have an idea of why. AsciiDoctor POC --- Key: HBASE-11533 URL: https://issues.apache.org/jira/browse/HBASE-11533 Project: HBase Issue Type: Sub-task Components: build, documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Priority: Minor Fix For: 2.0.0 Convert or create a subset of documentation in Asciidoc and integrate Asciidoctor into the Maven build. http://asciidoctor.org/ Get some folks to play around with it afterward. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-4201) Fix delayed RPC crash
[ https://issues.apache.org/jira/browse/HBASE-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4201: - Resolution: Invalid Status: Resolved (was: Patch Available) Resovling as no longer valid. We don't do delayed anymore. RPC is different now too. Fix delayed RPC crash - Key: HBASE-4201 URL: https://issues.apache.org/jira/browse/HBASE-4201 Project: HBase Issue Type: Bug Components: IPC/RPC Reporter: Vlad Dogaru Assignee: Vlad Dogaru Priority: Minor Attachments: 4201.txt, HBASE-4201-v1.patch Delayed RPC crashes if return value is not delayed and endDelay is called before the actual function returns a value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12748) RegionCoprocessorHost.execOperation creates too many iterator objects
[ https://issues.apache.org/jira/browse/HBASE-12748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-12748: -- Status: Open (was: Patch Available) RegionCoprocessorHost.execOperation creates too many iterator objects - Key: HBASE-12748 URL: https://issues.apache.org/jira/browse/HBASE-12748 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.9, 0.94.25 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 1.0.0, 2.0.0, 0.98.10, 0.94.27 Attachments: HBase-12748.patch This is typical pattern of enhanced for ... loop usage in a hot code path. For every HBase operation it instantiates iterator for coprocessor list twice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12748) RegionCoprocessorHost.execOperation creates too many iterator objects
[ https://issues.apache.org/jira/browse/HBASE-12748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-12748: -- Attachment: (was: HBase-12748.patch) RegionCoprocessorHost.execOperation creates too many iterator objects - Key: HBASE-12748 URL: https://issues.apache.org/jira/browse/HBASE-12748 Project: HBase Issue Type: Sub-task Affects Versions: 0.94.25, 0.98.9 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 1.0.0, 2.0.0, 0.98.10, 0.94.27 This is typical pattern of enhanced for ... loop usage in a hot code path. For every HBase operation it instantiates iterator for coprocessor list twice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12748) RegionCoprocessorHost.execOperation creates too many iterator objects
[ https://issues.apache.org/jira/browse/HBASE-12748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-12748: -- Attachment: HBase-12748.patch.2 The patch. RegionCoprocessorHost.execOperation creates too many iterator objects - Key: HBASE-12748 URL: https://issues.apache.org/jira/browse/HBASE-12748 Project: HBase Issue Type: Sub-task Affects Versions: 0.94.25, 0.98.9 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 1.0.0, 2.0.0, 0.98.10, 0.94.27 Attachments: HBase-12748.patch.2 This is typical pattern of enhanced for ... loop usage in a hot code path. For every HBase operation it instantiates iterator for coprocessor list twice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12748) RegionCoprocessorHost.execOperation creates too many iterator objects
[ https://issues.apache.org/jira/browse/HBASE-12748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-12748: -- Status: Patch Available (was: Open) RegionCoprocessorHost.execOperation creates too many iterator objects - Key: HBASE-12748 URL: https://issues.apache.org/jira/browse/HBASE-12748 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.9, 0.94.25 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 1.0.0, 2.0.0, 0.98.10, 0.94.27 Attachments: HBase-12748.patch.2 This is typical pattern of enhanced for ... loop usage in a hot code path. For every HBase operation it instantiates iterator for coprocessor list twice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
