[jira] [Commented] (HBASE-9410) Concurrent coprocessor endpoint executions slow down exponentially
[ https://issues.apache.org/jira/browse/HBASE-9410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13761212#comment-13761212 ] Kirubakaran Pakkirisamy commented on HBASE-9410: I do not see this issue in 0.95.2 Concurrent coprocessor endpoint executions slow down exponentially -- Key: HBASE-9410 URL: https://issues.apache.org/jira/browse/HBASE-9410 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 0.94.11 Environment: Amazon ec2 Reporter: Kirubakaran Pakkirisamy Attachments: jstack1.log, jstack2.log, jstack3.log, jstack.log, SearchEndpoint.java, Search.java, SearchProtocol.java Multiple concurrent executions of coprocessor endpoints slow down drastically. It is compounded further when there are more Htable connection setups happening. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9458) Intermittent TestFlushSnapshotFromClient#testTakeSnapshotAfterMerge failure
[
https://issues.apache.org/jira/browse/HBASE-9458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13761456#comment-13761456
]
Matteo Bertozzi commented on HBASE-9458:
that is by design.
The online snapshot operation is not guaranteed to succeed every time.
So in case like this where the machine is slow and the compaction of a split
happen during the snapshot you may get a failure.
Intermittent TestFlushSnapshotFromClient#testTakeSnapshotAfterMerge failure
---
Key: HBASE-9458
URL: https://issues.apache.org/jira/browse/HBASE-9458
Project: HBase
Issue Type: Test
Reporter: Ted Yu
From
https://builds.apache.org/job/HBase-0.96/20/testReport/org.apache.hadoop.hbase.snapshot/TestFlushSnapshotFromClient/testTakeSnapshotAfterMerge/
:
{code}
org.apache.hadoop.hbase.snapshot.HBaseSnapshotException:
org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot {
ss=snapshotAfterMerge table=test type=FLUSH } had an error. Procedure
snapshotAfterMerge { waiting=[] done=[] }
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:210)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:221)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:125)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:96)
at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3120)
at
org.apache.hadoop.hbase.client.HBaseAdmin.snapshot(HBaseAdmin.java:2672)
at
org.apache.hadoop.hbase.client.HBaseAdmin.snapshot(HBaseAdmin.java:2605)
at
org.apache.hadoop.hbase.client.HBaseAdmin.snapshot(HBaseAdmin.java:2612)
at
org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient.testTakeSnapshotAfterMerge(TestFlushSnapshotFromClient.java:336)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException:
org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot {
ss=snapshotAfterMerge table=test type=FLUSH } had an error. Procedure
snapshotAfterMerge { waiting=[] done=[] }
at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:365)
at
org.apache.hadoop.hbase.master.HMaster.isSnapshotDone(HMaster.java:2947)
at
org.apache.hadoop.hbase.protobuf.generated.MasterAdminProtos$MasterAdminService$2.callBlockingMethod(MasterAdminProtos.java:32890)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
Caused by: org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException via
Failed taking snapshot { ss=snapshotAfterMerge table=test type=FLUSH } due to
exception:Missing parent hfile for:
30b951996ef34885a2f5d64e4acb2467.6d1ed72bf95759cb606e8a6efdc6908e:org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException:
Missing parent hfile for:
30b951996ef34885a2f5d64e4acb2467.6d1ed72bf95759cb606e8a6efdc6908e
at
org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:85)
at
org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:318)
at
[jira] [Commented] (HBASE-9328) Table web UI is corrupted sometime
[
https://issues.apache.org/jira/browse/HBASE-9328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13761458#comment-13761458
]
Jimmy Xiang commented on HBASE-9328:
The compilation error seems not related. However, why do we miss java
executable?
{noformat}
[PreCommit-HBASE-Build] $ /bin/bash /tmp/hudson2093491801297215642.sh
asf002.sp2.ygridcore.net
Linux asf002.sp2.ygridcore.net 2.6.32-33-server #71-Ubuntu SMP Wed Jul 20
17:42:25 UTC 2011 x86_64 GNU/Linux
/tmp/hudson2093491801297215642.sh: line 5: java: command not found
/home/hudson/tools/java/latest1.6
/tmp/hudson2093491801297215642.sh: line 7:
/home/hudson/tools/java/latest1.6/bin/java: No such file or directory
{noformat}
Table web UI is corrupted sometime
--
Key: HBASE-9328
URL: https://issues.apache.org/jira/browse/HBASE-9328
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0, 0.95.2, 0.94.11
Reporter: Jimmy Xiang
Assignee: Jean-Marc Spaggiari
Labels: web-ui
Attachments: HBASE-9328-v0-trunk.patch, HBASE-9328-v1-trunk.patch,
HBASE-9328-v2-0.94.patch, HBASE-9328-v2-trunk.patch,
HBASE-9328-v3-trunk.patch, HBASE-9328-v4-trunk.patch, table.png
The web UI page source is like below:
{noformat}
h2Table Attributes/h2
table class=table table-striped
tr
thAttribute Name/th
thValue/th
thDescription/th
/tr
tr
tdEnabled/td
tdtrue/td
tdIs the table enabled/td
/tr
tr
tdCompaction/td
td
phr//p
{noformat}
No sure if it is a HBase issue, or a network/browser issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9458) Intermittent TestFlushSnapshotFromClient#testTakeSnapshotAfterMerge failure
[
https://issues.apache.org/jira/browse/HBASE-9458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13761468#comment-13761468
]
Ted Yu commented on HBASE-9458:
---
How about catching CorruptedSnapshotException from admin.snapshot() call so
that the test doesn't fail ?
Intermittent TestFlushSnapshotFromClient#testTakeSnapshotAfterMerge failure
---
Key: HBASE-9458
URL: https://issues.apache.org/jira/browse/HBASE-9458
Project: HBase
Issue Type: Test
Reporter: Ted Yu
From
https://builds.apache.org/job/HBase-0.96/20/testReport/org.apache.hadoop.hbase.snapshot/TestFlushSnapshotFromClient/testTakeSnapshotAfterMerge/
:
{code}
org.apache.hadoop.hbase.snapshot.HBaseSnapshotException:
org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot {
ss=snapshotAfterMerge table=test type=FLUSH } had an error. Procedure
snapshotAfterMerge { waiting=[] done=[] }
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:210)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:221)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:125)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:96)
at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3120)
at
org.apache.hadoop.hbase.client.HBaseAdmin.snapshot(HBaseAdmin.java:2672)
at
org.apache.hadoop.hbase.client.HBaseAdmin.snapshot(HBaseAdmin.java:2605)
at
org.apache.hadoop.hbase.client.HBaseAdmin.snapshot(HBaseAdmin.java:2612)
at
org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient.testTakeSnapshotAfterMerge(TestFlushSnapshotFromClient.java:336)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException:
org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot {
ss=snapshotAfterMerge table=test type=FLUSH } had an error. Procedure
snapshotAfterMerge { waiting=[] done=[] }
at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:365)
at
org.apache.hadoop.hbase.master.HMaster.isSnapshotDone(HMaster.java:2947)
at
org.apache.hadoop.hbase.protobuf.generated.MasterAdminProtos$MasterAdminService$2.callBlockingMethod(MasterAdminProtos.java:32890)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
Caused by: org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException via
Failed taking snapshot { ss=snapshotAfterMerge table=test type=FLUSH } due to
exception:Missing parent hfile for:
30b951996ef34885a2f5d64e4acb2467.6d1ed72bf95759cb606e8a6efdc6908e:org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException:
Missing parent hfile for:
30b951996ef34885a2f5d64e4acb2467.6d1ed72bf95759cb606e8a6efdc6908e
at
org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:85)
at
org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:318)
at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:355)
... 4 more
Caused by:
[jira] [Commented] (HBASE-9457) Master could fail start if region server with system table is down
[ https://issues.apache.org/jira/browse/HBASE-9457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13761477#comment-13761477 ] Jimmy Xiang commented on HBASE-9457: [~toffer], [~saint@gmail.com], is there any special reason we need to assign system tables before user tables? In case user tables depend on some system tables in the future? I don't see a system table server shutdown handler, so I assume user tables don't depend on system table for normal operations. At least user region assignment doesn't need system table to be available? If so, can we assign system tables together with user tables? For example, the access control table is assigned just like another user table. Master could fail start if region server with system table is down -- Key: HBASE-9457 URL: https://issues.apache.org/jira/browse/HBASE-9457 Project: HBase Issue Type: Bug Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Critical In the region server holding the system table is killed while master is starting, master will hang there waiting for system table to be assigned which won't happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8572) Enhance delete_snapshot.rb to call snapshot deletion API with regex
[ https://issues.apache.org/jira/browse/HBASE-8572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8572: -- Fix Version/s: 0.98.0 Enhance delete_snapshot.rb to call snapshot deletion API with regex --- Key: HBASE-8572 URL: https://issues.apache.org/jira/browse/HBASE-8572 Project: HBase Issue Type: Improvement Reporter: Ted Yu Fix For: 0.98.0 HBASE-8461 added the API to HBaseAdmin which allows user to specify regular expression for deleting snapshots. This JIRA would allow delete_snapshot.rb to utilize this functionality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9462) HBaseAdmin#isTableEnabled() should return false for non-existent table
Ted Yu created HBASE-9462: - Summary: HBaseAdmin#isTableEnabled() should return false for non-existent table Key: HBASE-9462 URL: https://issues.apache.org/jira/browse/HBASE-9462 Project: HBase Issue Type: Bug Affects Versions: 0.95.2 Reporter: Ted Yu Fix For: 0.96.0 HBaseAdmin#isTableEnabled() returns true for a table which doesn't exist. We should check table existence. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9440) Pass blocks of KVs from HFile scanner to the StoreFileScanner and up
[ https://issues.apache.org/jira/browse/HBASE-9440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13761539#comment-13761539 ] Matt Corgan commented on HBASE-9440: In cases where the client wants the majority of cells in the block, you'd basically be sending the same amount of data over the wire, but it could save the regionserver all the work of decompressing, decoding, iterating, reencoding, etc. All that fine-grained cell handling is is most of the work that hbase is doing, and sometimes it's just doing it to rebuild a near replica of what was on disk anyway. Pass blocks of KVs from HFile scanner to the StoreFileScanner and up Key: HBASE-9440 URL: https://issues.apache.org/jira/browse/HBASE-9440 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Currently we read KVs from an HFileScanner one-by-one and pass them up the scanner/heap tree. Many time the ranges of KVs retrieved from StoreFileScanner (by StoreScanners) and HFileScanner (by StoreFileScanner) will be non-overlapping. If chunks of KVs do not overlap we can sort entire chunks just by comparing the start/end key of the chunk. Only if chunks are overlapping do we need to sort KV by KV as we do now. I have no patch, but I wanted to float this idea. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8755) A new write thread model for HLog to improve the overall HBase write throughput
[ https://issues.apache.org/jira/browse/HBASE-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13761563#comment-13761563 ] Feng Honghua commented on HBASE-8755: - Thanks [~stack]. Looking forward to your test result on hdfs A new write thread model for HLog to improve the overall HBase write throughput --- Key: HBASE-8755 URL: https://issues.apache.org/jira/browse/HBASE-8755 Project: HBase Issue Type: Improvement Components: Performance, wal Reporter: Feng Honghua Assignee: stack Priority: Critical Fix For: 0.96.1 Attachments: 8755trunkV2.txt, HBASE-8755-0.94-V0.patch, HBASE-8755-0.94-V1.patch, HBASE-8755-trunk-V0.patch, HBASE-8755-trunk-V1.patch In current write model, each write handler thread (executing put()) will individually go through a full 'append (hlog local buffer) = HLog writer append (write to hdfs) = HLog writer sync (sync hdfs)' cycle for each write, which incurs heavy race condition on updateLock and flushLock. The only optimization where checking if current syncTillHere txid in expectation for other thread help write/sync its own txid to hdfs and omitting the write/sync actually help much less than expectation. Three of my colleagues(Ye Hangjun / Wu Zesheng / Zhang Peng) at Xiaomi proposed a new write thread model for writing hdfs sequence file and the prototype implementation shows a 4X improvement for throughput (from 17000 to 7+). I apply this new write thread model in HLog and the performance test in our test cluster shows about 3X throughput improvement (from 12150 to 31520 for 1 RS, from 22000 to 7 for 5 RS), the 1 RS write throughput (1K row-size) even beats the one of BigTable (Precolator published in 2011 says Bigtable's write throughput then is 31002). I can provide the detailed performance test results if anyone is interested. The change for new write thread model is as below: 1 All put handler threads append the edits to HLog's local pending buffer; (it notifies AsyncWriter thread that there is new edits in local buffer) 2 All put handler threads wait in HLog.syncer() function for underlying threads to finish the sync that contains its txid; 3 An single AsyncWriter thread is responsible for retrieve all the buffered edits in HLog's local pending buffer and write to the hdfs (hlog.writer.append); (it notifies AsyncFlusher thread that there is new writes to hdfs that needs a sync) 4 An single AsyncFlusher thread is responsible for issuing a sync to hdfs to persist the writes by AsyncWriter; (it notifies the AsyncNotifier thread that sync watermark increases) 5 An single AsyncNotifier thread is responsible for notifying all pending put handler threads which are waiting in the HLog.syncer() function 6 No LogSyncer thread any more (since there is always AsyncWriter/AsyncFlusher threads do the same job it does) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-8930) Filter evaluates KVs outside requested columns
[
https://issues.apache.org/jira/browse/HBASE-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl reopened HBASE-8930:
--
It turns out that this is likely cause for frequently failing tests in jenkins.
I will roll this back for now.
[~apurtell], [~vasu.mariy...@gmail.com]
Filter evaluates KVs outside requested columns
--
Key: HBASE-8930
URL: https://issues.apache.org/jira/browse/HBASE-8930
Project: HBase
Issue Type: Bug
Components: Filters
Affects Versions: 0.94.7
Reporter: Federico Gaule
Assignee: Vasu Mariyala
Priority: Critical
Labels: filters, hbase, keyvalue
Fix For: 0.98.0, 0.94.12, 0.96.1
Attachments: 0.94-HBASE-8930.patch, 0.94-HBASE-8930-rev1.patch,
0.95-HBASE-8930.patch, 0.95-HBASE-8930-rev1.patch,
0.96-HBASE-8930-rev2.patch, 8930-0.94.txt, HBASE-8930.patch,
HBASE-8930-rev1.patch, HBASE-8930-rev2.patch, HBASE-8930-rev3.patch,
HBASE-8930-rev4.patch, HBASE-8930-rev5.patch
1- Fill row with some columns
2- Get row with some columns less than universe - Use filter to print kvs
3- Filter prints not requested columns
Filter (AllwaysNextColFilter) always return ReturnCode.INCLUDE_AND_NEXT_COL
and prints KV's qualifier
SUFFIX_0 = 0
SUFFIX_1 = 1
SUFFIX_4 = 4
SUFFIX_6 = 6
P= Persisted
R= Requested
E= Evaluated
X= Returned
| 5580 | 5581 | 5584 | 5586 | 5590 | 5591 | 5594 | 5596 | 5600 | 5601 | 5604
| 5606 |...
| | P | P | | | P | P | | | P | P
| |...
| | R | R | R | | R | R | R | | |
| |...
| | E | E | | | E | E | | |
{color:red}E{color} | | |...
| | X | X | | | X | X | | | |
| |
{code:title=ExtraColumnTest.java|borderStyle=solid}
@Test
public void testFilter() throws Exception {
Configuration config = HBaseConfiguration.create();
config.set(hbase.zookeeper.quorum, myZK);
HTable hTable = new HTable(config, testTable);
byte[] cf = Bytes.toBytes(cf);
byte[] row = Bytes.toBytes(row);
byte[] col1 = new QualifierConverter().objectToByteArray(new
Qualifier((short) 558, (byte) SUFFIX_1));
byte[] col2 = new QualifierConverter().objectToByteArray(new
Qualifier((short) 559, (byte) SUFFIX_1));
byte[] col3 = new QualifierConverter().objectToByteArray(new
Qualifier((short) 560, (byte) SUFFIX_1));
byte[] col4 = new QualifierConverter().objectToByteArray(new
Qualifier((short) 561, (byte) SUFFIX_1));
byte[] col5 = new QualifierConverter().objectToByteArray(new
Qualifier((short) 562, (byte) SUFFIX_1));
byte[] col6 = new QualifierConverter().objectToByteArray(new
Qualifier((short) 563, (byte) SUFFIX_1));
byte[] col1g = new QualifierConverter().objectToByteArray(new
Qualifier((short) 558, (byte) SUFFIX_6));
byte[] col2g = new QualifierConverter().objectToByteArray(new
Qualifier((short) 559, (byte) SUFFIX_6));
byte[] col1v = new QualifierConverter().objectToByteArray(new
Qualifier((short) 558, (byte) SUFFIX_4));
byte[] col2v = new QualifierConverter().objectToByteArray(new
Qualifier((short) 559, (byte) SUFFIX_4));
byte[] col3v = new QualifierConverter().objectToByteArray(new
Qualifier((short) 560, (byte) SUFFIX_4));
byte[] col4v = new QualifierConverter().objectToByteArray(new
Qualifier((short) 561, (byte) SUFFIX_4));
byte[] col5v = new QualifierConverter().objectToByteArray(new
Qualifier((short) 562, (byte) SUFFIX_4));
byte[] col6v = new QualifierConverter().objectToByteArray(new
Qualifier((short) 563, (byte) SUFFIX_4));
// === INSERTION =//
Put put = new Put(row);
put.add(cf, col1, Bytes.toBytes((short) 1));
put.add(cf, col2, Bytes.toBytes((short) 1));
put.add(cf, col3, Bytes.toBytes((short) 3));
put.add(cf, col4, Bytes.toBytes((short) 3));
put.add(cf, col5, Bytes.toBytes((short) 3));
put.add(cf, col6, Bytes.toBytes((short) 3));
hTable.put(put);
put = new Put(row);
put.add(cf, col1v, Bytes.toBytes((short) 10));
put.add(cf, col2v, Bytes.toBytes((short) 10));
put.add(cf, col3v, Bytes.toBytes((short) 10));
put.add(cf, col4v, Bytes.toBytes((short) 10));
put.add(cf, col5v, Bytes.toBytes((short) 10));
put.add(cf, col6v, Bytes.toBytes((short) 10));
hTable.put(put);
hTable.flushCommits();
//==READING=//
Filter allwaysNextColFilter = new AllwaysNextColFilter();
Get get = new
[jira] [Commented] (HBASE-8930) Filter evaluates KVs outside requested columns
[
https://issues.apache.org/jira/browse/HBASE-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13761616#comment-13761616
]
Lars Hofhansl commented on HBASE-8930:
--
For the record, the errors we've seen:
{code}
Stacktrace
java.lang.AssertionError: Column family prefix used twice:
cf.cf.bt.Data.fsReadnumops
at
org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.validateMetricChanges(SchemaMetrics.java:885)
at
org.apache.hadoop.hbase.io.hfile.TestScannerSelectionUsingTTL.testScannerSelection(TestScannerSelectionUsingTTL.java:159)
{code}
{code}
Error Message
Column family prefix used twice: cf.cf.bt.Data.fsReadnumops
Stacktrace
java.lang.AssertionError: Column family prefix used twice:
cf.cf.bt.Data.fsReadnumops
at
org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.validateMetricChanges(SchemaMetrics.java:885)
at
org.apache.hadoop.hbase.io.hfile.TestScannerSelectionUsingKeyRange.testScannerSelection(TestScannerSelectionUsingKeyRange.java:125)
{code}
I do not immediately see how this change could have caused this, though.
Filter evaluates KVs outside requested columns
--
Key: HBASE-8930
URL: https://issues.apache.org/jira/browse/HBASE-8930
Project: HBase
Issue Type: Bug
Components: Filters
Affects Versions: 0.94.7
Reporter: Federico Gaule
Assignee: Vasu Mariyala
Priority: Critical
Labels: filters, hbase, keyvalue
Fix For: 0.98.0, 0.94.12, 0.96.1
Attachments: 0.94-HBASE-8930.patch, 0.94-HBASE-8930-rev1.patch,
0.95-HBASE-8930.patch, 0.95-HBASE-8930-rev1.patch,
0.96-HBASE-8930-rev2.patch, 8930-0.94.txt, HBASE-8930.patch,
HBASE-8930-rev1.patch, HBASE-8930-rev2.patch, HBASE-8930-rev3.patch,
HBASE-8930-rev4.patch, HBASE-8930-rev5.patch
1- Fill row with some columns
2- Get row with some columns less than universe - Use filter to print kvs
3- Filter prints not requested columns
Filter (AllwaysNextColFilter) always return ReturnCode.INCLUDE_AND_NEXT_COL
and prints KV's qualifier
SUFFIX_0 = 0
SUFFIX_1 = 1
SUFFIX_4 = 4
SUFFIX_6 = 6
P= Persisted
R= Requested
E= Evaluated
X= Returned
| 5580 | 5581 | 5584 | 5586 | 5590 | 5591 | 5594 | 5596 | 5600 | 5601 | 5604
| 5606 |...
| | P | P | | | P | P | | | P | P
| |...
| | R | R | R | | R | R | R | | |
| |...
| | E | E | | | E | E | | |
{color:red}E{color} | | |...
| | X | X | | | X | X | | | |
| |
{code:title=ExtraColumnTest.java|borderStyle=solid}
@Test
public void testFilter() throws Exception {
Configuration config = HBaseConfiguration.create();
config.set(hbase.zookeeper.quorum, myZK);
HTable hTable = new HTable(config, testTable);
byte[] cf = Bytes.toBytes(cf);
byte[] row = Bytes.toBytes(row);
byte[] col1 = new QualifierConverter().objectToByteArray(new
Qualifier((short) 558, (byte) SUFFIX_1));
byte[] col2 = new QualifierConverter().objectToByteArray(new
Qualifier((short) 559, (byte) SUFFIX_1));
byte[] col3 = new QualifierConverter().objectToByteArray(new
Qualifier((short) 560, (byte) SUFFIX_1));
byte[] col4 = new QualifierConverter().objectToByteArray(new
Qualifier((short) 561, (byte) SUFFIX_1));
byte[] col5 = new QualifierConverter().objectToByteArray(new
Qualifier((short) 562, (byte) SUFFIX_1));
byte[] col6 = new QualifierConverter().objectToByteArray(new
Qualifier((short) 563, (byte) SUFFIX_1));
byte[] col1g = new QualifierConverter().objectToByteArray(new
Qualifier((short) 558, (byte) SUFFIX_6));
byte[] col2g = new QualifierConverter().objectToByteArray(new
Qualifier((short) 559, (byte) SUFFIX_6));
byte[] col1v = new QualifierConverter().objectToByteArray(new
Qualifier((short) 558, (byte) SUFFIX_4));
byte[] col2v = new QualifierConverter().objectToByteArray(new
Qualifier((short) 559, (byte) SUFFIX_4));
byte[] col3v = new QualifierConverter().objectToByteArray(new
Qualifier((short) 560, (byte) SUFFIX_4));
byte[] col4v = new QualifierConverter().objectToByteArray(new
Qualifier((short) 561, (byte) SUFFIX_4));
byte[] col5v = new QualifierConverter().objectToByteArray(new
Qualifier((short) 562, (byte) SUFFIX_4));
byte[] col6v = new QualifierConverter().objectToByteArray(new
Qualifier((short) 563, (byte) SUFFIX_4));
// === INSERTION =//
Put put = new Put(row);
put.add(cf, col1, Bytes.toBytes((short) 1));
put.add(cf, col2,
[jira] [Commented] (HBASE-9458) Intermittent TestFlushSnapshotFromClient#testTakeSnapshotAfterMerge failure
[
https://issues.apache.org/jira/browse/HBASE-9458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13761635#comment-13761635
]
Matteo Bertozzi commented on HBASE-9458:
{quote}How about catching CorruptedSnapshotException from admin.snapshot() call
so that the test doesn't fail?{quote}
Yes, you can add a snapshot() method in SnapshotTestingUtil that wraps
admin.snapshot() with a while and a catch CorruptedException but then you don't
have much control on how many times it fails (which should be low) and what is
the cause.
but feel free to do it if you want the tests always passing and you don't
consider this failure something that you want to look at every time.
Intermittent TestFlushSnapshotFromClient#testTakeSnapshotAfterMerge failure
---
Key: HBASE-9458
URL: https://issues.apache.org/jira/browse/HBASE-9458
Project: HBase
Issue Type: Test
Reporter: Ted Yu
From
https://builds.apache.org/job/HBase-0.96/20/testReport/org.apache.hadoop.hbase.snapshot/TestFlushSnapshotFromClient/testTakeSnapshotAfterMerge/
:
{code}
org.apache.hadoop.hbase.snapshot.HBaseSnapshotException:
org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot {
ss=snapshotAfterMerge table=test type=FLUSH } had an error. Procedure
snapshotAfterMerge { waiting=[] done=[] }
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:210)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:221)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:125)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:96)
at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3120)
at
org.apache.hadoop.hbase.client.HBaseAdmin.snapshot(HBaseAdmin.java:2672)
at
org.apache.hadoop.hbase.client.HBaseAdmin.snapshot(HBaseAdmin.java:2605)
at
org.apache.hadoop.hbase.client.HBaseAdmin.snapshot(HBaseAdmin.java:2612)
at
org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient.testTakeSnapshotAfterMerge(TestFlushSnapshotFromClient.java:336)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException:
org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot {
ss=snapshotAfterMerge table=test type=FLUSH } had an error. Procedure
snapshotAfterMerge { waiting=[] done=[] }
at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:365)
at
org.apache.hadoop.hbase.master.HMaster.isSnapshotDone(HMaster.java:2947)
at
org.apache.hadoop.hbase.protobuf.generated.MasterAdminProtos$MasterAdminService$2.callBlockingMethod(MasterAdminProtos.java:32890)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
Caused by: org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException via
Failed taking snapshot { ss=snapshotAfterMerge table=test type=FLUSH } due to
exception:Missing parent hfile for:
30b951996ef34885a2f5d64e4acb2467.6d1ed72bf95759cb606e8a6efdc6908e:org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException:
Missing parent hfile for:
30b951996ef34885a2f5d64e4acb2467.6d1ed72bf95759cb606e8a6efdc6908e
at
