[jira] [Commented] (HBASE-6528) Raise the wait time for TestSplitLogWorker#testAcquireTaskAtStartup to reduce the failure probability
[ https://issues.apache.org/jira/browse/HBASE-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13454784#comment-13454784 ] ShiXing commented on HBASE-6528: [~lhofhansl] this test case was introduced when I fix HBASE-6520 which does not have any relationship with TestSplitLogWorker#testAcquireTaskAtStartup. I don't see any failing till now~ Raise the wait time for TestSplitLogWorker#testAcquireTaskAtStartup to reduce the failure probability - Key: HBASE-6528 URL: https://issues.apache.org/jira/browse/HBASE-6528 Project: HBase Issue Type: Bug Reporter: ShiXing Assignee: ShiXing Attachments: HBASE-6528-trunk-v1.patch All the code for the TestSplitLogWorker, only the testAcquireTaskAtStartup waits 100ms, other testCase wait 1000ms. The 100ms is short and sometimes causes testAcquireTaskAtStartup failure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6589) RegionServer can't load class for dynamically loaded coprocessors with self defined class
[
https://issues.apache.org/jira/browse/HBASE-6589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437680#comment-13437680
]
ShiXing commented on HBASE-6589:
It is when I try to invoke the CP. Because the MultiColumnSchema is not
recognized by the rs.
RegionServer can't load class for dynamically loaded coprocessors with self
defined class
-
Key: HBASE-6589
URL: https://issues.apache.org/jira/browse/HBASE-6589
Project: HBase
Issue Type: Bug
Components: coprocessors, regionserver
Reporter: ShiXing
When using coprocessor with custom classes like LongColumnInterpreter(mine is
MultiColumnSchema), the coprocessor can not work for hot deploy, if the
custom classes do not deploy in the regionserver's classpath. Although the
self-defined class is deployed in the regions' classpath through hdfs jar.
The exception threw at the regionserver's log:
{code}
2012-08-15 16:24:24,403 ERROR org.apache.hadoop.hbase.io.HbaseObjectWritable:
Error in readFields
java.io.IOException: Can't find class
com.taobao.hbase.coprocessor.MultiColumnSchema
at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:674)
at
org.apache.hadoop.hbase.client.coprocessor.Exec.readFields(Exec.java:114)
at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:682)
at
org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:125)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:1292)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1207)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:735)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:524)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:499)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassNotFoundException:
com.taobao.hbase.coprocessor.MultiColumnSchema
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:943)
at
org.apache.hadoop.hbase.io.HbaseObjectWritable.getClassByName(HbaseObjectWritable.java:784)
at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:671)
... 11 more
{code}
It is similar as HBASE-4946, but I do not know how to solve this bug.
If add these custom class to the RegionServer's classloader may fix it, but
it is conflicted with HBASE-6308 to prevent dependency conflicts.
Does anyone have some idea?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6589) RegionServer can't load class for dynamically loaded coprocessors with self defined class
ShiXing created HBASE-6589:
--
Summary: RegionServer can't load class for dynamically loaded
coprocessors with self defined class
Key: HBASE-6589
URL: https://issues.apache.org/jira/browse/HBASE-6589
Project: HBase
Issue Type: Bug
Components: coprocessors, regionserver
Reporter: ShiXing
When using coprocessor with custom classes like LongColumnInterpreter(mine is
MultiColumnSchema), the coprocessor can not work for hot deploy, if the custom
classes do not deploy in the regionserver's classpath. Although the
self-defined class is deployed in the regions' classpath through hdfs jar.
The exception threw at the regionserver's log:
{code}
2012-08-15 16:24:24,403 ERROR org.apache.hadoop.hbase.io.HbaseObjectWritable:
Error in readFields
java.io.IOException: Can't find class
com.taobao.hbase.coprocessor.MultiColumnSchema
at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:674)
at
org.apache.hadoop.hbase.client.coprocessor.Exec.readFields(Exec.java:114)
at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:682)
at
org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:125)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:1292)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1207)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:735)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:524)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:499)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassNotFoundException:
com.taobao.hbase.coprocessor.MultiColumnSchema
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:943)
at
org.apache.hadoop.hbase.io.HbaseObjectWritable.getClassByName(HbaseObjectWritable.java:784)
at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:671)
... 11 more
{code}
It is similar as HBASE-4946, but I do not know how to solve this bug.
If add these custom class to the RegionServer's classloader may fix it, but it
is conflicted with HBASE-6308 to prevent dependency conflicts.
Does anyone have some idea?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6520) MSLab May cause the Bytes.toLong not work correctly for increment
[
https://issues.apache.org/jira/browse/HBASE-6520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ShiXing updated HBASE-6520:
---
Attachment: HBASE-6520-0.94-v1.patch
patch for 0.94
MSLab May cause the Bytes.toLong not work correctly for increment
-
Key: HBASE-6520
URL: https://issues.apache.org/jira/browse/HBASE-6520
Project: HBase
Issue Type: Bug
Reporter: ShiXing
Assignee: ShiXing
Fix For: 0.96.0
Attachments: HBASE-6520-0.94-v1.patch, HBASE-6520-trunk-v1.patch
When use MemStoreLAB, the KeyValues will share the byte array allocated by
the MemStoreLAB, all the KeyValues' bytes attributes are the same byte
array. When use the functions such as Bytes.toLong(byte[] bytes, int offset):
{code}
public static long toLong(byte[] bytes, int offset) {
return toLong(bytes, offset, SIZEOF_LONG);
}
public static long toLong(byte[] bytes, int offset, final int length) {
if (length != SIZEOF_LONG || offset + length bytes.length) {
throw explainWrongLengthOrOffset(bytes, offset, length, SIZEOF_LONG);
}
long l = 0;
for(int i = offset; i offset + length; i++) {
l = 8;
l ^= bytes[i] 0xFF;
}
return l;
}
{code}
If we do not put a long value to the KeyValue, and read it as a long value in
HRegion.increment(),the check
{code}
offset + length bytes.length
{code}
will take no effects, because the bytes.length is not equal to
keyLength+valueLength, indeed it is MemStoreLAB chunkSize which is default
2048 * 1024.
I will paste the patch later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6520) MSLab May cause the Bytes.toLong does not work correctly for increment
[
https://issues.apache.org/jira/browse/HBASE-6520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ShiXing updated HBASE-6520:
---
Attachment: HBASE-6520-trunk-v1.patch
MSLab May cause the Bytes.toLong does not work correctly for increment
--
Key: HBASE-6520
URL: https://issues.apache.org/jira/browse/HBASE-6520
Project: HBase
Issue Type: Bug
Reporter: ShiXing
Assignee: ShiXing
Attachments: HBASE-6520-trunk-v1.patch
When use MemStoreLAB, the KeyValues will share the byte array allocated by
the MemStoreLAB, all the KeyValues' bytes attributes are the same byte
array. When use the functions such as Bytes.toLong(byte[] bytes, int offset):
{code}
public static long toLong(byte[] bytes, int offset) {
return toLong(bytes, offset, SIZEOF_LONG);
}
public static long toLong(byte[] bytes, int offset, final int length) {
if (length != SIZEOF_LONG || offset + length bytes.length) {
throw explainWrongLengthOrOffset(bytes, offset, length, SIZEOF_LONG);
}
long l = 0;
for(int i = offset; i offset + length; i++) {
l = 8;
l ^= bytes[i] 0xFF;
}
return l;
}
{code}
If we do not put a long value to the KeyValue, and read it as a long value in
HRegion.increment(),the check
{code}
offset + length bytes.length
{code}
will take no effects, because the bytes.length is not equal to
keyLength+valueLength, indeed it is MemStoreLAB chunkSize which is default
2048 * 1024.
I will paste the patch later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6520) MSLab May cause the Bytes.toLong does not work correctly for increment
[
https://issues.apache.org/jira/browse/HBASE-6520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13430418#comment-13430418
]
ShiXing commented on HBASE-6520:
bq. -1 core tests. The patch failed these unit tests:
bq.org.apache.hadoop.hbase.replication.TestReplication
bq.org.apache.hadoop.hbase.regionserver.TestSplitLogWorker
The patch does not modify anythong for the Replication and SplitLogWorker.
The TestReplication case failure may be caused by HBASE-3515 as the source code
described.
And through the TestSplitLog log:
{code}
2012-08-07 14:25:24,200 INFO [pool-1-thread-1]
regionserver.TestSplitLogWorker(127): testAcquireTaskAtStartup
2012-08-07 14:25:24,280 INFO [SplitLogWorker-rs,1,1]
regionserver.SplitLogWorker(135): SplitLogWorker rs,1,1 starting
2012-08-07 14:25:24,334 DEBUG [SplitLogWorker-rs,1,1] zookeeper.ZKUtil(1142):
split-log-worker-tests-0x1390179cef5 Retrieved 17 byte(s) of data from
znode /hbase/splitlog/tatas; data=PBUF\x08\x00\x12\x09\x0A\x03m...
2012-08-07 14:25:24,375 INFO [SplitLogWorker-rs,1,1]
regionserver.SplitLogWorker(267): worker rs,1,1 acquired task
/hbase/splitlog/tatas
2012-08-07 14:25:24,375 INFO [pool-1-thread-1]
regionserver.SplitLogWorker(521): Sending interrupt to stop the worker thread
2012-08-07 14:25:24,380 WARN [SplitLogWorker-rs,1,1]
regionserver.SplitLogWorker(292): task execution prempted /hbase/splitlog/tatas
{code}
The SplitLogWorker started 14:25:24,280, and we will wait 100ms ,yes, at
14:25:24,380, but at 14:25:24,375 we just grab the task, and our check thread
will sleep every 10ms, so the case failed.
Can we raise the wait time, such as 1000ms to reduce testSplitLogWorker case
failure probability?
MSLab May cause the Bytes.toLong does not work correctly for increment
--
Key: HBASE-6520
URL: https://issues.apache.org/jira/browse/HBASE-6520
Project: HBase
Issue Type: Bug
Reporter: ShiXing
Assignee: ShiXing
Fix For: 0.96.0
Attachments: HBASE-6520-trunk-v1.patch
When use MemStoreLAB, the KeyValues will share the byte array allocated by
the MemStoreLAB, all the KeyValues' bytes attributes are the same byte
array. When use the functions such as Bytes.toLong(byte[] bytes, int offset):
{code}
public static long toLong(byte[] bytes, int offset) {
return toLong(bytes, offset, SIZEOF_LONG);
}
public static long toLong(byte[] bytes, int offset, final int length) {
if (length != SIZEOF_LONG || offset + length bytes.length) {
throw explainWrongLengthOrOffset(bytes, offset, length, SIZEOF_LONG);
}
long l = 0;
for(int i = offset; i offset + length; i++) {
l = 8;
l ^= bytes[i] 0xFF;
}
return l;
}
{code}
If we do not put a long value to the KeyValue, and read it as a long value in
HRegion.increment(),the check
{code}
offset + length bytes.length
{code}
will take no effects, because the bytes.length is not equal to
keyLength+valueLength, indeed it is MemStoreLAB chunkSize which is default
2048 * 1024.
I will paste the patch later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6528) Raise the wait time for TestSplitLogWorker#testAcquireTaskAtStartup to reduce the failure probability
ShiXing created HBASE-6528: -- Summary: Raise the wait time for TestSplitLogWorker#testAcquireTaskAtStartup to reduce the failure probability Key: HBASE-6528 URL: https://issues.apache.org/jira/browse/HBASE-6528 Project: HBase Issue Type: Bug Reporter: ShiXing Assignee: ShiXing All the code for the TestSplitLogWorker, only the testAcquireTaskAtStartup waits 100ms, other testCase wait 1000ms. The 100ms is short and sometimes causes testAcquireTaskAtStartup failure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6528) Raise the wait time for TestSplitLogWorker#testAcquireTaskAtStartup to reduce the failure probability
[ https://issues.apache.org/jira/browse/HBASE-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ShiXing updated HBASE-6528: --- Attachment: HBASE-6528-trunk-v1.patch Raise the wait time for TestSplitLogWorker#testAcquireTaskAtStartup to reduce the failure probability - Key: HBASE-6528 URL: https://issues.apache.org/jira/browse/HBASE-6528 Project: HBase Issue Type: Bug Reporter: ShiXing Assignee: ShiXing Attachments: HBASE-6528-trunk-v1.patch All the code for the TestSplitLogWorker, only the testAcquireTaskAtStartup waits 100ms, other testCase wait 1000ms. The 100ms is short and sometimes causes testAcquireTaskAtStartup failure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6520) MSLab May cause the Bytes.toLong does not work correctly for increment
ShiXing created HBASE-6520:
--
Summary: MSLab May cause the Bytes.toLong does not work correctly
for increment
Key: HBASE-6520
URL: https://issues.apache.org/jira/browse/HBASE-6520
Project: HBase
Issue Type: Bug
Reporter: ShiXing
Assignee: ShiXing
When use MemStoreLAB, the KeyValues will share the byte array allocated by the
MemStoreLAB, all the KeyValues' bytes attributes are the same byte array.
When use the functions such as Bytes.toLong(byte[] bytes, int offset):
{code}
public static long toLong(byte[] bytes, int offset) {
return toLong(bytes, offset, SIZEOF_LONG);
}
public static long toLong(byte[] bytes, int offset, final int length) {
if (length != SIZEOF_LONG || offset + length bytes.length) {
throw explainWrongLengthOrOffset(bytes, offset, length, SIZEOF_LONG);
}
long l = 0;
for(int i = offset; i offset + length; i++) {
l = 8;
l ^= bytes[i] 0xFF;
}
return l;
}
{code}
If we do not put a long value to the KeyValue, and read it as a long value in
HRegion.increment(),the check
{code}
offset + length bytes.length
{code}
will take no effects, because the bytes.length is not equal to
keyLength+valueLength, indeed it is MemStoreLAB chunkSize which is default 2048
* 1024.
I will paste the patch later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3725) HBase increments from old value after delete and write to disk
[
https://issues.apache.org/jira/browse/HBASE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ShiXing updated HBASE-3725:
---
Attachment: HBASE-3725-0.92-V6.patch
toTed
bq. TestHRegion#testIncrementWithFlushAndDelete passed without that assignment.
Because the iscan is also read from memstore after I remove the code:
{code}
ListKeyValue fileResults = new ArrayListKeyValue();
- iscan.checkOnlyStoreFiles();
scanner = null;
try {
scanner = getScanner(iscan);
{code}
And there is no result in memstore, so increment will treat it as 0, it has the
same effect as delete.
I add this case in TestHRegion#testIncrementWithFlushAndDelete in V6.
HBase increments from old value after delete and write to disk
--
Key: HBASE-3725
URL: https://issues.apache.org/jira/browse/HBASE-3725
Project: HBase
Issue Type: Bug
Components: io, regionserver
Affects Versions: 0.90.1
Reporter: Nathaniel Cook
Assignee: Jonathan Gray
Attachments: HBASE-3725-0.92-V1.patch, HBASE-3725-0.92-V2.patch,
HBASE-3725-0.92-V3.patch, HBASE-3725-0.92-V4.patch, HBASE-3725-0.92-V5.patch,
HBASE-3725-0.92-V6.patch, HBASE-3725-Test-v1.patch, HBASE-3725-v3.patch,
HBASE-3725.patch
Deleted row values are sometimes used for starting points on new increments.
To reproduce:
Create a row r. Set column x to some default value.
Force hbase to write that value to the file system (such as restarting the
cluster).
Delete the row.
Call table.incrementColumnValue with some_value
Get the row.
The returned value in the column was incremented from the old value before
the row was deleted instead of being initialized to some_value.
Code to reproduce:
{code}
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.HTablePool;
import org.apache.hadoop.hbase.client.Increment;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.util.Bytes;
public class HBaseTestIncrement
{
static String tableName = testIncrement;
static byte[] infoCF = Bytes.toBytes(info);
static byte[] rowKey = Bytes.toBytes(test-rowKey);
static byte[] newInc = Bytes.toBytes(new);
static byte[] oldInc = Bytes.toBytes(old);
/**
* This code reproduces a bug with increment column values in hbase
* Usage: First run part one by passing '1' as the first arg
*Then restart the hbase cluster so it writes everything to disk
*Run part two by passing '2' as the first arg
*
* This will result in the old deleted data being found and used for
the increment calls
*
* @param args
* @throws IOException
*/
public static void main(String[] args) throws IOException
{
if(1.equals(args[0]))
partOne();
if(2.equals(args[0]))
partTwo();
if (both.equals(args[0]))
{
partOne();
partTwo();
}
}
/**
* Creates a table and increments a column value 10 times by 10 each
time.
* Results in a value of 100 for the column
*
* @throws IOException
*/
static void partOne()throws IOException
{
Configuration conf = HBaseConfiguration.create();
HBaseAdmin admin = new HBaseAdmin(conf);
HTableDescriptor tableDesc = new HTableDescriptor(tableName);
tableDesc.addFamily(new HColumnDescriptor(infoCF));
if(admin.tableExists(tableName))
{
admin.disableTable(tableName);
admin.deleteTable(tableName);
}
admin.createTable(tableDesc);
HTablePool pool = new HTablePool(conf, Integer.MAX_VALUE);
HTableInterface table = pool.getTable(Bytes.toBytes(tableName));
//Increment unitialized column
for (int j = 0; j 10; j++)
{
table.incrementColumnValue(rowKey, infoCF, oldInc,
(long)10);
Increment inc = new Increment(rowKey);
inc.addColumn(infoCF, newInc, (long)10);
table.increment(inc);
}
Get get = new Get(rowKey);
Result r
[jira] [Commented] (HBASE-3725) HBase increments from old value after delete and write to disk
[
https://issues.apache.org/jira/browse/HBASE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418892#comment-13418892
]
ShiXing commented on HBASE-3725:
@Ted
bq. I generate a region with 3 store files. The increment slows from 1810 tps
to 1020 tps, it slows 43.6%, .
The tps is increment the same rowkey.
The performance depends on how frequently the memstore flushed to the store
file. If I also do the same test case, the latest patch's performance is same
as the orig, because the increment rowkey is always in the memstore, and we do
not need to read the store file.
The difference is only for the rowKey that can't get the value from memstore,
it need do a more read from memstore , compared to the 0.92 trunk: read only
from store file.
You must know, the orig's high performance is just benefit by only read from
the memstore.
HBase increments from old value after delete and write to disk
--
Key: HBASE-3725
URL: https://issues.apache.org/jira/browse/HBASE-3725
Project: HBase
Issue Type: Bug
Components: io, regionserver
Affects Versions: 0.90.1
Reporter: Nathaniel Cook
Assignee: Jonathan Gray
Attachments: HBASE-3725-0.92-V1.patch, HBASE-3725-0.92-V2.patch,
HBASE-3725-0.92-V3.patch, HBASE-3725-0.92-V4.patch, HBASE-3725-0.92-V5.patch,
HBASE-3725-Test-v1.patch, HBASE-3725-v3.patch, HBASE-3725.patch
Deleted row values are sometimes used for starting points on new increments.
To reproduce:
Create a row r. Set column x to some default value.
Force hbase to write that value to the file system (such as restarting the
cluster).
Delete the row.
Call table.incrementColumnValue with some_value
Get the row.
The returned value in the column was incremented from the old value before
the row was deleted instead of being initialized to some_value.
Code to reproduce:
{code}
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.HTablePool;
import org.apache.hadoop.hbase.client.Increment;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.util.Bytes;
public class HBaseTestIncrement
{
static String tableName = testIncrement;
static byte[] infoCF = Bytes.toBytes(info);
static byte[] rowKey = Bytes.toBytes(test-rowKey);
static byte[] newInc = Bytes.toBytes(new);
static byte[] oldInc = Bytes.toBytes(old);
/**
* This code reproduces a bug with increment column values in hbase
* Usage: First run part one by passing '1' as the first arg
*Then restart the hbase cluster so it writes everything to disk
*Run part two by passing '2' as the first arg
*
* This will result in the old deleted data being found and used for
the increment calls
*
* @param args
* @throws IOException
*/
public static void main(String[] args) throws IOException
{
if(1.equals(args[0]))
partOne();
if(2.equals(args[0]))
partTwo();
if (both.equals(args[0]))
{
partOne();
partTwo();
}
}
/**
* Creates a table and increments a column value 10 times by 10 each
time.
* Results in a value of 100 for the column
*
* @throws IOException
*/
static void partOne()throws IOException
{
Configuration conf = HBaseConfiguration.create();
HBaseAdmin admin = new HBaseAdmin(conf);
HTableDescriptor tableDesc = new HTableDescriptor(tableName);
tableDesc.addFamily(new HColumnDescriptor(infoCF));
if(admin.tableExists(tableName))
{
admin.disableTable(tableName);
admin.deleteTable(tableName);
}
admin.createTable(tableDesc);
HTablePool pool = new HTablePool(conf, Integer.MAX_VALUE);
HTableInterface table = pool.getTable(Bytes.toBytes(tableName));
//Increment unitialized column
for (int j = 0; j 10; j++)
{
table.incrementColumnValue(rowKey, infoCF, oldInc,
(long)10);
Increment inc = new
[jira] [Commented] (HBASE-3725) HBase increments from old value after delete and write to disk
[
https://issues.apache.org/jira/browse/HBASE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418933#comment-13418933
]
ShiXing commented on HBASE-3725:
@Ted, the reassignment is because there is no interface to set the iscan back
to both memstore and filestore, because at the begining, the iscan is set
memstore
{code}
// memstore scan
iscan.checkOnlyMemStore();
{code}
HBase increments from old value after delete and write to disk
--
Key: HBASE-3725
URL: https://issues.apache.org/jira/browse/HBASE-3725
Project: HBase
Issue Type: Bug
Components: io, regionserver
Affects Versions: 0.90.1
Reporter: Nathaniel Cook
Assignee: Jonathan Gray
Attachments: HBASE-3725-0.92-V1.patch, HBASE-3725-0.92-V2.patch,
HBASE-3725-0.92-V3.patch, HBASE-3725-0.92-V4.patch, HBASE-3725-0.92-V5.patch,
HBASE-3725-Test-v1.patch, HBASE-3725-v3.patch, HBASE-3725.patch
Deleted row values are sometimes used for starting points on new increments.
To reproduce:
Create a row r. Set column x to some default value.
Force hbase to write that value to the file system (such as restarting the
cluster).
Delete the row.
Call table.incrementColumnValue with some_value
Get the row.
The returned value in the column was incremented from the old value before
the row was deleted instead of being initialized to some_value.
Code to reproduce:
{code}
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.HTablePool;
import org.apache.hadoop.hbase.client.Increment;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.util.Bytes;
public class HBaseTestIncrement
{
static String tableName = testIncrement;
static byte[] infoCF = Bytes.toBytes(info);
static byte[] rowKey = Bytes.toBytes(test-rowKey);
static byte[] newInc = Bytes.toBytes(new);
static byte[] oldInc = Bytes.toBytes(old);
/**
* This code reproduces a bug with increment column values in hbase
* Usage: First run part one by passing '1' as the first arg
*Then restart the hbase cluster so it writes everything to disk
*Run part two by passing '2' as the first arg
*
* This will result in the old deleted data being found and used for
the increment calls
*
* @param args
* @throws IOException
*/
public static void main(String[] args) throws IOException
{
if(1.equals(args[0]))
partOne();
if(2.equals(args[0]))
partTwo();
if (both.equals(args[0]))
{
partOne();
partTwo();
}
}
/**
* Creates a table and increments a column value 10 times by 10 each
time.
* Results in a value of 100 for the column
*
* @throws IOException
*/
static void partOne()throws IOException
{
Configuration conf = HBaseConfiguration.create();
HBaseAdmin admin = new HBaseAdmin(conf);
HTableDescriptor tableDesc = new HTableDescriptor(tableName);
tableDesc.addFamily(new HColumnDescriptor(infoCF));
if(admin.tableExists(tableName))
{
admin.disableTable(tableName);
admin.deleteTable(tableName);
}
admin.createTable(tableDesc);
HTablePool pool = new HTablePool(conf, Integer.MAX_VALUE);
HTableInterface table = pool.getTable(Bytes.toBytes(tableName));
//Increment unitialized column
for (int j = 0; j 10; j++)
{
table.incrementColumnValue(rowKey, infoCF, oldInc,
(long)10);
Increment inc = new Increment(rowKey);
inc.addColumn(infoCF, newInc, (long)10);
table.increment(inc);
}
Get get = new Get(rowKey);
Result r = table.get(get);
System.out.println(initial values: new +
Bytes.toLong(r.getValue(infoCF, newInc)) + old +
Bytes.toLong(r.getValue(infoCF, oldInc)));
}
/**
* First deletes the data then increments the column 10 times by 1
[jira] [Commented] (HBASE-6327) HLog can be null when create table
[ https://issues.apache.org/jira/browse/HBASE-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13417989#comment-13417989 ] ShiXing commented on HBASE-6327: @Ted, what is this issue's progress now? Can we commit it? HLog can be null when create table -- Key: HBASE-6327 URL: https://issues.apache.org/jira/browse/HBASE-6327 Project: HBase Issue Type: Bug Reporter: ShiXing Assignee: ShiXing Fix For: 0.96.0 Attachments: 6327.txt, HBASE-6327-trunk-V1.patch, createTableFailedMaster.log As HBASE-4010 discussed, the HLog can be null. We have meet createTable failed because the no use hlog. When createHReagion, the HLog.LogSyncer is run sync(), in under layer it call the DFSClient.DFSOutputStream.sync(). Then the hlog.closeAndDelete() was called，firstly the HLog.close() will interrupt the LogSyncer, and interrupt DFSClient.DFSOutputStream.sync().The DFSClient.DFSOutputStream will store the exception and throw it when we called DFSClient.close(). The HLog.close() call the writer.close()/DFSClient.close() after interrupt the LogSyncer. And there is no catch exception for the close(). So the Master throw exception to the client. There is no need to throw this exception, further， the hlog is no use. Our cluster is 0.90, the logs is attached, after closing hlog writer, there is no log for the createTable(). The trunk and 0.92, 0.94, we used just one hlog, and if the exception happends, the client will got createTable failed, but indeed ,we expect all the regions for the table can also be assigned. I will give the patch for this later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3725) HBase increments from old value after delete and write to disk
[
https://issues.apache.org/jira/browse/HBASE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ShiXing updated HBASE-3725:
---
Attachment: HBASE-3725-0.92-V5.patch
@stack, I generate a region with 3 store files. The increment slows from 1810
tps to 1020 tps, it slows 43.6%, :(.
I think we can fix the correctness by recreate the InternalScan when we can't
get the data from memstore, we will still get the data from memstore for the
delete not just from store files.
HBase increments from old value after delete and write to disk
--
Key: HBASE-3725
URL: https://issues.apache.org/jira/browse/HBASE-3725
Project: HBase
Issue Type: Bug
Components: io, regionserver
Affects Versions: 0.90.1
Reporter: Nathaniel Cook
Assignee: Jonathan Gray
Attachments: HBASE-3725-0.92-V1.patch, HBASE-3725-0.92-V2.patch,
HBASE-3725-0.92-V3.patch, HBASE-3725-0.92-V4.patch, HBASE-3725-0.92-V5.patch,
HBASE-3725-Test-v1.patch, HBASE-3725-v3.patch, HBASE-3725.patch
Deleted row values are sometimes used for starting points on new increments.
To reproduce:
Create a row r. Set column x to some default value.
Force hbase to write that value to the file system (such as restarting the
cluster).
Delete the row.
Call table.incrementColumnValue with some_value
Get the row.
The returned value in the column was incremented from the old value before
the row was deleted instead of being initialized to some_value.
Code to reproduce:
{code}
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.HTablePool;
import org.apache.hadoop.hbase.client.Increment;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.util.Bytes;
public class HBaseTestIncrement
{
static String tableName = testIncrement;
static byte[] infoCF = Bytes.toBytes(info);
static byte[] rowKey = Bytes.toBytes(test-rowKey);
static byte[] newInc = Bytes.toBytes(new);
static byte[] oldInc = Bytes.toBytes(old);
/**
* This code reproduces a bug with increment column values in hbase
* Usage: First run part one by passing '1' as the first arg
*Then restart the hbase cluster so it writes everything to disk
*Run part two by passing '2' as the first arg
*
* This will result in the old deleted data being found and used for
the increment calls
*
* @param args
* @throws IOException
*/
public static void main(String[] args) throws IOException
{
if(1.equals(args[0]))
partOne();
if(2.equals(args[0]))
partTwo();
if (both.equals(args[0]))
{
partOne();
partTwo();
}
}
/**
* Creates a table and increments a column value 10 times by 10 each
time.
* Results in a value of 100 for the column
*
* @throws IOException
*/
static void partOne()throws IOException
{
Configuration conf = HBaseConfiguration.create();
HBaseAdmin admin = new HBaseAdmin(conf);
HTableDescriptor tableDesc = new HTableDescriptor(tableName);
tableDesc.addFamily(new HColumnDescriptor(infoCF));
if(admin.tableExists(tableName))
{
admin.disableTable(tableName);
admin.deleteTable(tableName);
}
admin.createTable(tableDesc);
HTablePool pool = new HTablePool(conf, Integer.MAX_VALUE);
HTableInterface table = pool.getTable(Bytes.toBytes(tableName));
//Increment unitialized column
for (int j = 0; j 10; j++)
{
table.incrementColumnValue(rowKey, infoCF, oldInc,
(long)10);
Increment inc = new Increment(rowKey);
inc.addColumn(infoCF, newInc, (long)10);
table.increment(inc);
}
Get get = new Get(rowKey);
Result r = table.get(get);
System.out.println(initial values: new +
Bytes.toLong(r.getValue(infoCF, newInc)) + old +
Bytes.toLong(r.getValue(infoCF, oldInc)));
}
/**
*
[jira] [Commented] (HBASE-3725) HBase increments from old value after delete and write to disk
[
https://issues.apache.org/jira/browse/HBASE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418032#comment-13418032
]
ShiXing commented on HBASE-3725:
In V5, if the increment does not got the data it will got the data from store
file and memstore, compared to the orig implement(just from store file), I
think it will not slow the increment, it will only takes more time for the
memstore.
HBase increments from old value after delete and write to disk
--
Key: HBASE-3725
URL: https://issues.apache.org/jira/browse/HBASE-3725
Project: HBase
Issue Type: Bug
Components: io, regionserver
Affects Versions: 0.90.1
Reporter: Nathaniel Cook
Assignee: Jonathan Gray
Attachments: HBASE-3725-0.92-V1.patch, HBASE-3725-0.92-V2.patch,
HBASE-3725-0.92-V3.patch, HBASE-3725-0.92-V4.patch, HBASE-3725-0.92-V5.patch,
HBASE-3725-Test-v1.patch, HBASE-3725-v3.patch, HBASE-3725.patch
Deleted row values are sometimes used for starting points on new increments.
To reproduce:
Create a row r. Set column x to some default value.
Force hbase to write that value to the file system (such as restarting the
cluster).
Delete the row.
Call table.incrementColumnValue with some_value
Get the row.
The returned value in the column was incremented from the old value before
the row was deleted instead of being initialized to some_value.
Code to reproduce:
{code}
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.HTablePool;
import org.apache.hadoop.hbase.client.Increment;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.util.Bytes;
public class HBaseTestIncrement
{
static String tableName = testIncrement;
static byte[] infoCF = Bytes.toBytes(info);
static byte[] rowKey = Bytes.toBytes(test-rowKey);
static byte[] newInc = Bytes.toBytes(new);
static byte[] oldInc = Bytes.toBytes(old);
/**
* This code reproduces a bug with increment column values in hbase
* Usage: First run part one by passing '1' as the first arg
*Then restart the hbase cluster so it writes everything to disk
*Run part two by passing '2' as the first arg
*
* This will result in the old deleted data being found and used for
the increment calls
*
* @param args
* @throws IOException
*/
public static void main(String[] args) throws IOException
{
if(1.equals(args[0]))
partOne();
if(2.equals(args[0]))
partTwo();
if (both.equals(args[0]))
{
partOne();
partTwo();
}
}
/**
* Creates a table and increments a column value 10 times by 10 each
time.
* Results in a value of 100 for the column
*
* @throws IOException
*/
static void partOne()throws IOException
{
Configuration conf = HBaseConfiguration.create();
HBaseAdmin admin = new HBaseAdmin(conf);
HTableDescriptor tableDesc = new HTableDescriptor(tableName);
tableDesc.addFamily(new HColumnDescriptor(infoCF));
if(admin.tableExists(tableName))
{
admin.disableTable(tableName);
admin.deleteTable(tableName);
}
admin.createTable(tableDesc);
HTablePool pool = new HTablePool(conf, Integer.MAX_VALUE);
HTableInterface table = pool.getTable(Bytes.toBytes(tableName));
//Increment unitialized column
for (int j = 0; j 10; j++)
{
table.incrementColumnValue(rowKey, infoCF, oldInc,
(long)10);
Increment inc = new Increment(rowKey);
inc.addColumn(infoCF, newInc, (long)10);
table.increment(inc);
}
Get get = new Get(rowKey);
Result r = table.get(get);
System.out.println(initial values: new +
Bytes.toLong(r.getValue(infoCF, newInc)) + old +
Bytes.toLong(r.getValue(infoCF, oldInc)));
}
/**
* First deletes the data then increments the column 10
[jira] [Commented] (HBASE-6327) HLog can be null when create table
[ https://issues.apache.org/jira/browse/HBASE-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418037#comment-13418037 ] ShiXing commented on HBASE-6327: In deed, when create table, the hlog is no use completely. Also I can't decide whether all the exception could be catched. It will also faster the create table operation a little. Stack also mentioned this in HBASE-4010.So I choose not create the hlog when create table instead of catch exception. HLog can be null when create table -- Key: HBASE-6327 URL: https://issues.apache.org/jira/browse/HBASE-6327 Project: HBase Issue Type: Bug Reporter: ShiXing Assignee: ShiXing Fix For: 0.96.0 Attachments: 6327.txt, HBASE-6327-trunk-V1.patch, createTableFailedMaster.log As HBASE-4010 discussed, the HLog can be null. We have meet createTable failed because the no use hlog. When createHReagion, the HLog.LogSyncer is run sync(), in under layer it call the DFSClient.DFSOutputStream.sync(). Then the hlog.closeAndDelete() was called，firstly the HLog.close() will interrupt the LogSyncer, and interrupt DFSClient.DFSOutputStream.sync().The DFSClient.DFSOutputStream will store the exception and throw it when we called DFSClient.close(). The HLog.close() call the writer.close()/DFSClient.close() after interrupt the LogSyncer. And there is no catch exception for the close(). So the Master throw exception to the client. There is no need to throw this exception, further， the hlog is no use. Our cluster is 0.90, the logs is attached, after closing hlog writer, there is no log for the createTable(). The trunk and 0.92, 0.94, we used just one hlog, and if the exception happends, the client will got createTable failed, but indeed ,we expect all the regions for the table can also be assigned. I will give the patch for this later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6370) Add compression codec test at HMaster when createTable/modifyColumn/modifyTable
[ https://issues.apache.org/jira/browse/HBASE-6370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ShiXing updated HBASE-6370: --- Attachment: HBASE-6370-trunk-V2.patch Yes, I think the configuration is more acceptable for heterogeneous environment between master and regionservers. I set the configuration base.master.check.compression default true. Add compression codec test at HMaster when createTable/modifyColumn/modifyTable --- Key: HBASE-6370 URL: https://issues.apache.org/jira/browse/HBASE-6370 Project: HBase Issue Type: Improvement Reporter: ShiXing Assignee: ShiXing Priority: Minor Attachments: HBASE-6370-trunk-V1.patch, HBASE-6370-trunk-V2.patch We deployed a cluster that none of the regionserver supports the compression codec such like lzo, but the cluster user/client does not know this, and he specifies the family's compression codec by HColumnDescripto.setCompressionType(Compresson.Algorithm.LZO); Because the HBaseAdmin's createTable is async, so the client is waiting all the regions of the table to be online forever. And client does not know why the regions are not online until the HBase administrator find this problem. In deed, all of the regions are assigning by master, but regionserver's openHRegion always failed. In my option, we can suppose all the cluster's enviroment are the same, means if the master is deployed some lib, the regionserver should also be deployed. Of course above is just a suppose， in real deployment, the hbase dba may just deploy lib on regionserver or master. So I think this failure can be found earlier before master create the CreateTableHandler thread, and we can tell client quickly we didn't support this compression codec type. I will upload the patch later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6370) Add compression codec test at HMaster when createTable/modifyColumn/modifyTable
[ https://issues.apache.org/jira/browse/HBASE-6370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ShiXing updated HBASE-6370: --- Attachment: runAllTests.out The v3 looks nice, runAllTests passed on my server. Add compression codec test at HMaster when createTable/modifyColumn/modifyTable --- Key: HBASE-6370 URL: https://issues.apache.org/jira/browse/HBASE-6370 Project: HBase Issue Type: Improvement Reporter: ShiXing Assignee: ShiXing Priority: Minor Attachments: 6370v3.txt, HBASE-6370-trunk-V1.patch, HBASE-6370-trunk-V2.patch, runAllTests.out We deployed a cluster that none of the regionserver supports the compression codec such like lzo, but the cluster user/client does not know this, and he specifies the family's compression codec by HColumnDescripto.setCompressionType(Compresson.Algorithm.LZO); Because the HBaseAdmin's createTable is async, so the client is waiting all the regions of the table to be online forever. And client does not know why the regions are not online until the HBase administrator find this problem. In deed, all of the regions are assigning by master, but regionserver's openHRegion always failed. In my option, we can suppose all the cluster's enviroment are the same, means if the master is deployed some lib, the regionserver should also be deployed. Of course above is just a suppose， in real deployment, the hbase dba may just deploy lib on regionserver or master. So I think this failure can be found earlier before master create the CreateTableHandler thread, and we can tell client quickly we didn't support this compression codec type. I will upload the patch later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6370) Add compression codec test at HMaster when createTable/modifyColumn/modifyTable
[ https://issues.apache.org/jira/browse/HBASE-6370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ShiXing updated HBASE-6370: --- Summary: Add compression codec test at HMaster when createTable/modifyColumn/modifyTable (was: Add compression codec test at HMaster when create table) Add compression codec test at HMaster when createTable/modifyColumn/modifyTable --- Key: HBASE-6370 URL: https://issues.apache.org/jira/browse/HBASE-6370 Project: HBase Issue Type: Improvement Reporter: ShiXing Assignee: ShiXing Priority: Minor We deployed a cluster that none of the regionserver supports the compression codec such like lzo, but the cluster user/client does not know this, and he specifies the family's compression codec by HColumnDescripto.setCompressionType(Compresson.Algorithm.LZO); Because the HBaseAdmin's createTable is async, so the client is waiting all the regions of the table to be online forever. And client does not know why the regions are not online until the HBase administrator find this problem. In deed, all of the regions are assigning by master, but regionserver's openHRegion always failed. In my option, we can suppose all the cluster's enviroment are the same, means if the master is deployed some lib, the regionserver should also be deployed. Of course above is just a suppose， in real deployment, the hbase dba may just deploy lib on regionserver or master. So I think this failure can be found earlier before master create the CreateTableHandler thread, and we can tell client quickly we didn't support this compression codec type. I will upload the patch later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6370) Add compression codec test at HMaster when createTable/modifyColumn/modifyTable
[ https://issues.apache.org/jira/browse/HBASE-6370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ShiXing updated HBASE-6370: --- Attachment: HBASE-6370-trunk-V1.patch Use the already exists unit test through the HBaseAdmin's. Add compression codec test at HMaster when createTable/modifyColumn/modifyTable --- Key: HBASE-6370 URL: https://issues.apache.org/jira/browse/HBASE-6370 Project: HBase Issue Type: Improvement Reporter: ShiXing Assignee: ShiXing Priority: Minor Attachments: HBASE-6370-trunk-V1.patch We deployed a cluster that none of the regionserver supports the compression codec such like lzo, but the cluster user/client does not know this, and he specifies the family's compression codec by HColumnDescripto.setCompressionType(Compresson.Algorithm.LZO); Because the HBaseAdmin's createTable is async, so the client is waiting all the regions of the table to be online forever. And client does not know why the regions are not online until the HBase administrator find this problem. In deed, all of the regions are assigning by master, but regionserver's openHRegion always failed. In my option, we can suppose all the cluster's enviroment are the same, means if the master is deployed some lib, the regionserver should also be deployed. Of course above is just a suppose， in real deployment, the hbase dba may just deploy lib on regionserver or master. So I think this failure can be found earlier before master create the CreateTableHandler thread, and we can tell client quickly we didn't support this compression codec type. I will upload the patch later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6370) Add compression codec test at HMaster when create table
ShiXing created HBASE-6370: -- Summary: Add compression codec test at HMaster when create table Key: HBASE-6370 URL: https://issues.apache.org/jira/browse/HBASE-6370 Project: HBase Issue Type: Improvement Reporter: ShiXing Assignee: ShiXing Priority: Minor We deployed a cluster that none of the regionserver supports the compression codec such like lzo, but the cluster user/client does not know this, and he specifies the family's compression codec by HColumnDescripto.setCompressionType(Compresson.Algorithm.LZO); Because the HBaseAdmin's createTable is async, so the client is waiting all the regions of the table to be online forever. And client does not know why the regions are not online until the HBase administrator find this problem. In deed, all of the regions are assigning by master, but regionserver's openHRegion always failed. In my option, we can suppose all the cluster's enviroment are the same, means if the master is deployed some lib, the regionserver should also be deployed. Of course above is just a suppose， in real deployment, the hbase dba may just deploy lib on regionserver or master. So I think this failure can be found earlier before master create the CreateTableHandler thread, and we can tell client quickly we didn't support this compression codec type. I will upload the patch later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3725) HBase increments from old value after delete and write to disk
[
https://issues.apache.org/jira/browse/HBASE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13409205#comment-13409205
]
ShiXing commented on HBASE-3725:
The trunk does not have this bug, for the HRegion#getLastIncrement() has been
discarded already.
HBase increments from old value after delete and write to disk
--
Key: HBASE-3725
URL: https://issues.apache.org/jira/browse/HBASE-3725
Project: HBase
Issue Type: Bug
Components: io, regionserver
Affects Versions: 0.90.1
Reporter: Nathaniel Cook
Assignee: Jonathan Gray
Attachments: HBASE-3725-0.92-V1.patch, HBASE-3725-0.92-V2.patch,
HBASE-3725-0.92-V3.patch, HBASE-3725-0.92-V4.patch, HBASE-3725-Test-v1.patch,
HBASE-3725-v3.patch, HBASE-3725.patch
Deleted row values are sometimes used for starting points on new increments.
To reproduce:
Create a row r. Set column x to some default value.
Force hbase to write that value to the file system (such as restarting the
cluster).
Delete the row.
Call table.incrementColumnValue with some_value
Get the row.
The returned value in the column was incremented from the old value before
the row was deleted instead of being initialized to some_value.
Code to reproduce:
{code}
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.HTablePool;
import org.apache.hadoop.hbase.client.Increment;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.util.Bytes;
public class HBaseTestIncrement
{
static String tableName = testIncrement;
static byte[] infoCF = Bytes.toBytes(info);
static byte[] rowKey = Bytes.toBytes(test-rowKey);
static byte[] newInc = Bytes.toBytes(new);
static byte[] oldInc = Bytes.toBytes(old);
/**
* This code reproduces a bug with increment column values in hbase
* Usage: First run part one by passing '1' as the first arg
*Then restart the hbase cluster so it writes everything to disk
*Run part two by passing '2' as the first arg
*
* This will result in the old deleted data being found and used for
the increment calls
*
* @param args
* @throws IOException
*/
public static void main(String[] args) throws IOException
{
if(1.equals(args[0]))
partOne();
if(2.equals(args[0]))
partTwo();
if (both.equals(args[0]))
{
partOne();
partTwo();
}
}
/**
* Creates a table and increments a column value 10 times by 10 each
time.
* Results in a value of 100 for the column
*
* @throws IOException
*/
static void partOne()throws IOException
{
Configuration conf = HBaseConfiguration.create();
HBaseAdmin admin = new HBaseAdmin(conf);
HTableDescriptor tableDesc = new HTableDescriptor(tableName);
tableDesc.addFamily(new HColumnDescriptor(infoCF));
if(admin.tableExists(tableName))
{
admin.disableTable(tableName);
admin.deleteTable(tableName);
}
admin.createTable(tableDesc);
HTablePool pool = new HTablePool(conf, Integer.MAX_VALUE);
HTableInterface table = pool.getTable(Bytes.toBytes(tableName));
//Increment unitialized column
for (int j = 0; j 10; j++)
{
table.incrementColumnValue(rowKey, infoCF, oldInc,
(long)10);
Increment inc = new Increment(rowKey);
inc.addColumn(infoCF, newInc, (long)10);
table.increment(inc);
}
Get get = new Get(rowKey);
Result r = table.get(get);
System.out.println(initial values: new +
Bytes.toLong(r.getValue(infoCF, newInc)) + old +
Bytes.toLong(r.getValue(infoCF, oldInc)));
}
/**
* First deletes the data then increments the column 10 times by 1 each
time
*
* Should result in a value of 10 but it doesn't, it results in a
values of 110
*
* @throws IOException
[jira] [Commented] (HBASE-3725) HBase increments from old value after delete and write to disk
[
https://issues.apache.org/jira/browse/HBASE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13409213#comment-13409213
]
ShiXing commented on HBASE-3725:
Mean the performance is important than the correctness?
I think Correctness should always be first.
HBase increments from old value after delete and write to disk
--
Key: HBASE-3725
URL: https://issues.apache.org/jira/browse/HBASE-3725
Project: HBase
Issue Type: Bug
Components: io, regionserver
Affects Versions: 0.90.1
Reporter: Nathaniel Cook
Assignee: Jonathan Gray
Attachments: HBASE-3725-0.92-V1.patch, HBASE-3725-0.92-V2.patch,
HBASE-3725-0.92-V3.patch, HBASE-3725-0.92-V4.patch, HBASE-3725-Test-v1.patch,
HBASE-3725-v3.patch, HBASE-3725.patch
Deleted row values are sometimes used for starting points on new increments.
To reproduce:
Create a row r. Set column x to some default value.
Force hbase to write that value to the file system (such as restarting the
cluster).
Delete the row.
Call table.incrementColumnValue with some_value
Get the row.
The returned value in the column was incremented from the old value before
the row was deleted instead of being initialized to some_value.
Code to reproduce:
{code}
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.HTablePool;
import org.apache.hadoop.hbase.client.Increment;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.util.Bytes;
public class HBaseTestIncrement
{
static String tableName = testIncrement;
static byte[] infoCF = Bytes.toBytes(info);
static byte[] rowKey = Bytes.toBytes(test-rowKey);
static byte[] newInc = Bytes.toBytes(new);
static byte[] oldInc = Bytes.toBytes(old);
/**
* This code reproduces a bug with increment column values in hbase
* Usage: First run part one by passing '1' as the first arg
*Then restart the hbase cluster so it writes everything to disk
*Run part two by passing '2' as the first arg
*
* This will result in the old deleted data being found and used for
the increment calls
*
* @param args
* @throws IOException
*/
public static void main(String[] args) throws IOException
{
if(1.equals(args[0]))
partOne();
if(2.equals(args[0]))
partTwo();
if (both.equals(args[0]))
{
partOne();
partTwo();
}
}
/**
* Creates a table and increments a column value 10 times by 10 each
time.
* Results in a value of 100 for the column
*
* @throws IOException
*/
static void partOne()throws IOException
{
Configuration conf = HBaseConfiguration.create();
HBaseAdmin admin = new HBaseAdmin(conf);
HTableDescriptor tableDesc = new HTableDescriptor(tableName);
tableDesc.addFamily(new HColumnDescriptor(infoCF));
if(admin.tableExists(tableName))
{
admin.disableTable(tableName);
admin.deleteTable(tableName);
}
admin.createTable(tableDesc);
HTablePool pool = new HTablePool(conf, Integer.MAX_VALUE);
HTableInterface table = pool.getTable(Bytes.toBytes(tableName));
//Increment unitialized column
for (int j = 0; j 10; j++)
{
table.incrementColumnValue(rowKey, infoCF, oldInc,
(long)10);
Increment inc = new Increment(rowKey);
inc.addColumn(infoCF, newInc, (long)10);
table.increment(inc);
}
Get get = new Get(rowKey);
Result r = table.get(get);
System.out.println(initial values: new +
Bytes.toLong(r.getValue(infoCF, newInc)) + old +
Bytes.toLong(r.getValue(infoCF, oldInc)));
}
/**
* First deletes the data then increments the column 10 times by 1 each
time
*
* Should result in a value of 10 but it doesn't, it results in a
values of 110
*
* @throws IOException
[jira] [Updated] (HBASE-3725) HBase increments from old value after delete and write to disk
[
https://issues.apache.org/jira/browse/HBASE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ShiXing updated HBASE-3725:
---
Attachment: HBASE-3725-0.92-V4.patch
pls review it.
HBase increments from old value after delete and write to disk
--
Key: HBASE-3725
URL: https://issues.apache.org/jira/browse/HBASE-3725
Project: HBase
Issue Type: Bug
Components: io, regionserver
Affects Versions: 0.90.1
Reporter: Nathaniel Cook
Assignee: Jonathan Gray
Attachments: HBASE-3725-0.92-V1.patch, HBASE-3725-0.92-V2.patch,
HBASE-3725-0.92-V3.patch, HBASE-3725-0.92-V4.patch, HBASE-3725-Test-v1.patch,
HBASE-3725-v3.patch, HBASE-3725.patch
Deleted row values are sometimes used for starting points on new increments.
To reproduce:
Create a row r. Set column x to some default value.
Force hbase to write that value to the file system (such as restarting the
cluster).
Delete the row.
Call table.incrementColumnValue with some_value
Get the row.
The returned value in the column was incremented from the old value before
the row was deleted instead of being initialized to some_value.
Code to reproduce:
{code}
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.HTablePool;
import org.apache.hadoop.hbase.client.Increment;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.util.Bytes;
public class HBaseTestIncrement
{
static String tableName = testIncrement;
static byte[] infoCF = Bytes.toBytes(info);
static byte[] rowKey = Bytes.toBytes(test-rowKey);
static byte[] newInc = Bytes.toBytes(new);
static byte[] oldInc = Bytes.toBytes(old);
/**
* This code reproduces a bug with increment column values in hbase
* Usage: First run part one by passing '1' as the first arg
*Then restart the hbase cluster so it writes everything to disk
*Run part two by passing '2' as the first arg
*
* This will result in the old deleted data being found and used for
the increment calls
*
* @param args
* @throws IOException
*/
public static void main(String[] args) throws IOException
{
if(1.equals(args[0]))
partOne();
if(2.equals(args[0]))
partTwo();
if (both.equals(args[0]))
{
partOne();
partTwo();
}
}
/**
* Creates a table and increments a column value 10 times by 10 each
time.
* Results in a value of 100 for the column
*
* @throws IOException
*/
static void partOne()throws IOException
{
Configuration conf = HBaseConfiguration.create();
HBaseAdmin admin = new HBaseAdmin(conf);
HTableDescriptor tableDesc = new HTableDescriptor(tableName);
tableDesc.addFamily(new HColumnDescriptor(infoCF));
if(admin.tableExists(tableName))
{
admin.disableTable(tableName);
admin.deleteTable(tableName);
}
admin.createTable(tableDesc);
HTablePool pool = new HTablePool(conf, Integer.MAX_VALUE);
HTableInterface table = pool.getTable(Bytes.toBytes(tableName));
//Increment unitialized column
for (int j = 0; j 10; j++)
{
table.incrementColumnValue(rowKey, infoCF, oldInc,
(long)10);
Increment inc = new Increment(rowKey);
inc.addColumn(infoCF, newInc, (long)10);
table.increment(inc);
}
Get get = new Get(rowKey);
Result r = table.get(get);
System.out.println(initial values: new +
Bytes.toLong(r.getValue(infoCF, newInc)) + old +
Bytes.toLong(r.getValue(infoCF, oldInc)));
}
/**
* First deletes the data then increments the column 10 times by 1 each
time
*
* Should result in a value of 10 but it doesn't, it results in a
values of 110
*
* @throws IOException
*/
static void partTwo()throws IOException
{
Configuration conf =
[jira] [Commented] (HBASE-3725) HBase increments from old value after delete and write to disk
[
https://issues.apache.org/jira/browse/HBASE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13406928#comment-13406928
]
ShiXing commented on HBASE-3725:
bq.Is this a fix for 0.92 only ShiXing?
yes
bq.Doing a get instead of a getLastIncrement is good because we'll handle
deletes. Does it slow down increments doing a get rather than getLastIncrement?
I think it will slow down increments a little. I have not test the performance.
But the correctness is affected.
In test case, if there is a put , a flush, and a delete, the increment will
read out the data in the StoreFile by flushing , because the getLastIncrement
will read the memstore first, and return empty because there is no row just one
delete, and then read the StoreFile without merge with delete. You can find the
bug by my test case or described as Description of this issue.
{code}
junit.framework.AssertionFailedError: expected:10 but was:20
at junit.framework.Assert.fail(Assert.java:50)
at junit.framework.Assert.failNotEquals(Assert.java:287)
at junit.framework.Assert.assertEquals(Assert.java:67)
at junit.framework.Assert.assertEquals(Assert.java:134)
at junit.framework.Assert.assertEquals(Assert.java:140)
at
org.apache.hadoop.hbase.regionserver.TestHRegion.testIncrementWithFlushAndDelete(TestHRegion.java:3446)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
{code}
HBase increments from old value after delete and write to disk
--
Key: HBASE-3725
URL: https://issues.apache.org/jira/browse/HBASE-3725
Project: HBase
Issue Type: Bug
Components: io, regionserver
Affects Versions: 0.90.1
Reporter: Nathaniel Cook
Assignee: Jonathan Gray
Attachments: HBASE-3725-0.92-V1.patch, HBASE-3725-0.92-V2.patch,
HBASE-3725-0.92-V3.patch, HBASE-3725-Test-v1.patch, HBASE-3725-v3.patch,
HBASE-3725.patch
Deleted row values are sometimes used for starting points on new increments.
To reproduce:
Create a row r. Set column x to some default value.
Force hbase to write that value to the file system (such as restarting the
cluster).
Delete the row.
Call table.incrementColumnValue with some_value
Get the row.
The returned value in the column was incremented from the old value before
the row was deleted instead of being initialized to some_value.
Code to reproduce:
{code}
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.HTablePool;
import org.apache.hadoop.hbase.client.Increment;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.util.Bytes;
public class HBaseTestIncrement
{
static String tableName = testIncrement;
static byte[] infoCF = Bytes.toBytes(info);
static byte[] rowKey = Bytes.toBytes(test-rowKey);
static byte[] newInc = Bytes.toBytes(new);
static byte[] oldInc = Bytes.toBytes(old);
/**
* This
[jira] [Commented] (HBASE-3725) HBase increments from old value after delete and write to disk
[
https://issues.apache.org/jira/browse/HBASE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13406929#comment-13406929
]
ShiXing commented on HBASE-3725:
bq. But the correctness is affected.
I mean the 0.92 trunk code for increment will make data wrong for some case.
Because once the increment row's memstore is flushed, if there is no Delete, it
will be no affected, if there is a Delete, the data will be wrong undoubtly.
HBase increments from old value after delete and write to disk
--
Key: HBASE-3725
URL: https://issues.apache.org/jira/browse/HBASE-3725
Project: HBase
Issue Type: Bug
Components: io, regionserver
Affects Versions: 0.90.1
Reporter: Nathaniel Cook
Assignee: Jonathan Gray
Attachments: HBASE-3725-0.92-V1.patch, HBASE-3725-0.92-V2.patch,
HBASE-3725-0.92-V3.patch, HBASE-3725-Test-v1.patch, HBASE-3725-v3.patch,
HBASE-3725.patch
Deleted row values are sometimes used for starting points on new increments.
To reproduce:
Create a row r. Set column x to some default value.
Force hbase to write that value to the file system (such as restarting the
cluster).
Delete the row.
Call table.incrementColumnValue with some_value
Get the row.
The returned value in the column was incremented from the old value before
the row was deleted instead of being initialized to some_value.
Code to reproduce:
{code}
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.HTablePool;
import org.apache.hadoop.hbase.client.Increment;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.util.Bytes;
public class HBaseTestIncrement
{
static String tableName = testIncrement;
static byte[] infoCF = Bytes.toBytes(info);
static byte[] rowKey = Bytes.toBytes(test-rowKey);
static byte[] newInc = Bytes.toBytes(new);
static byte[] oldInc = Bytes.toBytes(old);
/**
* This code reproduces a bug with increment column values in hbase
* Usage: First run part one by passing '1' as the first arg
*Then restart the hbase cluster so it writes everything to disk
*Run part two by passing '2' as the first arg
*
* This will result in the old deleted data being found and used for
the increment calls
*
* @param args
* @throws IOException
*/
public static void main(String[] args) throws IOException
{
if(1.equals(args[0]))
partOne();
if(2.equals(args[0]))
partTwo();
if (both.equals(args[0]))
{
partOne();
partTwo();
}
}
/**
* Creates a table and increments a column value 10 times by 10 each
time.
* Results in a value of 100 for the column
*
* @throws IOException
*/
static void partOne()throws IOException
{
Configuration conf = HBaseConfiguration.create();
HBaseAdmin admin = new HBaseAdmin(conf);
HTableDescriptor tableDesc = new HTableDescriptor(tableName);
tableDesc.addFamily(new HColumnDescriptor(infoCF));
if(admin.tableExists(tableName))
{
admin.disableTable(tableName);
admin.deleteTable(tableName);
}
admin.createTable(tableDesc);
HTablePool pool = new HTablePool(conf, Integer.MAX_VALUE);
HTableInterface table = pool.getTable(Bytes.toBytes(tableName));
//Increment unitialized column
for (int j = 0; j 10; j++)
{
table.incrementColumnValue(rowKey, infoCF, oldInc,
(long)10);
Increment inc = new Increment(rowKey);
inc.addColumn(infoCF, newInc, (long)10);
table.increment(inc);
}
Get get = new Get(rowKey);
Result r = table.get(get);
System.out.println(initial values: new +
Bytes.toLong(r.getValue(infoCF, newInc)) + old +
Bytes.toLong(r.getValue(infoCF, oldInc)));
}
/**
* First deletes the data then increments the column 10 times by 1 each
[jira] [Commented] (HBASE-6327) HLog can be null when create table
[ https://issues.apache.org/jira/browse/HBASE-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13406948#comment-13406948 ] ShiXing commented on HBASE-6327: @Ted Not yet, I have commit this patch into our code, deploy it in our test cluster and run several days. HLog can be null when create table -- Key: HBASE-6327 URL: https://issues.apache.org/jira/browse/HBASE-6327 Project: HBase Issue Type: Bug Reporter: ShiXing Assignee: ShiXing Attachments: HBASE-6327-trunk-V1.patch, createTableFailedMaster.log As HBASE-4010 discussed, the HLog can be null. We have meet createTable failed because the no use hlog. When createHReagion, the HLog.LogSyncer is run sync(), in under layer it call the DFSClient.DFSOutputStream.sync(). Then the hlog.closeAndDelete() was called，firstly the HLog.close() will interrupt the LogSyncer, and interrupt DFSClient.DFSOutputStream.sync().The DFSClient.DFSOutputStream will store the exception and throw it when we called DFSClient.close(). The HLog.close() call the writer.close()/DFSClient.close() after interrupt the LogSyncer. And there is no catch exception for the close(). So the Master throw exception to the client. There is no need to throw this exception, further， the hlog is no use. Our cluster is 0.90, the logs is attached, after closing hlog writer, there is no log for the createTable(). The trunk and 0.92, 0.94, we used just one hlog, and if the exception happends, the client will got createTable failed, but indeed ,we expect all the regions for the table can also be assigned. I will give the patch for this later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6327) HLog can be null when create table
[
https://issues.apache.org/jira/browse/HBASE-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13407687#comment-13407687
]
ShiXing commented on HBASE-6327:
TestServerCustomProtocol passed locally for me.
---
Test set: org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol
---
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.787 sec
In the testReport, it shows that table.coprocessorExec()'s result does not
contain some region.
{code}
Error Message
Results should contain region
test,bbb,1341531289968.2e838a3a40aeebafb0c3093c5098ce23. for row 'bbb'
Stacktrace
java.lang.AssertionError: Results should contain region
test,bbb,1341531289968.2e838a3a40aeebafb0c3093c5098ce23. for row 'bbb'
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol.verifyRegionResults(TestServerCustomProtocol.java:361)
at
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol.verifyRegionResults(TestServerCustomProtocol.java:353)
at
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol.testSingleMethod(TestServerCustomProtocol.java:191)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{code}
But the logs shows that the regions are all assigned:
{code}
2012-07-05 23:34:50,184 INFO [pool-1-thread-1] master.AssignmentManager(1751):
Assigning region test,,1341531289965.cc5dddba941b937a5ebd9f80a582024a. to
asf001.sp2.ygridcore.net,59854,1341531283318
2012-07-05 23:34:50,242 INFO [pool-1-thread-1] master.AssignmentManager(1751):
Assigning region test,bbb,1341531289968.2e838a3a40aeebafb0c3093c5098ce23. to
asf001.sp2.ygridcore.net,59854,1341531283318
2012-07-05 23:34:50,258 INFO [pool-1-thread-1] master.AssignmentManager(1751):
Assigning region test,ccc,1341531289970.9bcbc7b96efbb051d046fb87d207bc5d. to
asf001.sp2.ygridcore.net,59854,1341531283318
{code}
Also I found there are maybe thread leak log:
{code}
Standard Error
2012-07-05 23:34:51,775 INFO [pool-1-thread-1] hbase.ResourceChecker(145):
before regionserver.TestServerCustomProtocol#testSingleMethod: 144 threads, 271
file descriptors 3 connections,
2012-07-05 23:34:51,793 DEBUG [pool-1-thread-1] client.MetaScanner(202):
Scanning .META. starting at row=test,,00 for max=2147483647 rows
using hconnection 0x81b83c
2012-07-05 23:34:51,879 INFO [pool-1-thread-1] hbase.ResourceChecker(145):
after regionserver.TestServerCustomProtocol#testSingleMethod: 147 threads (was
144), 271 file descriptors (was 271). 3 connections (was 3), -thread leak?-
{code}
I also looked several other test case log, for example some of the coprocessors
https://builds.apache.org/job/PreCommit-HBASE-Build/2333//testReport/org.apache.hadoop.hbase.coprocessor/,
there is no thread leak log.
HLog can be null when create table
--
Key: HBASE-6327
URL: https://issues.apache.org/jira/browse/HBASE-6327
Project: HBase
Issue Type: Bug
Reporter: ShiXing
Assignee: ShiXing
Attachments: 6327.txt, HBASE-6327-trunk-V1.patch,
createTableFailedMaster.log
As HBASE-4010 discussed, the HLog can be null.
We have meet createTable failed because the no use hlog.
When createHReagion, the HLog.LogSyncer is run sync(), in under layer it call
the DFSClient.DFSOutputStream.sync().
Then the hlog.closeAndDelete() was called，firstly the HLog.close() will
interrupt the LogSyncer, and interrupt DFSClient.DFSOutputStream.sync().The
DFSClient.DFSOutputStream will store the exception and throw it when we
called DFSClient.close().
The HLog.close() call the writer.close()/DFSClient.close() after interrupt
the LogSyncer. And there is no catch exception for the close().
So the Master throw exception to the client. There is no need to throw this
exception, further， the hlog is no use.
Our cluster is 0.90, the logs is attached, after closing hlog writer, there
is no log for the createTable().
The trunk and 0.92, 0.94, we used just one hlog, and if the exception
happends, the client will got createTable failed, but indeed ,we expect all
the regions for the table can also be assigned.
I will give the patch for this later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6327) HLog can be null when create table
[ https://issues.apache.org/jira/browse/HBASE-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13407688#comment-13407688 ] ShiXing commented on HBASE-6327: Maybe the thread leak is normal, because the TestServerCustomProtocol#testSingleMethod used the HTable#coprocessorExec, and there are 3 regions for this table, so it creates 3 threads to get the result. HLog can be null when create table -- Key: HBASE-6327 URL: https://issues.apache.org/jira/browse/HBASE-6327 Project: HBase Issue Type: Bug Reporter: ShiXing Assignee: ShiXing Attachments: 6327.txt, HBASE-6327-trunk-V1.patch, createTableFailedMaster.log As HBASE-4010 discussed, the HLog can be null. We have meet createTable failed because the no use hlog. When createHReagion, the HLog.LogSyncer is run sync(), in under layer it call the DFSClient.DFSOutputStream.sync(). Then the hlog.closeAndDelete() was called，firstly the HLog.close() will interrupt the LogSyncer, and interrupt DFSClient.DFSOutputStream.sync().The DFSClient.DFSOutputStream will store the exception and throw it when we called DFSClient.close(). The HLog.close() call the writer.close()/DFSClient.close() after interrupt the LogSyncer. And there is no catch exception for the close(). So the Master throw exception to the client. There is no need to throw this exception, further， the hlog is no use. Our cluster is 0.90, the logs is attached, after closing hlog writer, there is no log for the createTable(). The trunk and 0.92, 0.94, we used just one hlog, and if the exception happends, the client will got createTable failed, but indeed ,we expect all the regions for the table can also be assigned. I will give the patch for this later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6327) HLog can be null when create table
ShiXing created HBASE-6327: -- Summary: HLog can be null when create table Key: HBASE-6327 URL: https://issues.apache.org/jira/browse/HBASE-6327 Project: HBase Issue Type: Bug Reporter: ShiXing Assignee: ShiXing Attachments: createTableFailedMaster.log As HBASE-4010 discussed, the HLog can be null. We have meet createTable failed because the no use hlog. When createHReagion, the HLog.LogSyncer is run sync(), in under layer it call the DFSClient.DFSOutputStream.sync(). Then the hlog.closeAndDelete() was called，firstly the HLog.close() will interrupt the LogSyncer, and interrupt DFSClient.DFSOutputStream.sync().The DFSClient.DFSOutputStream will store the exception and throw it when we called DFSClient.close(). The HLog.close() call the writer.close()/DFSClient.close() after interrupt the LogSyncer. And there is no catch exception for the close(). So the Master throw exception to the client. There is no need to throw this exception, further， the hlog is no use. Our cluster is 0.90, the logs is attached, after closing hlog writer, there is no log for the createTable(). The trunk and 0.92, 0.94, we used just one hlog, and if the exception happends, the client will got createTable failed, but indeed ,all the regions for the table can also be assigned. I will give the patch for this later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6327) HLog can be null when create table
[ https://issues.apache.org/jira/browse/HBASE-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ShiXing updated HBASE-6327: --- Attachment: createTableFailedMaster.log HLog can be null when create table -- Key: HBASE-6327 URL: https://issues.apache.org/jira/browse/HBASE-6327 Project: HBase Issue Type: Bug Reporter: ShiXing Assignee: ShiXing Attachments: createTableFailedMaster.log As HBASE-4010 discussed, the HLog can be null. We have meet createTable failed because the no use hlog. When createHReagion, the HLog.LogSyncer is run sync(), in under layer it call the DFSClient.DFSOutputStream.sync(). Then the hlog.closeAndDelete() was called，firstly the HLog.close() will interrupt the LogSyncer, and interrupt DFSClient.DFSOutputStream.sync().The DFSClient.DFSOutputStream will store the exception and throw it when we called DFSClient.close(). The HLog.close() call the writer.close()/DFSClient.close() after interrupt the LogSyncer. And there is no catch exception for the close(). So the Master throw exception to the client. There is no need to throw this exception, further， the hlog is no use. Our cluster is 0.90, the logs is attached, after closing hlog writer, there is no log for the createTable(). The trunk and 0.92, 0.94, we used just one hlog, and if the exception happends, the client will got createTable failed, but indeed ,all the regions for the table can also be assigned. I will give the patch for this later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6327) HLog can be null when create table
[ https://issues.apache.org/jira/browse/HBASE-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ShiXing updated HBASE-6327: --- Description: As HBASE-4010 discussed, the HLog can be null. We have meet createTable failed because the no use hlog. When createHReagion, the HLog.LogSyncer is run sync(), in under layer it call the DFSClient.DFSOutputStream.sync(). Then the hlog.closeAndDelete() was called，firstly the HLog.close() will interrupt the LogSyncer, and interrupt DFSClient.DFSOutputStream.sync().The DFSClient.DFSOutputStream will store the exception and throw it when we called DFSClient.close(). The HLog.close() call the writer.close()/DFSClient.close() after interrupt the LogSyncer. And there is no catch exception for the close(). So the Master throw exception to the client. There is no need to throw this exception, further， the hlog is no use. Our cluster is 0.90, the logs is attached, after closing hlog writer, there is no log for the createTable(). The trunk and 0.92, 0.94, we used just one hlog, and if the exception happends, the client will got createTable failed, but indeed ,we expect all the regions for the table can also be assigned. I will give the patch for this later. was: As HBASE-4010 discussed, the HLog can be null. We have meet createTable failed because the no use hlog. When createHReagion, the HLog.LogSyncer is run sync(), in under layer it call the DFSClient.DFSOutputStream.sync(). Then the hlog.closeAndDelete() was called，firstly the HLog.close() will interrupt the LogSyncer, and interrupt DFSClient.DFSOutputStream.sync().The DFSClient.DFSOutputStream will store the exception and throw it when we called DFSClient.close(). The HLog.close() call the writer.close()/DFSClient.close() after interrupt the LogSyncer. And there is no catch exception for the close(). So the Master throw exception to the client. There is no need to throw this exception, further， the hlog is no use. Our cluster is 0.90, the logs is attached, after closing hlog writer, there is no log for the createTable(). The trunk and 0.92, 0.94, we used just one hlog, and if the exception happends, the client will got createTable failed, but indeed ,all the regions for the table can also be assigned. I will give the patch for this later. HLog can be null when create table -- Key: HBASE-6327 URL: https://issues.apache.org/jira/browse/HBASE-6327 Project: HBase Issue Type: Bug Reporter: ShiXing Assignee: ShiXing Attachments: createTableFailedMaster.log As HBASE-4010 discussed, the HLog can be null. We have meet createTable failed because the no use hlog. When createHReagion, the HLog.LogSyncer is run sync(), in under layer it call the DFSClient.DFSOutputStream.sync(). Then the hlog.closeAndDelete() was called，firstly the HLog.close() will interrupt the LogSyncer, and interrupt DFSClient.DFSOutputStream.sync().The DFSClient.DFSOutputStream will store the exception and throw it when we called DFSClient.close(). The HLog.close() call the writer.close()/DFSClient.close() after interrupt the LogSyncer. And there is no catch exception for the close(). So the Master throw exception to the client. There is no need to throw this exception, further， the hlog is no use. Our cluster is 0.90, the logs is attached, after closing hlog writer, there is no log for the createTable(). The trunk and 0.92, 0.94, we used just one hlog, and if the exception happends, the client will got createTable failed, but indeed ,we expect all the regions for the table can also be assigned. I will give the patch for this later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6327) HLog can be null when create table
[ https://issues.apache.org/jira/browse/HBASE-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ShiXing updated HBASE-6327: --- Attachment: HBASE-6327-trunk-V1.patch The trunk code of the HLog.sync() use group sync, the interruption as described will not affect the createTable(). But I think we can save little time and simplify the createTable() logic. There is no ut. HLog can be null when create table -- Key: HBASE-6327 URL: https://issues.apache.org/jira/browse/HBASE-6327 Project: HBase Issue Type: Bug Reporter: ShiXing Assignee: ShiXing Attachments: HBASE-6327-trunk-V1.patch, createTableFailedMaster.log As HBASE-4010 discussed, the HLog can be null. We have meet createTable failed because the no use hlog. When createHReagion, the HLog.LogSyncer is run sync(), in under layer it call the DFSClient.DFSOutputStream.sync(). Then the hlog.closeAndDelete() was called，firstly the HLog.close() will interrupt the LogSyncer, and interrupt DFSClient.DFSOutputStream.sync().The DFSClient.DFSOutputStream will store the exception and throw it when we called DFSClient.close(). The HLog.close() call the writer.close()/DFSClient.close() after interrupt the LogSyncer. And there is no catch exception for the close(). So the Master throw exception to the client. There is no need to throw this exception, further， the hlog is no use. Our cluster is 0.90, the logs is attached, after closing hlog writer, there is no log for the createTable(). The trunk and 0.92, 0.94, we used just one hlog, and if the exception happends, the client will got createTable failed, but indeed ,we expect all the regions for the table can also be assigned. I will give the patch for this later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6292) Compact can skip the security access control
[
https://issues.apache.org/jira/browse/HBASE-6292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13404667#comment-13404667
]
ShiXing commented on HBASE-6292:
@Andrew, I see the ACL Matrix in HBASE-6192
{code}
RegionServer | CompactSelection | REGION | INTERNAL | NA
{code}
And this patch has changed it by add preCompactSelection requirePermission().
Should we change the doc for the ACL Matrix?
Compact can skip the security access control
Key: HBASE-6292
URL: https://issues.apache.org/jira/browse/HBASE-6292
Project: HBase
Issue Type: Sub-task
Components: security
Affects Versions: 0.94.0, 0.96.0, 0.94.1
Reporter: ShiXing
Assignee: ShiXing
Labels: acl, security
Fix For: 0.96.0, 0.94.1
Attachments: HBASE-6292-trunk-V1.patch
When client sends compact command to rs, the rs just create a
CompactionRequest, and then put it into the thread pool to process the
CompactionRequest. And when the region do the compact, it uses the rs's ugi
to process the compact, so the compact can successfully done.
Example:
user mapred do not have permission Admin,
{code}
hbase(main):001:0 user_permission 'Security'
UserTable,Family,Qualifier:Permission
mapred Security,f1,c1: [Permission:
actions=READ,WRITE]
hbase(main):004:0 put 'Security', 'r6', 'f1:c1', 'v9'
0 row(s) in 0.0590 seconds
hbase(main):005:0 put 'Security', 'r6', 'f1:c1', 'v10'
0 row(s) in 0.0040 seconds
hbase(main):006:0 compact 'Security'
0 row(s) in 0.0260 seconds
{code}
Maybe we can add permission check in the preCompactSelection() ?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3725) HBase increments from old value after delete and write to disk
[
https://issues.apache.org/jira/browse/HBASE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ShiXing updated HBASE-3725:
---
Attachment: HBASE-3725-0.92-V3.patch
If the Delete and next Increment occured the same millionsecond, the Increment
will not take effect for followed Increment, because the Delete will lay over
the Increment.
HBase increments from old value after delete and write to disk
--
Key: HBASE-3725
URL: https://issues.apache.org/jira/browse/HBASE-3725
Project: HBase
Issue Type: Bug
Components: io, regionserver
Affects Versions: 0.90.1
Reporter: Nathaniel Cook
Assignee: Jonathan Gray
Attachments: HBASE-3725-0.92-V1.patch, HBASE-3725-0.92-V2.patch,
HBASE-3725-0.92-V3.patch, HBASE-3725-Test-v1.patch, HBASE-3725-v3.patch,
HBASE-3725.patch
Deleted row values are sometimes used for starting points on new increments.
To reproduce:
Create a row r. Set column x to some default value.
Force hbase to write that value to the file system (such as restarting the
cluster).
Delete the row.
Call table.incrementColumnValue with some_value
Get the row.
The returned value in the column was incremented from the old value before
the row was deleted instead of being initialized to some_value.
Code to reproduce:
{code}
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.HTablePool;
import org.apache.hadoop.hbase.client.Increment;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.util.Bytes;
public class HBaseTestIncrement
{
static String tableName = testIncrement;
static byte[] infoCF = Bytes.toBytes(info);
static byte[] rowKey = Bytes.toBytes(test-rowKey);
static byte[] newInc = Bytes.toBytes(new);
static byte[] oldInc = Bytes.toBytes(old);
/**
* This code reproduces a bug with increment column values in hbase
* Usage: First run part one by passing '1' as the first arg
*Then restart the hbase cluster so it writes everything to disk
*Run part two by passing '2' as the first arg
*
* This will result in the old deleted data being found and used for
the increment calls
*
* @param args
* @throws IOException
*/
public static void main(String[] args) throws IOException
{
if(1.equals(args[0]))
partOne();
if(2.equals(args[0]))
partTwo();
if (both.equals(args[0]))
{
partOne();
partTwo();
}
}
/**
* Creates a table and increments a column value 10 times by 10 each
time.
* Results in a value of 100 for the column
*
* @throws IOException
*/
static void partOne()throws IOException
{
Configuration conf = HBaseConfiguration.create();
HBaseAdmin admin = new HBaseAdmin(conf);
HTableDescriptor tableDesc = new HTableDescriptor(tableName);
tableDesc.addFamily(new HColumnDescriptor(infoCF));
if(admin.tableExists(tableName))
{
admin.disableTable(tableName);
admin.deleteTable(tableName);
}
admin.createTable(tableDesc);
HTablePool pool = new HTablePool(conf, Integer.MAX_VALUE);
HTableInterface table = pool.getTable(Bytes.toBytes(tableName));
//Increment unitialized column
for (int j = 0; j 10; j++)
{
table.incrementColumnValue(rowKey, infoCF, oldInc,
(long)10);
Increment inc = new Increment(rowKey);
inc.addColumn(infoCF, newInc, (long)10);
table.increment(inc);
}
Get get = new Get(rowKey);
Result r = table.get(get);
System.out.println(initial values: new +
Bytes.toLong(r.getValue(infoCF, newInc)) + old +
Bytes.toLong(r.getValue(infoCF, oldInc)));
}
/**
* First deletes the data then increments the column 10 times by 1 each
time
*
* Should result in a value of 10 but it doesn't, it results in a
values of 110
*
[jira] [Created] (HBASE-6292) Compact can skip the security access control
ShiXing created HBASE-6292:
--
Summary: Compact can skip the security access control
Key: HBASE-6292
URL: https://issues.apache.org/jira/browse/HBASE-6292
Project: HBase
Issue Type: Bug
Components: security
Reporter: ShiXing
Assignee: ShiXing
When client sends compact command to rs, the rs just create a
CompactionRequest, and then put it into the thread pool to process the
CompactionRequest. And when the region do the compact, it uses the rs's ugi to
process the compact, so the compact can successfully done.
Example:
user mapred do not have permission Admin,
{code}
hbase(main):001:0 user_permission 'Security'
UserTable,Family,Qualifier:Permission
mapred Security,f1,c1: [Permission:
actions=READ,WRITE]
hbase(main):004:0 put 'Security', 'r6', 'f1:c1', 'v9'
0 row(s) in 0.0590 seconds
hbase(main):005:0 put 'Security', 'r6', 'f1:c1', 'v10'
0 row(s) in 0.0040 seconds
hbase(main):006:0 compact 'Security'
0 row(s) in 0.0260 seconds
{code}
Maybe we can add permission check in the preCompactSelection() ?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6195) Increment data will be lost when the memstore is flushed
[ https://issues.apache.org/jira/browse/HBASE-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403811#comment-13403811 ] ShiXing commented on HBASE-6195: yes, I think so. Increment data will be lost when the memstore is flushed Key: HBASE-6195 URL: https://issues.apache.org/jira/browse/HBASE-6195 Project: HBase Issue Type: Bug Components: regionserver Reporter: ShiXing Assignee: ShiXing Fix For: 0.96.0, 0.94.1 Attachments: 6195-trunk-V7.patch, 6195.addendum, HBASE-6195-trunk-V2.patch, HBASE-6195-trunk-V3.patch, HBASE-6195-trunk-V4.patch, HBASE-6195-trunk-V5.patch, HBASE-6195-trunk-V6.patch, HBASE-6195-trunk.patch There are two problems in increment() now: First: I see that the timestamp(the variable now) in HRegion's Increment() is generated before got the rowLock, so when there are multi-thread increment the same row, although it generate earlier, it may got the lock later. Because increment just store one version, so till now, the result will still be right. When the region is flushing, these increment will read the kv from snapshot and memstore with whose timestamp is larger, and write it back to memstore. If the snapshot's timestamp larger than the memstore, the increment will got the old data and then do the increment, it's wrong. Secondly: Also there is a risk in increment. Because it writes the memstore first and then HLog, so if it writes HLog failed, the client will also read the incremented value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6292) Compact can skip the security access control
[
https://issues.apache.org/jira/browse/HBASE-6292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ShiXing updated HBASE-6292:
---
Attachment: HBASE-6292-trunk-V1.patch
Patch for trunk
If this patch can be commited, then the patches for 0.92 0.94.0 and 0.94.1 I
will supply.
Compact can skip the security access control
Key: HBASE-6292
URL: https://issues.apache.org/jira/browse/HBASE-6292
Project: HBase
Issue Type: Sub-task
Components: security
Affects Versions: 0.94.0, 0.96.0, 0.94.1
Reporter: ShiXing
Assignee: ShiXing
Labels: acl, security
Attachments: HBASE-6292-trunk-V1.patch
When client sends compact command to rs, the rs just create a
CompactionRequest, and then put it into the thread pool to process the
CompactionRequest. And when the region do the compact, it uses the rs's ugi
to process the compact, so the compact can successfully done.
Example:
user mapred do not have permission Admin,
{code}
hbase(main):001:0 user_permission 'Security'
UserTable,Family,Qualifier:Permission
mapred Security,f1,c1: [Permission:
actions=READ,WRITE]
hbase(main):004:0 put 'Security', 'r6', 'f1:c1', 'v9'
0 row(s) in 0.0590 seconds
hbase(main):005:0 put 'Security', 'r6', 'f1:c1', 'v10'
0 row(s) in 0.0040 seconds
hbase(main):006:0 compact 'Security'
0 row(s) in 0.0260 seconds
{code}
Maybe we can add permission check in the preCompactSelection() ?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3725) HBase increments from old value after delete and write to disk
[
https://issues.apache.org/jira/browse/HBASE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403165#comment-13403165
]
ShiXing commented on HBASE-3725:
Is there any progress?
HBase increments from old value after delete and write to disk
--
Key: HBASE-3725
URL: https://issues.apache.org/jira/browse/HBASE-3725
Project: HBase
Issue Type: Bug
Components: io, regionserver
Affects Versions: 0.90.1
Reporter: Nathaniel Cook
Assignee: Jonathan Gray
Attachments: HBASE-3725-Test-v1.patch, HBASE-3725-v3.patch,
HBASE-3725.patch
Deleted row values are sometimes used for starting points on new increments.
To reproduce:
Create a row r. Set column x to some default value.
Force hbase to write that value to the file system (such as restarting the
cluster).
Delete the row.
Call table.incrementColumnValue with some_value
Get the row.
The returned value in the column was incremented from the old value before
the row was deleted instead of being initialized to some_value.
Code to reproduce:
{code}
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.HTablePool;
import org.apache.hadoop.hbase.client.Increment;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.util.Bytes;
public class HBaseTestIncrement
{
static String tableName = testIncrement;
static byte[] infoCF = Bytes.toBytes(info);
static byte[] rowKey = Bytes.toBytes(test-rowKey);
static byte[] newInc = Bytes.toBytes(new);
static byte[] oldInc = Bytes.toBytes(old);
/**
* This code reproduces a bug with increment column values in hbase
* Usage: First run part one by passing '1' as the first arg
*Then restart the hbase cluster so it writes everything to disk
*Run part two by passing '2' as the first arg
*
* This will result in the old deleted data being found and used for
the increment calls
*
* @param args
* @throws IOException
*/
public static void main(String[] args) throws IOException
{
if(1.equals(args[0]))
partOne();
if(2.equals(args[0]))
partTwo();
if (both.equals(args[0]))
{
partOne();
partTwo();
}
}
/**
* Creates a table and increments a column value 10 times by 10 each
time.
* Results in a value of 100 for the column
*
* @throws IOException
*/
static void partOne()throws IOException
{
Configuration conf = HBaseConfiguration.create();
HBaseAdmin admin = new HBaseAdmin(conf);
HTableDescriptor tableDesc = new HTableDescriptor(tableName);
tableDesc.addFamily(new HColumnDescriptor(infoCF));
if(admin.tableExists(tableName))
{
admin.disableTable(tableName);
admin.deleteTable(tableName);
}
admin.createTable(tableDesc);
HTablePool pool = new HTablePool(conf, Integer.MAX_VALUE);
HTableInterface table = pool.getTable(Bytes.toBytes(tableName));
//Increment unitialized column
for (int j = 0; j 10; j++)
{
table.incrementColumnValue(rowKey, infoCF, oldInc,
(long)10);
Increment inc = new Increment(rowKey);
inc.addColumn(infoCF, newInc, (long)10);
table.increment(inc);
}
Get get = new Get(rowKey);
Result r = table.get(get);
System.out.println(initial values: new +
Bytes.toLong(r.getValue(infoCF, newInc)) + old +
Bytes.toLong(r.getValue(infoCF, oldInc)));
}
/**
* First deletes the data then increments the column 10 times by 1 each
time
*
* Should result in a value of 10 but it doesn't, it results in a
values of 110
*
* @throws IOException
*/
static void partTwo()throws IOException
{
Configuration conf = HBaseConfiguration.create();
HTablePool pool = new HTablePool(conf,
[jira] [Commented] (HBASE-3725) HBase increments from old value after delete and write to disk
[
https://issues.apache.org/jira/browse/HBASE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403170#comment-13403170
]
ShiXing commented on HBASE-3725:
Sorry for mistake of Ctrl+Enter.
I think the fixup could just change the calls of getLastIncrement() to get(), I
see that in 0.94 the code is alreay remove the getLastIncrement() function.
HBase increments from old value after delete and write to disk
--
Key: HBASE-3725
URL: https://issues.apache.org/jira/browse/HBASE-3725
Project: HBase
Issue Type: Bug
Components: io, regionserver
Affects Versions: 0.90.1
Reporter: Nathaniel Cook
Assignee: Jonathan Gray
Attachments: HBASE-3725-Test-v1.patch, HBASE-3725-v3.patch,
HBASE-3725.patch
Deleted row values are sometimes used for starting points on new increments.
To reproduce:
Create a row r. Set column x to some default value.
Force hbase to write that value to the file system (such as restarting the
cluster).
Delete the row.
Call table.incrementColumnValue with some_value
Get the row.
The returned value in the column was incremented from the old value before
the row was deleted instead of being initialized to some_value.
Code to reproduce:
{code}
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.HTablePool;
import org.apache.hadoop.hbase.client.Increment;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.util.Bytes;
public class HBaseTestIncrement
{
static String tableName = testIncrement;
static byte[] infoCF = Bytes.toBytes(info);
static byte[] rowKey = Bytes.toBytes(test-rowKey);
static byte[] newInc = Bytes.toBytes(new);
static byte[] oldInc = Bytes.toBytes(old);
/**
* This code reproduces a bug with increment column values in hbase
* Usage: First run part one by passing '1' as the first arg
*Then restart the hbase cluster so it writes everything to disk
*Run part two by passing '2' as the first arg
*
* This will result in the old deleted data being found and used for
the increment calls
*
* @param args
* @throws IOException
*/
public static void main(String[] args) throws IOException
{
if(1.equals(args[0]))
partOne();
if(2.equals(args[0]))
partTwo();
if (both.equals(args[0]))
{
partOne();
partTwo();
}
}
/**
* Creates a table and increments a column value 10 times by 10 each
time.
* Results in a value of 100 for the column
*
* @throws IOException
*/
static void partOne()throws IOException
{
Configuration conf = HBaseConfiguration.create();
HBaseAdmin admin = new HBaseAdmin(conf);
HTableDescriptor tableDesc = new HTableDescriptor(tableName);
tableDesc.addFamily(new HColumnDescriptor(infoCF));
if(admin.tableExists(tableName))
{
admin.disableTable(tableName);
admin.deleteTable(tableName);
}
admin.createTable(tableDesc);
HTablePool pool = new HTablePool(conf, Integer.MAX_VALUE);
HTableInterface table = pool.getTable(Bytes.toBytes(tableName));
//Increment unitialized column
for (int j = 0; j 10; j++)
{
table.incrementColumnValue(rowKey, infoCF, oldInc,
(long)10);
Increment inc = new Increment(rowKey);
inc.addColumn(infoCF, newInc, (long)10);
table.increment(inc);
}
Get get = new Get(rowKey);
Result r = table.get(get);
System.out.println(initial values: new +
Bytes.toLong(r.getValue(infoCF, newInc)) + old +
Bytes.toLong(r.getValue(infoCF, oldInc)));
}
/**
* First deletes the data then increments the column 10 times by 1 each
time
*
* Should result in a value of 10 but it doesn't, it results in a
values of 110
*
* @throws IOException
*/
[jira] [Updated] (HBASE-3725) HBase increments from old value after delete and write to disk
[
https://issues.apache.org/jira/browse/HBASE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ShiXing updated HBASE-3725:
---
Attachment: HBASE-3725-0.92-V1.patch
remove the getLastIncrement function.
HBase increments from old value after delete and write to disk
--
Key: HBASE-3725
URL: https://issues.apache.org/jira/browse/HBASE-3725
Project: HBase
Issue Type: Bug
Components: io, regionserver
Affects Versions: 0.90.1
Reporter: Nathaniel Cook
Assignee: Jonathan Gray
Attachments: HBASE-3725-0.92-V1.patch, HBASE-3725-Test-v1.patch,
HBASE-3725-v3.patch, HBASE-3725.patch
Deleted row values are sometimes used for starting points on new increments.
To reproduce:
Create a row r. Set column x to some default value.
Force hbase to write that value to the file system (such as restarting the
cluster).
Delete the row.
Call table.incrementColumnValue with some_value
Get the row.
The returned value in the column was incremented from the old value before
the row was deleted instead of being initialized to some_value.
Code to reproduce:
{code}
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.HTablePool;
import org.apache.hadoop.hbase.client.Increment;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.util.Bytes;
public class HBaseTestIncrement
{
static String tableName = testIncrement;
static byte[] infoCF = Bytes.toBytes(info);
static byte[] rowKey = Bytes.toBytes(test-rowKey);
static byte[] newInc = Bytes.toBytes(new);
static byte[] oldInc = Bytes.toBytes(old);
/**
* This code reproduces a bug with increment column values in hbase
* Usage: First run part one by passing '1' as the first arg
*Then restart the hbase cluster so it writes everything to disk
*Run part two by passing '2' as the first arg
*
* This will result in the old deleted data being found and used for
the increment calls
*
* @param args
* @throws IOException
*/
public static void main(String[] args) throws IOException
{
if(1.equals(args[0]))
partOne();
if(2.equals(args[0]))
partTwo();
if (both.equals(args[0]))
{
partOne();
partTwo();
}
}
/**
* Creates a table and increments a column value 10 times by 10 each
time.
* Results in a value of 100 for the column
*
* @throws IOException
*/
static void partOne()throws IOException
{
Configuration conf = HBaseConfiguration.create();
HBaseAdmin admin = new HBaseAdmin(conf);
HTableDescriptor tableDesc = new HTableDescriptor(tableName);
tableDesc.addFamily(new HColumnDescriptor(infoCF));
if(admin.tableExists(tableName))
{
admin.disableTable(tableName);
admin.deleteTable(tableName);
}
admin.createTable(tableDesc);
HTablePool pool = new HTablePool(conf, Integer.MAX_VALUE);
HTableInterface table = pool.getTable(Bytes.toBytes(tableName));
//Increment unitialized column
for (int j = 0; j 10; j++)
{
table.incrementColumnValue(rowKey, infoCF, oldInc,
(long)10);
Increment inc = new Increment(rowKey);
inc.addColumn(infoCF, newInc, (long)10);
table.increment(inc);
}
Get get = new Get(rowKey);
Result r = table.get(get);
System.out.println(initial values: new +
Bytes.toLong(r.getValue(infoCF, newInc)) + old +
Bytes.toLong(r.getValue(infoCF, oldInc)));
}
/**
* First deletes the data then increments the column 10 times by 1 each
time
*
* Should result in a value of 10 but it doesn't, it results in a
values of 110
*
* @throws IOException
*/
static void partTwo()throws IOException
{
Configuration conf = HBaseConfiguration.create();
HTablePool
[jira] [Updated] (HBASE-6269) Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
[
https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ShiXing updated HBASE-6269:
---
Attachment: runAllTests.out.txt
I run
mvn test -P runAllTests
and the output is in the runAllTests.out.txt
Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest
KeyValue
-
Key: HBASE-6269
URL: https://issues.apache.org/jira/browse/HBASE-6269
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.0
Reporter: ShiXing
Assignee: ShiXing
Attachments: HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch,
runAllTests.out.txt
When I fix the bug HBASE-6195, there is happened to find sometimes the test
case will fail, https://builds.apache.org/job/HBase-0.94/259/.
If there are two Put/Increment with same row, family, qualifier, timestamp
and different memstoreTS, after each Put/Increment, we do a memstore flush.
So there will be two StoreFile with same KeyValue(except memstoreTS and
SequenceId).
When I got the row, I always got the old records, the test case like this:
{code}
public void testPutWithMemStoreFlush() throws Exception {
Configuration conf = HBaseConfiguration.create();
String method = testPutWithMemStoreFlush;
byte[] tableName = Bytes.toBytes(method);
byte[] family = Bytes.toBytes(family);;
byte[] qualifier = Bytes.toBytes(qualifier);
byte[] row = Bytes.toBytes(putRow);
byte[] value = null;
this.region = initHRegion(tableName, method, conf, family);
Put put = null;
Get get = null;
ListKeyValue kvs = null;
Result res = null;
put = new Put(row);
value = Bytes.toBytes(value0);
put.add(family, qualifier, 1234567l, value);
region.put(put);
System.out.print(get value before flush after put value0 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
region.flushcache();
System.out.print(get value after flush after put value0 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
put = new Put(row);
value = Bytes.toBytes(value1);
put.add(family, qualifier, 1234567l, value);
region.put(put);
System.out.print(get value before flush after put value1 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
region.flushcache();
System.out.print(get value after flush after put value1 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
put = new Put(row);
value = Bytes.toBytes(value2);
put.add(family, qualifier, 1234567l, value);
region.put(put);
System.out.print(get value before flush after put value2 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
region.flushcache();
System.out.print(get value after flush after put value2 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
}
{code}
and the result print as followed:
{code}
get value before flush after put value0 : value0
get value after flush after put value0 : value0
get value before flush after put value1 : value1
get value after flush after put value1 : value0
get value before flush after put value2 : value2
get value after flush after put value2 : value0
{code}
I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners
are sorted by SequenceId, so the latest
[jira] [Created] (HBASE-6269) Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
ShiXing created HBASE-6269:
--
Summary: Lazyseek should use the maxSequenseId StoreFile's
KeyValue as the latest KeyValue
Key: HBASE-6269
URL: https://issues.apache.org/jira/browse/HBASE-6269
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.0
Reporter: ShiXing
Assignee: ShiXing
Attachments: HBASE-6269-v1.patch
When I fix the bug HBASE-6195, there is happened to find sometimes the test
case will fail, https://builds.apache.org/job/HBase-0.94/259/.
If there are two Put/Increment with same row, family, qualifier, timestamp and
different memstoreTS, after each Put/Increment, we do a memstore flush. So
there will be two StoreFile with same KeyValue(except memstoreTS and
SequenceId).
When I got the row, I always got the old records, the test case like this:
{code}
public void testPutWithMemStoreFlush() throws Exception {
Configuration conf = HBaseConfiguration.create();
String method = testPutWithMemStoreFlush;
byte[] tableName = Bytes.toBytes(method);
byte[] family = Bytes.toBytes(family);;
byte[] qualifier = Bytes.toBytes(qualifier);
byte[] row = Bytes.toBytes(putRow);
byte[] value = null;
this.region = initHRegion(tableName, method, conf, family);
Put put = null;
Get get = null;
ListKeyValue kvs = null;
Result res = null;
put = new Put(row);
value = Bytes.toBytes(value0);
put.add(family, qualifier, 1234567l, value);
region.put(put);
System.out.print(get value before flush after put value0 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
region.flushcache();
System.out.print(get value after flush after put value0 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
put = new Put(row);
value = Bytes.toBytes(value1);
put.add(family, qualifier, 1234567l, value);
region.put(put);
System.out.print(get value before flush after put value1 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
region.flushcache();
System.out.print(get value after flush after put value1 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
put = new Put(row);
value = Bytes.toBytes(value2);
put.add(family, qualifier, 1234567l, value);
region.put(put);
System.out.print(get value before flush after put value2 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
region.flushcache();
System.out.print(get value after flush after put value2 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
}
{code}
and the result print as followed:
{code}
get value before flush after put value0 : value0
get value after flush after put value0 : value0
get value before flush after put value1 : value1
get value after flush after put value1 : value0
get value before flush after put value2 : value2
get value after flush after put value2 : value0
{code}
I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners
are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap,
and the KeyValue for latest StoreFile will comapre to the second latest
StoreFile, but the second latest StoreFile generated the fake row for same row,
family, qualifier excepts the timestamp( maximum), memstoreTS(0). And the
latest KeyValue recognized as not latest than the second latest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
[jira] [Updated] (HBASE-6269) Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
[
https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ShiXing updated HBASE-6269:
---
Attachment: HBASE-6269-v1.patch
Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest
KeyValue
-
Key: HBASE-6269
URL: https://issues.apache.org/jira/browse/HBASE-6269
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.0
Reporter: ShiXing
Assignee: ShiXing
Attachments: HBASE-6269-v1.patch
When I fix the bug HBASE-6195, there is happened to find sometimes the test
case will fail, https://builds.apache.org/job/HBase-0.94/259/.
If there are two Put/Increment with same row, family, qualifier, timestamp
and different memstoreTS, after each Put/Increment, we do a memstore flush.
So there will be two StoreFile with same KeyValue(except memstoreTS and
SequenceId).
When I got the row, I always got the old records, the test case like this:
{code}
public void testPutWithMemStoreFlush() throws Exception {
Configuration conf = HBaseConfiguration.create();
String method = testPutWithMemStoreFlush;
byte[] tableName = Bytes.toBytes(method);
byte[] family = Bytes.toBytes(family);;
byte[] qualifier = Bytes.toBytes(qualifier);
byte[] row = Bytes.toBytes(putRow);
byte[] value = null;
this.region = initHRegion(tableName, method, conf, family);
Put put = null;
Get get = null;
ListKeyValue kvs = null;
Result res = null;
put = new Put(row);
value = Bytes.toBytes(value0);
put.add(family, qualifier, 1234567l, value);
region.put(put);
System.out.print(get value before flush after put value0 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
region.flushcache();
System.out.print(get value after flush after put value0 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
put = new Put(row);
value = Bytes.toBytes(value1);
put.add(family, qualifier, 1234567l, value);
region.put(put);
System.out.print(get value before flush after put value1 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
region.flushcache();
System.out.print(get value after flush after put value1 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
put = new Put(row);
value = Bytes.toBytes(value2);
put.add(family, qualifier, 1234567l, value);
region.put(put);
System.out.print(get value before flush after put value2 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
region.flushcache();
System.out.print(get value after flush after put value2 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
}
{code}
and the result print as followed:
{code}
get value before flush after put value0 : value0
get value after flush after put value0 : value0
get value before flush after put value1 : value1
get value after flush after put value1 : value0
get value before flush after put value2 : value2
get value after flush after put value2 : value0
{code}
I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners
are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap,
and the KeyValue for latest StoreFile will comapre to the second latest
StoreFile, but the second latest
[jira] [Commented] (HBASE-6195) Increment data will be lost when the memstore is flushed
[ https://issues.apache.org/jira/browse/HBASE-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13401267#comment-13401267 ] ShiXing commented on HBASE-6195: I find that the problem is introduced by the lazyseek. I have open a jira for this problem HBASE-6269. Increment data will be lost when the memstore is flushed Key: HBASE-6195 URL: https://issues.apache.org/jira/browse/HBASE-6195 Project: HBase Issue Type: Bug Components: regionserver Reporter: ShiXing Assignee: ShiXing Fix For: 0.96.0, 0.94.1 Attachments: 6195-trunk-V7.patch, 6195.addendum, HBASE-6195-trunk-V2.patch, HBASE-6195-trunk-V3.patch, HBASE-6195-trunk-V4.patch, HBASE-6195-trunk-V5.patch, HBASE-6195-trunk-V6.patch, HBASE-6195-trunk.patch There are two problems in increment() now: First: I see that the timestamp(the variable now) in HRegion's Increment() is generated before got the rowLock, so when there are multi-thread increment the same row, although it generate earlier, it may got the lock later. Because increment just store one version, so till now, the result will still be right. When the region is flushing, these increment will read the kv from snapshot and memstore with whose timestamp is larger, and write it back to memstore. If the snapshot's timestamp larger than the memstore, the increment will got the old data and then do the increment, it's wrong. Secondly: Also there is a risk in increment. Because it writes the memstore first and then HLog, so if it writes HLog failed, the client will also read the incremented value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6269) Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
[
https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13401383#comment-13401383
]
ShiXing commented on HBASE-6269:
@anoop
There are 2 StoreFiles after flush two times, the sf2's sequenceId sf1's
sequenceId.
When get:
step1. the sf2 is the highest StoreFileScanner, and it enforceSeek() in
KeyValueHeap.pollRealKV(), so the KeyValue2 is read out from StoreFile by real
seek. And it compares to the fake KeyValue(called FakeKeyValue) that generated
by KeyValue.createFirstOnRow() in StoreScanner.next(), and the FakeKeyValue's
row, family, qualifier, timestamp, memstoreTS(always 0 for StoreFileScanner)
are the same as KeyValue2 excepts Key type is Maximum, and Key type in
KeyValue2 is Put. And the {code}comparator.compare(curKV=KeyValue2,
nextKV=FakeKeyValue) = 251 0{code}. It means that the highest
StoreFileScanner's highest KeyValue is not higher than the second. Followed is
the value for example
{code}
KeyValue2 : putRow/family:qualifier/1234567/Put/vlen=6/ts=0
FakeKeyValue : putRow/family:qualifier/1234567/Maximum/vlen=0/ts=0
{code}
And then the second highest StoreFileScanner becomes the highest, and the
highest is added to the heap.
Step2. The sf1's highest KeyValue is read out , we call it KeyValue1, the real
value is the same as KeyValue2 fetched again by heap.peek():
{code}
KeyValue1 : putRow/family:qualifier/1234567/Put/vlen=6/ts=0
{code}
Step3. KeyValue1 compares KeyValue2, and the
{code}comparator.compare(curKV=KeyValue1, nextKV=KeyValue2) = 0{code}, and
return the sf1's scanner as the highest StoreFileScanner.
My solution is that:
If all the highest KeyValue read out from the StoreFileScanners are the
same(compare return 0), then we should keep the Scanners orig order by
sequenceId.
Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest
KeyValue
-
Key: HBASE-6269
URL: https://issues.apache.org/jira/browse/HBASE-6269
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.0
Reporter: ShiXing
Assignee: ShiXing
Attachments: HBASE-6269-v1.patch
When I fix the bug HBASE-6195, there is happened to find sometimes the test
case will fail, https://builds.apache.org/job/HBase-0.94/259/.
If there are two Put/Increment with same row, family, qualifier, timestamp
and different memstoreTS, after each Put/Increment, we do a memstore flush.
So there will be two StoreFile with same KeyValue(except memstoreTS and
SequenceId).
When I got the row, I always got the old records, the test case like this:
{code}
public void testPutWithMemStoreFlush() throws Exception {
Configuration conf = HBaseConfiguration.create();
String method = testPutWithMemStoreFlush;
byte[] tableName = Bytes.toBytes(method);
byte[] family = Bytes.toBytes(family);;
byte[] qualifier = Bytes.toBytes(qualifier);
byte[] row = Bytes.toBytes(putRow);
byte[] value = null;
this.region = initHRegion(tableName, method, conf, family);
Put put = null;
Get get = null;
ListKeyValue kvs = null;
Result res = null;
put = new Put(row);
value = Bytes.toBytes(value0);
put.add(family, qualifier, 1234567l, value);
region.put(put);
System.out.print(get value before flush after put value0 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
region.flushcache();
System.out.print(get value after flush after put value0 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
put = new Put(row);
value = Bytes.toBytes(value1);
put.add(family, qualifier, 1234567l, value);
region.put(put);
System.out.print(get value before flush after put value1 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
region.flushcache();
System.out.print(get value after flush after put value1 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs =
[jira] [Updated] (HBASE-6269) Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
[
https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ShiXing updated HBASE-6269:
---
Attachment: HBASE-6269-trunk-V1.patch
Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest
KeyValue
-
Key: HBASE-6269
URL: https://issues.apache.org/jira/browse/HBASE-6269
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.0
Reporter: ShiXing
Assignee: ShiXing
Attachments: HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch
When I fix the bug HBASE-6195, there is happened to find sometimes the test
case will fail, https://builds.apache.org/job/HBase-0.94/259/.
If there are two Put/Increment with same row, family, qualifier, timestamp
and different memstoreTS, after each Put/Increment, we do a memstore flush.
So there will be two StoreFile with same KeyValue(except memstoreTS and
SequenceId).
When I got the row, I always got the old records, the test case like this:
{code}
public void testPutWithMemStoreFlush() throws Exception {
Configuration conf = HBaseConfiguration.create();
String method = testPutWithMemStoreFlush;
byte[] tableName = Bytes.toBytes(method);
byte[] family = Bytes.toBytes(family);;
byte[] qualifier = Bytes.toBytes(qualifier);
byte[] row = Bytes.toBytes(putRow);
byte[] value = null;
this.region = initHRegion(tableName, method, conf, family);
Put put = null;
Get get = null;
ListKeyValue kvs = null;
Result res = null;
put = new Put(row);
value = Bytes.toBytes(value0);
put.add(family, qualifier, 1234567l, value);
region.put(put);
System.out.print(get value before flush after put value0 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
region.flushcache();
System.out.print(get value after flush after put value0 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
put = new Put(row);
value = Bytes.toBytes(value1);
put.add(family, qualifier, 1234567l, value);
region.put(put);
System.out.print(get value before flush after put value1 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
region.flushcache();
System.out.print(get value after flush after put value1 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
put = new Put(row);
value = Bytes.toBytes(value2);
put.add(family, qualifier, 1234567l, value);
region.put(put);
System.out.print(get value before flush after put value2 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
region.flushcache();
System.out.print(get value after flush after put value2 : );
get = new Get(row);
get.addColumn(family, qualifier);
get.setMaxVersions();
res = this.region.get(get, null);
kvs = res.getColumn(family, qualifier);
for (int i = 0; i kvs.size(); i++) {
System.out.println(Bytes.toString(kvs.get(i).getValue()));
}
}
{code}
and the result print as followed:
{code}
get value before flush after put value0 : value0
get value after flush after put value0 : value0
get value before flush after put value1 : value1
get value after flush after put value1 : value0
get value before flush after put value2 : value2
get value after flush after put value2 : value0
{code}
I analyze the code for StoreFileScanner with lazy seek, the StoreFileScanners
are sorted by SequenceId, so the latest StoreFile is on the top KeyValueHeap,
and the KeyValue for latest StoreFile will comapre to the second latest
