[
https://issues.apache.org/jira/browse/HBASE-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398521#comment-13398521
]
Xing Shi commented on HBASE-6195:
---------------------------------
@Ted, It's not because of the snapshot, but the store files.
When there are serveral same timestamp storefiles, the get will get the row of
one of the store files' KeyValue. You can think that the got KeyValue is
almost random between the same timestamp storefiles.
How to improve it:
I make all the timestamps (variable now) of the increment the same value to
simulate there are high parallelism. Then the flushcache after 2 increments
and 5 increments, the code like this just in one threads:
{code}
public void runInOneThread() {
int count = 0;
while (count < 10) {
Increment inc = new Increment(incRow);
inc.addColumn(family, qualifier, ONE);
count++;
try {
region.increment(inc, null, true);
Get get = new Get(Incrementer.incRow);
get.addColumn(Incrementer.family, Incrementer.qualifier);
get.setMaxVersions();
List<KeyValue> kvs = this.region.get(get, false);
for (KeyValue tmpKv : kvs) {
System.out.println("get return : " +
Bytes.toLong(tmpKv.getBuffer(), tmpKv.getValueOffset()) + " after " + count + "
increment");
}
if (count == 2) {
System.out.println("We do flush when execute " + count + "
increments");
region.flushcache();
} else if (count == 5) {
System.out.println("We do flush when execute " + count + "
increments");
region.flushcache();
}
} catch (Exception e) {
e.printStackTrace();
break;
}
}
}
{code}
the print out result is :
{code}
get return : 1 after 1 increment
get return : 2 after 2 increment
We do flush when execute 2 increments
2012-06-21 23:16:32,847 DEBUG [Thread-3] regionserver.HRegion(1367): Started
memstore flush for
testParallelismIncrementWithMemStoreFlush,,1340291792603.b2a89da7ad8457cab8a1c758c32ed418.,
current region memstore size 176.0
2012-06-21 23:16:32,848 DEBUG [Thread-3] regionserver.HRegion(1415): Finished
snapshotting
testParallelismIncrementWithMemStoreFlush,,1340291792603.b2a89da7ad8457cab8a1c758c32ed418.,
commencing wait for mvcc, flushsize=176
2012-06-21 23:16:32,849 DEBUG [Thread-3] regionserver.HRegion(1425): Finished
snapshotting, commencing flushing stores
2012-06-21 23:16:32,859 DEBUG [Thread-3] util.FSUtils(149): Creating
file:/home/shixing/project/0.94.0-ali-1.0/target/test-data/19e96a0f-6721-4fe7-8ede-f5ca359739a0/TestHRegiontestParallelismIncrementWithMemStoreFlush/testParallelismIncrementWithMemStoreFlush/b2a89da7ad8457cab8a1c758c32ed418/.tmp/4382d83027cc48b6b9a3f411157d0749with
permission:rwxrwxrwx
2012-06-21 23:16:32,871 DEBUG [Thread-3] hfile.HFileWriterV2(141): Initialized
with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false]
[cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false]
[cacheEvictOnClose=false] [cacheCompressed=false]
2012-06-21 23:16:32,874 INFO [Thread-3] regionserver.StoreFile$Writer(995):
Delete Family Bloom filter type for
/home/shixing/project/0.94.0-ali-1.0/target/test-data/19e96a0f-6721-4fe7-8ede-f5ca359739a0/TestHRegiontestParallelismIncrementWithMemStoreFlush/testParallelismIncrementWithMemStoreFlush/b2a89da7ad8457cab8a1c758c32ed418/.tmp/4382d83027cc48b6b9a3f411157d0749:
CompoundBloomFilterWriter
2012-06-21 23:16:32,880 INFO [Thread-3] regionserver.StoreFile$Writer(1218):
NO General Bloom and NO DeleteFamily was added to HFile
(/home/shixing/project/0.94.0-ali-1.0/target/test-data/19e96a0f-6721-4fe7-8ede-f5ca359739a0/TestHRegiontestParallelismIncrementWithMemStoreFlush/testParallelismIncrementWithMemStoreFlush/b2a89da7ad8457cab8a1c758c32ed418/.tmp/4382d83027cc48b6b9a3f411157d0749)
2012-06-21 23:16:32,880 INFO [Thread-3] regionserver.Store(753): Flushed ,
sequenceid=3, memsize=176.0, into tmp file
/home/shixing/project/0.94.0-ali-1.0/target/test-data/19e96a0f-6721-4fe7-8ede-f5ca359739a0/TestHRegiontestParallelismIncrementWithMemStoreFlush/testParallelismIncrementWithMemStoreFlush/b2a89da7ad8457cab8a1c758c32ed418/.tmp/4382d83027cc48b6b9a3f411157d0749
2012-06-21 23:16:32,905 DEBUG [Thread-3] regionserver.Store(778): Renaming
flushed file at
/home/shixing/project/0.94.0-ali-1.0/target/test-data/19e96a0f-6721-4fe7-8ede-f5ca359739a0/TestHRegiontestParallelismIncrementWithMemStoreFlush/testParallelismIncrementWithMemStoreFlush/b2a89da7ad8457cab8a1c758c32ed418/.tmp/4382d83027cc48b6b9a3f411157d0749
to
/home/shixing/project/0.94.0-ali-1.0/target/test-data/19e96a0f-6721-4fe7-8ede-f5ca359739a0/TestHRegiontestParallelismIncrementWithMemStoreFlush/testParallelismIncrementWithMemStoreFlush/b2a89da7ad8457cab8a1c758c32ed418/family/4382d83027cc48b6b9a3f411157d0749
2012-06-21 23:16:32,909 INFO [Thread-3] regionserver.Store(801): Added
/home/shixing/project/0.94.0-ali-1.0/target/test-data/19e96a0f-6721-4fe7-8ede-f5ca359739a0/TestHRegiontestParallelismIncrementWithMemStoreFlush/testParallelismIncrementWithMemStoreFlush/b2a89da7ad8457cab8a1c758c32ed418/family/4382d83027cc48b6b9a3f411157d0749,
entries=1, sequenceid=3, filesize=788.0
2012-06-21 23:16:32,910 INFO [Thread-3] regionserver.HRegion(1499): Finished
memstore flush of ~176.0/176, currentsize=0.0/0 for region
testParallelismIncrementWithMemStoreFlush,,1340291792603.b2a89da7ad8457cab8a1c758c32ed418.
in 63ms, sequenceid=3, compaction requested=false
get return : 2 after 3 increment
get return : 2 after 4 increment
get return : 2 after 5 increment
We do flush when execute 5 increments
2012-06-21 23:16:32,922 DEBUG [Thread-3] regionserver.HRegion(1367): Started
memstore flush for
testParallelismIncrementWithMemStoreFlush,,1340291792603.b2a89da7ad8457cab8a1c758c32ed418.,
current region memstore size 176.0
2012-06-21 23:16:32,923 DEBUG [Thread-3] regionserver.HRegion(1415): Finished
snapshotting
testParallelismIncrementWithMemStoreFlush,,1340291792603.b2a89da7ad8457cab8a1c758c32ed418.,
commencing wait for mvcc, flushsize=176
2012-06-21 23:16:32,923 DEBUG [Thread-3] regionserver.HRegion(1425): Finished
snapshotting, commencing flushing stores
2012-06-21 23:16:32,923 DEBUG [Thread-3] util.FSUtils(149): Creating
file:/home/shixing/project/0.94.0-ali-1.0/target/test-data/19e96a0f-6721-4fe7-8ede-f5ca359739a0/TestHRegiontestParallelismIncrementWithMemStoreFlush/testParallelismIncrementWithMemStoreFlush/b2a89da7ad8457cab8a1c758c32ed418/.tmp/a1313e3239e840c7ac9fa9f25387b59fwith
permission:rwxrwxrwx
2012-06-21 23:16:32,926 DEBUG [Thread-3] hfile.HFileWriterV2(141): Initialized
with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false]
[cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false]
[cacheEvictOnClose=false] [cacheCompressed=false]
2012-06-21 23:16:32,926 INFO [Thread-3] regionserver.StoreFile$Writer(995):
Delete Family Bloom filter type for
/home/shixing/project/0.94.0-ali-1.0/target/test-data/19e96a0f-6721-4fe7-8ede-f5ca359739a0/TestHRegiontestParallelismIncrementWithMemStoreFlush/testParallelismIncrementWithMemStoreFlush/b2a89da7ad8457cab8a1c758c32ed418/.tmp/a1313e3239e840c7ac9fa9f25387b59f:
CompoundBloomFilterWriter
2012-06-21 23:16:32,927 INFO [Thread-3] regionserver.StoreFile$Writer(1218):
NO General Bloom and NO DeleteFamily was added to HFile
(/home/shixing/project/0.94.0-ali-1.0/target/test-data/19e96a0f-6721-4fe7-8ede-f5ca359739a0/TestHRegiontestParallelismIncrementWithMemStoreFlush/testParallelismIncrementWithMemStoreFlush/b2a89da7ad8457cab8a1c758c32ed418/.tmp/a1313e3239e840c7ac9fa9f25387b59f)
2012-06-21 23:16:32,927 INFO [Thread-3] regionserver.Store(753): Flushed ,
sequenceid=7, memsize=176.0, into tmp file
/home/shixing/project/0.94.0-ali-1.0/target/test-data/19e96a0f-6721-4fe7-8ede-f5ca359739a0/TestHRegiontestParallelismIncrementWithMemStoreFlush/testParallelismIncrementWithMemStoreFlush/b2a89da7ad8457cab8a1c758c32ed418/.tmp/a1313e3239e840c7ac9fa9f25387b59f
2012-06-21 23:16:32,929 DEBUG [Thread-3] regionserver.Store(778): Renaming
flushed file at
/home/shixing/project/0.94.0-ali-1.0/target/test-data/19e96a0f-6721-4fe7-8ede-f5ca359739a0/TestHRegiontestParallelismIncrementWithMemStoreFlush/testParallelismIncrementWithMemStoreFlush/b2a89da7ad8457cab8a1c758c32ed418/.tmp/a1313e3239e840c7ac9fa9f25387b59f
to
/home/shixing/project/0.94.0-ali-1.0/target/test-data/19e96a0f-6721-4fe7-8ede-f5ca359739a0/TestHRegiontestParallelismIncrementWithMemStoreFlush/testParallelismIncrementWithMemStoreFlush/b2a89da7ad8457cab8a1c758c32ed418/family/a1313e3239e840c7ac9fa9f25387b59f
2012-06-21 23:16:32,932 INFO [Thread-3] regionserver.Store(801): Added
/home/shixing/project/0.94.0-ali-1.0/target/test-data/19e96a0f-6721-4fe7-8ede-f5ca359739a0/TestHRegiontestParallelismIncrementWithMemStoreFlush/testParallelismIncrementWithMemStoreFlush/b2a89da7ad8457cab8a1c758c32ed418/family/a1313e3239e840c7ac9fa9f25387b59f,
entries=1, sequenceid=7, filesize=788.0
2012-06-21 23:16:32,933 INFO [Thread-3] regionserver.HRegion(1499): Finished
memstore flush of ~176.0/176, currentsize=0.0/0 for region
testParallelismIncrementWithMemStoreFlush,,1340291792603.b2a89da7ad8457cab8a1c758c32ed418.
in 11ms, sequenceid=7, compaction requested=false
get return : 2 after 6 increment
get return : 2 after 7 increment
get return : 2 after 8 increment
get return : 2 after 9 increment
get return : 2 after 10 increment
result : 2
{code}
In memstore, the value is 3, but the Get doesn't get it.
I will also don't know which KeyValue to choose if they have the same
timestamp, like this:
{code}
/ | \
/ | \
/ | \
/ | \
/ | \
__________ __________ __________
|r1|t1|v1| |r1|t1|v2| |r1|t1|v3|
{code}
v1 or v2 or v3?
In real scene, the memstore flush will not happend so quickly, and there will
be almost no same timestamp KeyValue in increment.
The different between Put and Increment is that the memstoreTS, in Increment
the memstoreTS are always 0, and in Put ,the memstoreTS progressively increases.
> Increment data will be lost when the memstore is flushed
> --------------------------------------------------------
>
> Key: HBASE-6195
> URL: https://issues.apache.org/jira/browse/HBASE-6195
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Xing Shi
> Assignee: ShiXing
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 6195-trunk-V7.patch, 6195.addendum,
> HBASE-6195-trunk-V2.patch, HBASE-6195-trunk-V3.patch,
> HBASE-6195-trunk-V4.patch, HBASE-6195-trunk-V5.patch,
> HBASE-6195-trunk-V6.patch, HBASE-6195-trunk.patch
>
>
> There are two problems in increment() now:
> First:
> I see that the timestamp(the variable now) in HRegion's Increment() is
> generated before got the rowLock, so when there are multi-thread increment
> the same row, although it generate earlier, it may got the lock later.
> Because increment just store one version, so till now, the result will still
> be right.
> When the region is flushing, these increment will read the kv from snapshot
> and memstore with whose timestamp is larger, and write it back to memstore.
> If the snapshot's timestamp larger than the memstore, the increment will got
> the old data and then do the increment, it's wrong.
> Secondly:
> Also there is a risk in increment. Because it writes the memstore first and
> then HLog, so if it writes HLog failed, the client will also read the
> incremented value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
