[
https://issues.apache.org/jira/browse/HBASE-21200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16646158#comment-16646158
]
Toshihiro Suzuki edited comment on HBASE-21200 at 10/11/18 8:59 AM:
--------------------------------------------------------------------
I just attached v2 patch. It's a different approach from v1 patch.
In the current updateCurrent(), it skips the cells with irrelevant MVCC until
the row of "next" less than the row "startKV"("current"):
{code}
protected void updateCurrent() {
Cell startKV = current;
Cell next = null;
try {
while (iter.hasNext()) {
next = iter.next();
if (next.getSequenceId() <= this.readPoint) {
current = next;
return;// skip irrelevant versions
}
if (stopSkippingKVsIfNextRow && // for backwardSeek() stay in the
startKV != null && // boundaries of a single row
segment.compareRows(next, startKV) > 0) {
current = null;
return;
}
} // end of while
current = null; // nothing found
} finally {
if (next != null) {
// in all cases, remember the last KV we iterated to, needed for
reseek()
last = next;
}
}
}
{code}
I don't think it's correct because the intention of the logic is to stay in the
boundaries of a single row as the comment in the code. The "current" is not
always the previous row, so it can skip the cells with irrelevant MVCC for
multiple rows.
It looks like this causes the repeatedly search for cells that are already
searched, described in the following image in HBASE-15871:
https://issues.apache.org/jira/secure/attachment/12805207/memstore_backwardSeek%28%29.PNG
The v2 patch takes care of this issue. Could you please take a look at the
patch? [~ram_krish]
was (Author: brfrn169):
I just attached v2 patch. It's a different approach from v1 patch.
In the current updateCurrent(), it skips the cells with irrelevant MVCC until
the row of "next" less than the row "startKV"("current"):
{code}
protected void updateCurrent() {
Cell startKV = current;
Cell next = null;
try {
while (iter.hasNext()) {
next = iter.next();
if (next.getSequenceId() <= this.readPoint) {
current = next;
return;// skip irrelevant versions
}
if (stopSkippingKVsIfNextRow && // for backwardSeek() stay in the
startKV != null && // boundaries of a single row
segment.compareRows(next, startKV) > 0) {
current = null;
return;
}
} // end of while
current = null; // nothing found
} finally {
if (next != null) {
// in all cases, remember the last KV we iterated to, needed for
reseek()
last = next;
}
}
}
{code}
I don't think it's correct because the intention of the logic is to stay in the
boundaries of a single row as the comment in the code. The "current" is not
always the previous row, so it can skip the cells with irrelevant MVCC for
multiple rows.
It looks like this causes the repeatedly search for cells that are already
searched, described in the following image in HBASE-15871:
https://issues.apache.org/jira/secure/attachment/12805207/memstore_backwardSeek%28%29.PNG
The v2 patch takes case of this issue. Could you please take a look at the
patch? [~ram_krish]
> Memstore flush doesn't finish because of seekToPreviousRow() in memstore
> scanner.
> ---------------------------------------------------------------------------------
>
> Key: HBASE-21200
> URL: https://issues.apache.org/jira/browse/HBASE-21200
> Project: HBase
> Issue Type: Bug
> Components: Scanners
> Reporter: dongjin2193.jeon
> Assignee: Toshihiro Suzuki
> Priority: Major
> Attachments: HBASE-21200-UT.patch, HBASE-21200.master.001.patch,
> HBASE-21200.master.002.patch, RegionServerJstack.log
>
>
> The issue of delaying memstore flush still occurs after backport hbase-15871.
> Reverse scan takes a long time to seek previous row in the memstore full of
> deleted cells.
>
> jstack :
> "MemStoreFlusher.0" #114 prio=5 os_prio=0 tid=0x00007fa3d0729000 nid=0x486a
> waiting on condition [0x00007fa3b9b6b000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000a465fe60> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
> at
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
> at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
> at
> org.apache.hadoop.hbase.regionserver.*StoreScanner.updateReaders(StoreScanner.java:695)*
> at
> org.apache.hadoop.hbase.regionserver.HStore.notifyChangedReadersObservers(HStore.java:1127)
> at
> org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1106)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$600(HStore.java:130)
> at
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2455)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2519)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2256)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2218)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2110)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:2036)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:501)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$800(MemStoreFlusher.java:75)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259)
> at java.lang.Thread.run(Thread.java:748)
>
> "RpcServer.FifoWFPBQ.default.handler=27,queue=0,port=16020" #65 daemon prio=5
> os_prio=0 tid=0x00007fa3e6280000 nid=0x4801 runnable [0x00007fa3bd29a000]
> java.lang.Thread.State: RUNNABLE
> at
> org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.getNext(DefaultMemStore.java:780)
> at
> org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.seekInSubLists(DefaultMemStore.java:826)
> - locked <0x00000000b45aa5b8> (a
> org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner)
> at
> org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.seek(DefaultMemStore.java:818)
> - locked <0x00000000b45aa5b8> (a
> org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner)
> at
> org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.seekToPreviousRow(DefaultMemStore.java:1000)
> - locked <0x00000000b45aa5b8> (a
> org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner)
> at
> org.apache.hadoop.hbase.regionserver.ReversedKeyValueHeap.next(ReversedKeyValueHeap.java:136)
> at
> org.apache.hadoop.hbase.regionserver.*StoreScanner.next(StoreScanner.java:629)*
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5876)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6027)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5814)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2552)
> - locked <0x000000009ee8ca10> (a
> org.apache.hadoop.hbase.regionserver.ReversedRegionScannerImpl)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32385)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
