[
https://issues.apache.org/jira/browse/HBASE-21200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16639976#comment-16639976
]
Toshihiro Suzuki commented on HBASE-21200:
------------------------------------------
Thank you for reviewing [~ram_krish].
{quote}
So before this patch - the same test runs successfully is it? So the cells
greater than the readPt getting skipped inside the seek() call I believe.
{quote}
Yes, the same test runs successfully before applying this patch. it's just a
optimization for speed.
{quote}
Even there we were doing getIterator() and iterating through the elements and
comparing the seqId with the readPt.
So exactly how does the speed up happen here?
{quote}
Yes, we were doing getIterator() and iterating through the elements and
comparing the seqId with the readPt. In original logic in seekToPreviousRow(),
we seek row while going backward the row until finding right previous cell
whose seqId is less than or equal to readPt. When after opening scanner we put
a lot of cells whose seqId is greater than the readPt of the scanner to the
memstore, a lot of seeking row will happen as described in the following image
in HBASE-15871:
https://issues.apache.org/jira/secure/attachment/12805207/memstore_backwardSeek%28%29.PNG
This causes that seek() repeatedly search cells that are already searched, and
it's very slow.
In the optimization I added in the patch, we seek row only when the seqId of
any of the cells of the row is less than or equal to the readPt. We can do this
because if seqIds of all the cells are greater than the readPt, we won't find
right previous cell. So we can skip seeking row in this case.
Thanks.
> Memstore flush doesn't finish because of seekToPreviousRow() in memstore
> scanner.
> ---------------------------------------------------------------------------------
>
> Key: HBASE-21200
> URL: https://issues.apache.org/jira/browse/HBASE-21200
> Project: HBase
> Issue Type: Bug
> Components: Scanners
> Reporter: dongjin2193.jeon
> Assignee: Toshihiro Suzuki
> Priority: Major
> Attachments: HBASE-21200-UT.patch, HBASE-21200.master.001.patch,
> RegionServerJstack.log
>
>
> The issue of delaying memstore flush still occurs after backport hbase-15871.
> Reverse scan takes a long time to seek previous row in the memstore full of
> deleted cells.
>
> jstack :
> "MemStoreFlusher.0" #114 prio=5 os_prio=0 tid=0x00007fa3d0729000 nid=0x486a
> waiting on condition [0x00007fa3b9b6b000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000a465fe60> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
> at
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
> at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
> at
> org.apache.hadoop.hbase.regionserver.*StoreScanner.updateReaders(StoreScanner.java:695)*
> at
> org.apache.hadoop.hbase.regionserver.HStore.notifyChangedReadersObservers(HStore.java:1127)
> at
> org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1106)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$600(HStore.java:130)
> at
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2455)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2519)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2256)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2218)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2110)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:2036)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:501)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$800(MemStoreFlusher.java:75)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259)
> at java.lang.Thread.run(Thread.java:748)
>
> "RpcServer.FifoWFPBQ.default.handler=27,queue=0,port=16020" #65 daemon prio=5
> os_prio=0 tid=0x00007fa3e6280000 nid=0x4801 runnable [0x00007fa3bd29a000]
> java.lang.Thread.State: RUNNABLE
> at
> org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.getNext(DefaultMemStore.java:780)
> at
> org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.seekInSubLists(DefaultMemStore.java:826)
> - locked <0x00000000b45aa5b8> (a
> org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner)
> at
> org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.seek(DefaultMemStore.java:818)
> - locked <0x00000000b45aa5b8> (a
> org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner)
> at
> org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.seekToPreviousRow(DefaultMemStore.java:1000)
> - locked <0x00000000b45aa5b8> (a
> org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner)
> at
> org.apache.hadoop.hbase.regionserver.ReversedKeyValueHeap.next(ReversedKeyValueHeap.java:136)
> at
> org.apache.hadoop.hbase.regionserver.*StoreScanner.next(StoreScanner.java:629)*
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5876)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6027)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5814)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2552)
> - locked <0x000000009ee8ca10> (a
> org.apache.hadoop.hbase.regionserver.ReversedRegionScannerImpl)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32385)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
