Hi In your case you have large compactions going on and at the same time heavy reads happening. Since there are lot of deletes the scan is spending sufficient time in file reads. Since compactions/flushes happens every now and then the readers are getting reset and that is causing the lock to be acquired and since there are multiple threads competing your scans suffer more because they are not able to reset themselves.
Yes - if the above said case is true for your scenario - then HBASE-13082 will help you out. Since it avoid scanners being reset on compactions and only one flushes and that too it is not a hard call to reset. If the scanner finds the boolean to be set then it resets if not the scan just goes on . Regards Ram On Fri, Jan 10, 2020 at 10:01 AM Kang Minwoo <minwoo.k...@outlook.com> wrote: > Thank you for reply. > > All Regions or just the one? > => just one > > Do thread dumps lock thread reading against hdfs every time you take one? > => yes > > Is it always inside in updateReaders? Is there a bad file or lots of files > to add to the list? > => always inside in updateReaders. > > ---- > > Sorry for the delay in reply. > > I had to handle this issue. > Temporarily, I fixed my code that does not occur in that situation that is > read worthlessness cell. > After that, The issue hasn't occurred. > > Background: > My application deletes out of date data every day. > And Region is extremely big. Major compaction spent a lot of time. > tombstone cell remains a long time. > If the client read full data. there is a lot of worthlessness cells. > I think it is a reason for lock thread reading hdfs files. > > I'm looking at the HBASE-13082[1]. > (I am not sure HBASE-13082 is related.) > > [1]: https://issues.apache.org/jira/browse/HBASE-13082 > > Best regards, > Minwoo Kang > > ________________________________________ > 보낸 사람: Stack <st...@duboce.net> > 보낸 날짜: 2020년 1월 4일 토요일 03:40 > 받는 사람: Hbase-User > 제목: Re: Extremely long flush times > > All Regions or just the one? > > Do thread dumps lock thread reading against hdfs every time you take one? > > Is it always inside in updateReaders? Is there a bad file or lots of files > to add to the list? > > Yours, > S > > > > On Thu, Jan 2, 2020 at 8:34 PM Kang Minwoo <minwoo.k...@outlook.com> > wrote: > > > Hello Users, > > > > I met an issue that is flush times is too long. > > > > MemStoreFlusher is waiting for a lock. > > ``` > > "MemStoreFlusher.0" > > java.lang.Thread.State: WAITING (parking) > > at sun.misc.Unsafe.park(Native Method) > > - parking to wait for <0x00007f0412bddcb8> (a > > java.util.concurrent.locks.ReentrantLock$NonfairSync) > > at > > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) > > at > > > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) > > at > > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) > > at > > > org.apache.hadoop.hbase.regionserver.StoreScanner.updateReaders(StoreScanner.java:692) > > at > > > org.apache.hadoop.hbase.regionserver.HStore.notifyChangedReadersObservers(HStore.java:1100) > > at > > > org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1079) > > at > > org.apache.hadoop.hbase.regionserver.HStore.access$700(HStore.java:118) > > at > > > org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2321) > > at > > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2430) > > at > > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2153) > > at > > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2115) > > at > > > org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2005) > > at > > org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1930) > > at > > > org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:514) > > at > > > org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:475) > > at > > > org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:75) > > at > > > org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:263) > > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > > - None > > ``` > > > > > > RPC Handler had the lock. > > ``` > > "B.defaultRpcServer.handler" > > java.lang.Thread.State: RUNNABLE > > at org.apache.log4j.Category.getEffectiveLevel(Category.java:442) > > at org.apache.log4j.Category.isEnabledFor(Category.java:751) > > at > > > org.apache.commons.logging.impl.Log4JLogger.isTraceEnabled(Log4JLogger.java:303) > > at > > org.apache.hadoop.hdfs.BlockReaderLocal.skip(BlockReaderLocal.java:622) > > - locked <0x00007f0271bcc1c0> (a > > org.apache.hadoop.hdfs.BlockReaderLocal) > > at > > org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1534) > > - locked <0x00007f038265f980> (a > > org.apache.hadoop.hdfs.DFSInputStream) > > at > > org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:62) > > at org.apache.hadoop.hbase.io > > .hfile.HFileUtil.seekOnMultipleSources(HFileUtil.java:38) > > at org.apache.hadoop.hbase.io > > .hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1431) > > at org.apache.hadoop.hbase.io > > > .hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1654) > > at org.apache.hadoop.hbase.io > > .hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1538) > > at org.apache.hadoop.hbase.io > > .hfile.HFileReaderV2.readBlock(HFileReaderV2.java:452) > > at org.apache.hadoop.hbase.io > > > .hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:729) > > at org.apache.hadoop.hbase.io > > .hfile.HFileReaderV2$ScannerV2.isNextBlock(HFileReaderV2.java:854) > > at org.apache.hadoop.hbase.io > > > .hfile.HFileReaderV2$ScannerV2.positionForNextBlock(HFileReaderV2.java:849) > > at org.apache.hadoop.hbase.io > > .hfile.HFileReaderV2$ScannerV2._next(HFileReaderV2.java:866) > > at org.apache.hadoop.hbase.io > > .hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:886) > > at > > > org.apache.hadoop.hbase.regionserver.StoreFileScanner.skipKVsNewerThanReadpoint(StoreFileScanner.java:247) > > at > > > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:194) > > at > > > org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:350) > > at > > > org.apache.hadoop.hbase.regionserver.StoreScanner.resetScannerStack(StoreScanner.java:750) > > at > > > org.apache.hadoop.hbase.regionserver.StoreScanner.checkReseek(StoreScanner.java:725) > > at > > > org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:475) > > at > > > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:150) > > at > > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5824) > > at > > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:5994) > > at > > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5761) > > at > > > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2622) > > - locked <0x00007f035a4df0f8> (a > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl) > > at > > > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33770) > > at > org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2216) > > at > org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) > > at > > > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) > > at > > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) > > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > > - <0x00007f02bfaf7a90> (a > > java.util.concurrent.locks.ReentrantLock$NonfairSync) > > - <0x00007f0412bddcb8> (a > > java.util.concurrent.locks.ReentrantLock$NonfairSync) > > ``` > > > > ``` > > B.defaultRpcServer.handler > > java.lang.Thread.State: RUNNABLE > > at java.io.FilterInputStream.read(FilterInputStream.java:83) > > at > > > org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:145) > > at > > > org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:74) > > at > > > org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85) > > at > java.io.BufferedInputStream.read1(BufferedInputStream.java:284) > > at java.io.BufferedInputStream.read(BufferedInputStream.java:345) > > - locked <0x00007f024acf0bf8> (a java.io.BufferedInputStream) > > at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:200) > > at org.apache.hadoop.hbase.io > > .compress.Compression.decompress(Compression.java:453) > > at org.apache.hadoop.hbase.io > > > .encoding.HFileBlockDefaultDecodingContext.prepareDecoding(HFileBlockDefaultDecodingContext.java:90) > > at org.apache.hadoop.hbase.io > > .hfile.HFileBlock.unpack(HFileBlock.java:549) > > at org.apache.hadoop.hbase.io > > .hfile.HFileReaderV2.readBlock(HFileReaderV2.java:455) > > at org.apache.hadoop.hbase.io > > > .hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:729) > > at org.apache.hadoop.hbase.io > > .hfile.HFileReaderV2$ScannerV2.isNextBlock(HFileReaderV2.java:854) > > at org.apache.hadoop.hbase.io > > > .hfile.HFileReaderV2$ScannerV2.positionForNextBlock(HFileReaderV2.java:849) > > at org.apache.hadoop.hbase.io > > .hfile.HFileReaderV2$ScannerV2._next(HFileReaderV2.java:866) > > at org.apache.hadoop.hbase.io > > .hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:886) > > at > > > org.apache.hadoop.hbase.regionserver.StoreFileScanner.skipKVsNewerThanReadpoint(StoreFileScanner.java:247) > > at > > > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:194) > > at > > > org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:350) > > at > > > org.apache.hadoop.hbase.regionserver.StoreScanner.resetScannerStack(StoreScanner.java:750) > > at > > > org.apache.hadoop.hbase.regionserver.StoreScanner.checkReseek(StoreScanner.java:725) > > at > > > org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:475) > > at > > > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:150) > > at > > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5824) > > at > > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:5994) > > at > > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5761) > > at > > > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2622) > > - locked <0x00007f035a4df0f8> (a > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl) > > at > > > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33770) > > at > org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2216) > > at > org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) > > at > > > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) > > at > > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) > > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > > - <0x00007f0412bddcb8> (a > > java.util.concurrent.locks.ReentrantLock$NonfairSync) > > ``` > > > > Versions: > > - HBase: 1.2.9 > > - HDFS: 2.7.7 > > > > Best regards, > > Minwoo Kang > > >
