Hi Minwoo Kang Any updates here? Where you able to over come the issue with the upgrade? Or by applying the patch?
Regards Ram On Fri, Jan 10, 2020 at 11:44 AM Kang Minwoo <minwoo.k...@outlook.com> wrote: > Thanks for the reply. > It is a lot of help to me. > > Best regards, > Minwoo Kang > > ________________________________________ > 보낸 사람: ramkrishna vasudevan <ramkrishna.s.vasude...@gmail.com> > 보낸 날짜: 2020년 1월 10일 금요일 14:35 > 받는 사람: Hbase-User > 참조: Stack > 제목: Re: Extremely long flush times > > Hi > > In your case you have large compactions going on and at the same time heavy > reads happening. Since there are lot of deletes the scan is spending > sufficient time in file reads. > Since compactions/flushes happens every now and then the readers are > getting reset and that is causing the lock to be acquired and since there > are multiple threads competing your scans suffer more because they are not > able to reset themselves. > > Yes - if the above said case is true for your scenario - then HBASE-13082 > will help you out. Since it avoid scanners being reset on compactions and > only one flushes and that too it is not a hard call to reset. If the > scanner finds the boolean to be set then it resets if not the scan just > goes on . > > Regards > Ram > > On Fri, Jan 10, 2020 at 10:01 AM Kang Minwoo <minwoo.k...@outlook.com> > wrote: > > > Thank you for reply. > > > > All Regions or just the one? > > => just one > > > > Do thread dumps lock thread reading against hdfs every time you take one? > > => yes > > > > Is it always inside in updateReaders? Is there a bad file or lots of > files > > to add to the list? > > => always inside in updateReaders. > > > > ---- > > > > Sorry for the delay in reply. > > > > I had to handle this issue. > > Temporarily, I fixed my code that does not occur in that situation that > is > > read worthlessness cell. > > After that, The issue hasn't occurred. > > > > Background: > > My application deletes out of date data every day. > > And Region is extremely big. Major compaction spent a lot of time. > > tombstone cell remains a long time. > > If the client read full data. there is a lot of worthlessness cells. > > I think it is a reason for lock thread reading hdfs files. > > > > I'm looking at the HBASE-13082[1]. > > (I am not sure HBASE-13082 is related.) > > > > [1]: https://issues.apache.org/jira/browse/HBASE-13082 > > > > Best regards, > > Minwoo Kang > > > > ________________________________________ > > 보낸 사람: Stack <st...@duboce.net> > > 보낸 날짜: 2020년 1월 4일 토요일 03:40 > > 받는 사람: Hbase-User > > 제목: Re: Extremely long flush times > > > > All Regions or just the one? > > > > Do thread dumps lock thread reading against hdfs every time you take one? > > > > Is it always inside in updateReaders? Is there a bad file or lots of > files > > to add to the list? > > > > Yours, > > S > > > > > > > > On Thu, Jan 2, 2020 at 8:34 PM Kang Minwoo <minwoo.k...@outlook.com> > > wrote: > > > > > Hello Users, > > > > > > I met an issue that is flush times is too long. > > > > > > MemStoreFlusher is waiting for a lock. > > > ``` > > > "MemStoreFlusher.0" > > > java.lang.Thread.State: WAITING (parking) > > > at sun.misc.Unsafe.park(Native Method) > > > - parking to wait for <0x00007f0412bddcb8> (a > > > java.util.concurrent.locks.ReentrantLock$NonfairSync) > > > at > > > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > > > at > > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) > > > at > > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) > > > at > > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) > > > at > > > > > > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) > > > at > > > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.StoreScanner.updateReaders(StoreScanner.java:692) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.HStore.notifyChangedReadersObservers(HStore.java:1100) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1079) > > > at > > > org.apache.hadoop.hbase.regionserver.HStore.access$700(HStore.java:118) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2321) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2430) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2153) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2115) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2005) > > > at > > > org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1930) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:514) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:475) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:75) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:263) > > > at java.lang.Thread.run(Thread.java:748) > > > Locked ownable synchronizers: > > > - None > > > ``` > > > > > > > > > RPC Handler had the lock. > > > ``` > > > "B.defaultRpcServer.handler" > > > java.lang.Thread.State: RUNNABLE > > > at > org.apache.log4j.Category.getEffectiveLevel(Category.java:442) > > > at org.apache.log4j.Category.isEnabledFor(Category.java:751) > > > at > > > > > > org.apache.commons.logging.impl.Log4JLogger.isTraceEnabled(Log4JLogger.java:303) > > > at > > > org.apache.hadoop.hdfs.BlockReaderLocal.skip(BlockReaderLocal.java:622) > > > - locked <0x00007f0271bcc1c0> (a > > > org.apache.hadoop.hdfs.BlockReaderLocal) > > > at > > > org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1534) > > > - locked <0x00007f038265f980> (a > > > org.apache.hadoop.hdfs.DFSInputStream) > > > at > > > org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:62) > > > at org.apache.hadoop.hbase.io > > > .hfile.HFileUtil.seekOnMultipleSources(HFileUtil.java:38) > > > at org.apache.hadoop.hbase.io > > > .hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1431) > > > at org.apache.hadoop.hbase.io > > > > > > .hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1654) > > > at org.apache.hadoop.hbase.io > > > .hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1538) > > > at org.apache.hadoop.hbase.io > > > .hfile.HFileReaderV2.readBlock(HFileReaderV2.java:452) > > > at org.apache.hadoop.hbase.io > > > > > > .hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:729) > > > at org.apache.hadoop.hbase.io > > > .hfile.HFileReaderV2$ScannerV2.isNextBlock(HFileReaderV2.java:854) > > > at org.apache.hadoop.hbase.io > > > > > > .hfile.HFileReaderV2$ScannerV2.positionForNextBlock(HFileReaderV2.java:849) > > > at org.apache.hadoop.hbase.io > > > .hfile.HFileReaderV2$ScannerV2._next(HFileReaderV2.java:866) > > > at org.apache.hadoop.hbase.io > > > .hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:886) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.StoreFileScanner.skipKVsNewerThanReadpoint(StoreFileScanner.java:247) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:194) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:350) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.StoreScanner.resetScannerStack(StoreScanner.java:750) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.StoreScanner.checkReseek(StoreScanner.java:725) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:475) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:150) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5824) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:5994) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5761) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2622) > > > - locked <0x00007f035a4df0f8> (a > > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl) > > > at > > > > > > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33770) > > > at > > org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2216) > > > at > > org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) > > > at > > > > > > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) > > > at > > > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) > > > at java.lang.Thread.run(Thread.java:748) > > > Locked ownable synchronizers: > > > - <0x00007f02bfaf7a90> (a > > > java.util.concurrent.locks.ReentrantLock$NonfairSync) > > > - <0x00007f0412bddcb8> (a > > > java.util.concurrent.locks.ReentrantLock$NonfairSync) > > > ``` > > > > > > ``` > > > B.defaultRpcServer.handler > > > java.lang.Thread.State: RUNNABLE > > > at java.io.FilterInputStream.read(FilterInputStream.java:83) > > > at > > > > > > org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:145) > > > at > > > > > > org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:74) > > > at > > > > > > org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85) > > > at > > java.io.BufferedInputStream.read1(BufferedInputStream.java:284) > > > at > java.io.BufferedInputStream.read(BufferedInputStream.java:345) > > > - locked <0x00007f024acf0bf8> (a java.io.BufferedInputStream) > > > at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:200) > > > at org.apache.hadoop.hbase.io > > > .compress.Compression.decompress(Compression.java:453) > > > at org.apache.hadoop.hbase.io > > > > > > .encoding.HFileBlockDefaultDecodingContext.prepareDecoding(HFileBlockDefaultDecodingContext.java:90) > > > at org.apache.hadoop.hbase.io > > > .hfile.HFileBlock.unpack(HFileBlock.java:549) > > > at org.apache.hadoop.hbase.io > > > .hfile.HFileReaderV2.readBlock(HFileReaderV2.java:455) > > > at org.apache.hadoop.hbase.io > > > > > > .hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:729) > > > at org.apache.hadoop.hbase.io > > > .hfile.HFileReaderV2$ScannerV2.isNextBlock(HFileReaderV2.java:854) > > > at org.apache.hadoop.hbase.io > > > > > > .hfile.HFileReaderV2$ScannerV2.positionForNextBlock(HFileReaderV2.java:849) > > > at org.apache.hadoop.hbase.io > > > .hfile.HFileReaderV2$ScannerV2._next(HFileReaderV2.java:866) > > > at org.apache.hadoop.hbase.io > > > .hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:886) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.StoreFileScanner.skipKVsNewerThanReadpoint(StoreFileScanner.java:247) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:194) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:350) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.StoreScanner.resetScannerStack(StoreScanner.java:750) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.StoreScanner.checkReseek(StoreScanner.java:725) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:475) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:150) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5824) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:5994) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5761) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2622) > > > - locked <0x00007f035a4df0f8> (a > > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl) > > > at > > > > > > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33770) > > > at > > org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2216) > > > at > > org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) > > > at > > > > > > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) > > > at > > > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) > > > at java.lang.Thread.run(Thread.java:748) > > > Locked ownable synchronizers: > > > - <0x00007f0412bddcb8> (a > > > java.util.concurrent.locks.ReentrantLock$NonfairSync) > > > ``` > > > > > > Versions: > > > - HBase: 1.2.9 > > > - HDFS: 2.7.7 > > > > > > Best regards, > > > Minwoo Kang > > > > > >
