So now the memstore flush includes data from other persisted store files? That is bad.
----- Original Message ----- From: ramkrishna vasudevan <ramkrishna.s.vasude...@gmail.com> To: "dev@hbase.apache.org" <dev@hbase.apache.org>; lars hofhansl <la...@apache.org> Cc: Sent: Thursday, January 30, 2014 9:52 PM Subject: Re: StoreScanner created for memstore flush should be bothered about updated readers? Yes there was concurrent compaction happening. This was the cause for the scanner reset and so finally ended up in seeking/next in the encoded block of those files under the storefilescanner. Adding the trace to show how a memstore flusher was trying to read a hfile. org.apache.hadoop.hbase.DroppedSnapshotException: region: usertable,user5152654437639860133,1391056599393.654e89edf63813d2120e9d287afff889. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1694) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1556) at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1471) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:456) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:430) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$800(MemStoreFlusher.java:66) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:248) at java.lang.Thread.run(Thread.java:744) Caused by: java.lang.IndexOutOfBoundsException: index (16161) must be less than size (7) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) at org.apache.hadoop.hbase.io.util.LRUDictionary$BidirectionalLRUMap.get(LRUDictionary.java:139) at org.apache.hadoop.hbase.io.util.LRUDictionary$BidirectionalLRUMap.access$000(LRUDictionary.java:76) at org.apache.hadoop.hbase.io.util.LRUDictionary.getEntry(LRUDictionary.java:43) at org.apache.hadoop.hbase.io.TagCompressionContext.uncompressTags(TagCompressionContext.java:159) at org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.decodeTags(BufferedDataBlockEncoder.java:273) at org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:522) at org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:540) at org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:262) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.next(HFileReaderV2.java:1063) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:137) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:108) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:509) at org.apache.hadoop.hbase.regionserver.StoreFlusher.performFlush(StoreFlusher.java:128) at org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:73) at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:786) at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:1943) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1669) On Fri, Jan 31, 2014 at 11:17 AM, lars hofhansl <la...@apache.org> wrote: > Interesting. Did you see the cause for the scanner reset? Was it a > concurrent compaction? > > > > ----- Original Message ----- > From: ramkrishna vasudevan <ramkrishna.s.vasude...@gmail.com> > To: "dev@hbase.apache.org" <dev@hbase.apache.org>; lars hofhansl < > la...@apache.org> > Cc: > Sent: Thursday, January 30, 2014 9:41 PM > Subject: Re: StoreScanner created for memstore flush should be bothered > about updated readers? > > >> The scanner stack is only reset if the set of HFiles for this store > changes, i.e. a compaction or a concurrent flush (when using multithreaded > flushing). It seems that would relatively rare. > In our test scenario this happens. While trying to find out the root cause > for HBASE-10443, hit this issue. It is not directly related to the flush > scenario but found this issue while debugging it. > I was not trying to improve the performance here, but the fact that we > updating the kv heap does make the flush to read those Hfiles on a > StoreScanner.next() call and it is expensive. > > Regards > Ram > > > > > > > On Fri, Jan 31, 2014 at 11:02 AM, lars hofhansl <la...@apache.org> wrote: > > > From what I found is that the main performance detriment comes from the > > fact that we need to take a lock for each next/peek call of the > > StoreScanner. Even when those are uncontended (which they are in 99.9% of > > the cases) the memory read/writes barriers are expensive. > > > > I doubt you'll see much improvement from this. The scanner stack is only > > reset if the set of HFiles for this store changes, i.e. a compaction or a > > concurrent flush (when using multithreaded flushing). It seems that would > > relatively rare. > > > > If anything we could a class like StoreScanner that does not need to > > synchronize any of its calls, but even there, the flush is asynchronous > to > > any user action (unless we're blocked on the number of store files, in > > which case there bigger problem anyway). > > > > > > Did you see a specific issue? > > > > -- Lars > > > > > > > > ----- Original Message ----- > > From: ramkrishna vasudevan <ramkrishna.s.vasude...@gmail.com> > > To: "dev@hbase.apache.org" <dev@hbase.apache.org> > > Cc: > > Sent: Thursday, January 30, 2014 11:48 AM > > Subject: StoreScanner created for memstore flush should be bothered about > > updated readers? > > > > Hi All > > > > In case of flush we create a memstore flusher which in turn creates a > > StoreScanner backed by a Single ton MemstoreScanner. > > > > But this scanner also registers for any updates in the reader in the > > HStore. Is this needed? > > If this happens then any update on the reader may nullify the current > heap > > and the entire Scanner Stack is reset, but this time with the other > > scanners for all the files that satisfies the last top key. So the flush > > that happens on the memstore holds the storefile scanners also in the > heap > > that was recreated but originally the intention was to create a scanner > on > > the memstore alone. > > > > Am i missing something here? Or what i observed is right? If so, then I > > feel that this step can be avoided. > > > > Regards > > Ram > > > > > >
