If there are too many store files, flush will wait till compaction clears them [1] or hbase.hstore.blockingWaitTime is over! Do you see similar logs?
[1] https://github.com/apache/hbase/blob/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java#L363 On Mon, Jul 14, 2014 at 9:27 AM, vito <[email protected]> wrote: > Hi , > > Will compaction(minor or major) block memstore flush to disk(or HDFS)? > That's, when region server is doing compaction, is it allowed to flush > memstore to HDFS? Let's base the answer in HBase 0.94. > > Here are some info I dig out, but I'm not sure of the anwser: > > hbase.hstore.compactionThreshold：（0.89） > > If more than this number of HStoreFiles in any one HStore (one > HStoreFile is written per flush of memstore) then a compaction is run to > rewrite all HStoreFiles files as one. Larger numbers put off compaction > but > when it runs, it takes longer to complete. *During a compaction, updates > cannot be flushed to disk. * Long compactions require memory sufficient to > carry the logging of all updates across the duration of the compaction. > If > too large, clients timeout during compaction. > > hbase.hstore.compactionThreshold：（0.94） > If more than this number of HStoreFiles in any one HStore (one HStoreFile > is > written per flush of memstore) then a compaction is run to rewrite all > HStoreFiles files as one. Larger numbers put off compaction but when it > runs, it takes longer to complete. > > > In the source code of 0.94, the comment says new storeFiles can be written > from memstore. > org.apache.hadoop.hbase.regionserver.Store.java:compact() > > > > ////////////////////////////////////////////////////////////////////////////// > // Compaction > > > ////////////////////////////////////////////////////////////////////////////// > > /** > * Compact the StoreFiles. This method may take some time, so the > calling > * thread must be able to block for long periods. > * > * <p>During this time, the Store can work as usual, getting values from > * StoreFiles and writing new StoreFiles from the memstore. > * > * Existing StoreFiles are not destroyed until the new compacted > StoreFile > is > * completely written-out to disk. > ... > > > Regards, > Vito Lee > > > > > -- > View this message in context: > http://apache-hbase.679495.n3.nabble.com/Will-compaction-block-memstore-flush-to-disk-tp4061302.html > Sent from the HBase User mailing list archive at Nabble.com. > -- Bharath Vissapragada <http://www.cloudera.com>
