Thank you very much. Actually I was trying to manage splitting manually, and to leave the compaction automatical. I really do not want to block my inserts by waiting for compactions. Is it possible to keep minor compaction working in the background, like starting a periodic job to do 'compact mytable' in the shell?
Regards, Sean 2011/7/7 Zhoushuaifeng <[email protected]> > hbase.hstore.blockingStoreFiles=Integer.MAX_VALUE > It's better not setting this to max_value. Setting this to a limited value > will help hbase to control the insert speed, and have enough time to finish > compact. Suggest to setting this to less than 20. > > hbase.hstore.max.filesize = 100G, do you mean hbase.hregion.max.filesize ? > This may be too large, if a store file too large, it will take very long > time to compact. I suggest to set this less than 8G. > > And the compaction queue size means how many regions need compaction, not > the number of files to compact. So, you have 10 regions, all need > compaction, and the compaction queue is 10. > > > Zhou Shuaifeng(Frank) > > -----Original Message----- > From: Sheng Chen [mailto:[email protected]] > Sent: Thursday, July 07, 2011 3:36 PM > To: [email protected] > Subject: Re: Is there a way to increase the compaction queue size? > > Thank you for your reply. > > No more minor compaction is going on, while there are still about 100 > HStoreFiles in each region. > I think it might because the compactionQueueSize has a maximum of 10 for a > region server (only one per region?), so after I stopped inserting and the > 10 compactions in the queue were finished, no more compaction is triggered. > > I think that "hbase.hstore.compaction.max" sets the max number of files for > one minor compaction, does it has anything to do with my problem? > > Thanks, > Sean > > 2011/7/7 Stack <[email protected]> > > > Try setting "hbase.hstore.compaction.max" to a value other than 10, its > > default. > > > > What do you mean by 'no further minor compaction is queuing'? There > > is no more compaction going on? Are there exceptions in the log? > > > > St.Ack > > > > On Wed, Jul 6, 2011 at 10:55 PM, Sheng Chen <[email protected]> > > wrote: > > > Hi, all, > > > I have a hbase region server of 10 regions and find the compaction > queue > > > size is always 10 at maximum. > > > After I finished inserting 1 billion records, there are still hundreds > of > > > HFiles in each region and no further minor compaction is queuing. > > > > > > My settings, > > > hbase.hstore.blockingStoreFiles=Integer.MAX_VALUE > > > hbase.hstore.compactionThreshold=3 > > > hbase.hstore.max.filesize=100G > > > > > > Is it the expected behavior? > > > Can I make the compaction queue larger in order to keep doing minor > > > compaction in the background? > > > > > > Thank you. > > > > > > Sean > > > > > >
