hbase.hstore.blockingStoreFiles=Integer.MAX_VALUE It's better not setting this to max_value. Setting this to a limited value will help hbase to control the insert speed, and have enough time to finish compact. Suggest to setting this to less than 20.
hbase.hstore.max.filesize = 100G, do you mean hbase.hregion.max.filesize ? This may be too large, if a store file too large, it will take very long time to compact. I suggest to set this less than 8G. And the compaction queue size means how many regions need compaction, not the number of files to compact. So, you have 10 regions, all need compaction, and the compaction queue is 10. Zhou Shuaifeng(Frank) -----Original Message----- From: Sheng Chen [mailto:[email protected]] Sent: Thursday, July 07, 2011 3:36 PM To: [email protected] Subject: Re: Is there a way to increase the compaction queue size? Thank you for your reply. No more minor compaction is going on, while there are still about 100 HStoreFiles in each region. I think it might because the compactionQueueSize has a maximum of 10 for a region server (only one per region?), so after I stopped inserting and the 10 compactions in the queue were finished, no more compaction is triggered. I think that "hbase.hstore.compaction.max" sets the max number of files for one minor compaction, does it has anything to do with my problem? Thanks, Sean 2011/7/7 Stack <[email protected]> > Try setting "hbase.hstore.compaction.max" to a value other than 10, its > default. > > What do you mean by 'no further minor compaction is queuing'? There > is no more compaction going on? Are there exceptions in the log? > > St.Ack > > On Wed, Jul 6, 2011 at 10:55 PM, Sheng Chen <[email protected]> > wrote: > > Hi, all, > > I have a hbase region server of 10 regions and find the compaction queue > > size is always 10 at maximum. > > After I finished inserting 1 billion records, there are still hundreds of > > HFiles in each region and no further minor compaction is queuing. > > > > My settings, > > hbase.hstore.blockingStoreFiles=Integer.MAX_VALUE > > hbase.hstore.compactionThreshold=3 > > hbase.hstore.max.filesize=100G > > > > Is it the expected behavior? > > Can I make the compaction queue larger in order to keep doing minor > > compaction in the background? > > > > Thank you. > > > > Sean > > >
