O, I understand your requirement. Because hbase put op is write into memory, insert speed will be very high. Files in the store will aways more than 3, then compact is aways needed, so your compaction queen size is aways 10. This means the minor compaction is already aways working in back ground. But insert is to quick, compact cann't follow it.(There is only 1 compact thread in 1 regionserver, 0.90.x). If the region doesn't split in time, the compact file size may be larger and larger, and may take too long to finish a compact.
Change this two settings may help: hbase.hstore.compaction.max: this makes one compact job can compact more files. hbase.hregion.memstore.flush.size: this will make your files to compact bigger, and decrease total file number in the store. Zhou Shuaifeng(Frank) -----Original Message----- From: Sheng Chen [mailto:[email protected]] Sent: Thursday, July 07, 2011 6:49 PM To: [email protected] Subject: Re: Is there a way to increase the compaction queue size? Thank you very much. Actually I was trying to manage splitting manually, and to leave the compaction automatical. I really do not want to block my inserts by waiting for compactions. Is it possible to keep minor compaction working in the background, like starting a periodic job to do 'compact mytable' in the shell? Regards, Sean 2011/7/7 Zhoushuaifeng <[email protected]> > hbase.hstore.blockingStoreFiles=Integer.MAX_VALUE > It's better not setting this to max_value. Setting this to a limited value > will help hbase to control the insert speed, and have enough time to finish > compact. Suggest to setting this to less than 20. > > hbase.hstore.max.filesize = 100G, do you mean hbase.hregion.max.filesize ? > This may be too large, if a store file too large, it will take very long > time to compact. I suggest to set this less than 8G. > > And the compaction queue size means how many regions need compaction, not > the number of files to compact. So, you have 10 regions, all need > compaction, and the compaction queue is 10. > > > Zhou Shuaifeng(Frank) > > -----Original Message----- > From: Sheng Chen [mailto:[email protected]] > Sent: Thursday, July 07, 2011 3:36 PM > To: [email protected] > Subject: Re: Is there a way to increase the compaction queue size? > > Thank you for your reply. > > No more minor compaction is going on, while there are still about 100 > HStoreFiles in each region. > I think it might because the compactionQueueSize has a maximum of 10 for a > region server (only one per region?), so after I stopped inserting and the > 10 compactions in the queue were finished, no more compaction is triggered. > > I think that "hbase.hstore.compaction.max" sets the max number of files for > one minor compaction, does it has anything to do with my problem? > > Thanks, > Sean > > 2011/7/7 Stack <[email protected]> > > > Try setting "hbase.hstore.compaction.max" to a value other than 10, its > > default. > > > > What do you mean by 'no further minor compaction is queuing'? There > > is no more compaction going on? Are there exceptions in the log? > > > > St.Ack > > > > On Wed, Jul 6, 2011 at 10:55 PM, Sheng Chen <[email protected]> > > wrote: > > > Hi, all, > > > I have a hbase region server of 10 regions and find the compaction > queue > > > size is always 10 at maximum. > > > After I finished inserting 1 billion records, there are still hundreds > of > > > HFiles in each region and no further minor compaction is queuing. > > > > > > My settings, > > > hbase.hstore.blockingStoreFiles=Integer.MAX_VALUE > > > hbase.hstore.compactionThreshold=3 > > > hbase.hstore.max.filesize=100G > > > > > > Is it the expected behavior? > > > Can I make the compaction queue larger in order to keep doing minor > > > compaction in the background? > > > > > > Thank you. > > > > > > Sean > > > > > >
