Is it possible for one or more of the parameters dynamic ? Meaning embedding tuning heuristic in HBase code.
On Wed, Jun 9, 2010 at 6:08 PM, Ryan Rawson <[email protected]> wrote: > One issue you may run into is that 0.20 doesn't have > https://issues.apache.org/jira/browse/HBASE-2066 > > a dev preview of 0.21 which does include that, and does improve > performance should be available soon. > > On Wed, Jun 9, 2010 at 5:55 PM, Jinsong Hu <[email protected]> wrote: > > Yes, I have done all the suggestion of the > > http://wiki.apache.org/hadoop/PerformanceTuning. > > > > I just restarted the hbase cluster and recreated the table, the data > > insertion looks fine for now and > > I am getting about 1k record/second . I consider that to be reasonable > > giving that my record is about > > 10k bytes per record. but this is the beginning of the writing and I > notice > > that when the table is small, > > the hbase works fine. when there are lots of records in the table > already, > > problem begin to happen. > > I will report back and see how it goes after some more time. > > > > Jimmy. > > > > > > -------------------------------------------------- > > From: "Ryan Rawson" <[email protected]> > > Sent: Wednesday, June 09, 2010 5:20 PM > > To: <[email protected]> > > Subject: Re: ideas to improve throughput of the base writting > > > >> I am not familiar with that exception, I have not seen of it before... > >> perhaps someone else has? > >> > >> And my 200k rows/sec is over 19 machines. It is the average over many > >> hours. My calculation of row size might not match how much data was > >> flowing to disk, but I think it isn't too far off. > >> > >> Unfortunately comparing raw disk speed in a trivial benchmark (such as > >> hdparm -t is) doesn't tell us how absolute speed of HBase must > >> perform. This is because HBase does much more work than a raw disk > >> write benchmark -- doing so to maintain structure and sorting. We can > >> say that 'faster disks = faster HBase performance'. > >> > >> From the log lines you have pasted it sounds like the regionserver's > >> flush ability is not keeping up with your rate of data input. How big > >> are your records? What is your target input speed? Have you done > >> anything on this page: > >> http://wiki.apache.org/hadoop/PerformanceTuning > >> > >> > >> > >> On Wed, Jun 9, 2010 at 4:58 PM, Jinsong Hu <[email protected]> > wrote: > >>> > >>> My hardware has 2 disks. I did a file copy on the machine and found > that > >>> I > >>> can get 300 mbyte/second. > >>> > >>> At this time, I see my insertion is less than 1k/second. my row size is > . > >>> in > >>> terms of disk writing. my record > >>> insertion rate is far less than the hardware limit. my row size is > about > >>> 10K byte > >>> > >>> if in your i7-based server, you are doing 200k row/sec, each row is 200 > >>> byte, then you are doing 40M byte/sec. > >>> > >>> in my case, if it behaves normally, I can get 100 row/sec * 10K byte > =1M > >>> /sec. > >>> that is far from the disk speed. occasionally I can see 1k row/second. > >>> which > >>> is more reasonable in my case, > >>> but I rarely get that performance. > >>> > >>> even worse, with the change done, now I have seem lots of compaction > >>> failure: > >>> > >>> 2010-06-09 23:40:55,117 ERROR > >>> org.apache.hadoop.hbase.regionserver.CompactSplitT > >>> hread: Compaction failed for region Spam_MsgEventTable,2010-06-09 > >>> 20:05:20\x0905 > >>> 860d4bf1cb268ef69391cf97de9f64,1276121160527 > >>> java.lang.RuntimeException: java.io.IOException: Could not find target > >>> position > >>> 65588 > >>> at > >>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileS > >>> canner.java:61) > >>> at > >>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.j > >>> ava:79) > >>> at > >>> org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next > >>> (MinorCompactingStoreScanner.java:96) > >>> at > >>> org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:920) > >>> at > >>> org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:764) > >>> at > >>> org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.ja > >>> va:832) > >>> at > >>> org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.ja > >>> va:785) > >>> at > >>> org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSp > >>> litThread.java:93) > >>> Caused by: java.io.IOException: Could not find target position 65588 > >>> at > >>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.fetchBlockAt(DFSClien > >>> t.java:1556) > >>> at > >>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient > >>> .java:1666) > >>> at > >>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1 > >>> 780) > >>> at java.io.DataInputStream.read(DataInputStream.java:132) > >>> at > >>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bou > >>> ndedRangeFileInputStream.java:105) > >>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100) > >>> at > >>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1 > >>> 018) > >>> at > >>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:96 > >>> 6) > >>> at > >>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java > >>> :1159) > >>> at > >>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileS > >>> canner.java:58) > >>> ... 7 more > >>> > >>> I can't stop this unless I restarted the regionserver. After restart I > >>> truncate the table, and when I list the table again in shell, > >>> it appears 2 times. now I can't even disable the table and drop it. > >>> > >>> I will restart the whole hbase cluster and report the progress. > >>> > >>> Jimmy/ > >>> > >>> > >>> > >>> > >>> > >>> -------------------------------------------------- > >>> From: "Ryan Rawson" <[email protected]> > >>> Sent: Wednesday, June 09, 2010 4:16 PM > >>> To: <[email protected]> > >>> Subject: Re: ideas to improve throughput of the base writting > >>> > >>>> Hey, > >>>> > >>>> Sounds like you are hitting limits of your hardware... I dont think > >>>> you mentioned the hardware spec you are running in this thread... > >>>> > >>>> What you are seeing is essentially the limits of HDFS's ability to > >>>> take writes. The errors might be due to various HDFS setup problems > >>>> (eg: xceiver count, file handle count, all outlined in various HBase > >>>> "startup" docs)... But the overall performance might be limited by > >>>> your hardware. > >>>> > >>>> For example, I use i7-based servers with 4 disks. This gives a > >>>> reasonable IO bandwidth, and can cope with high rates of inserts (upto > >>>> 100-200k rows/sec (each row is ~ 100-300 bytes). If you are running a > >>>> 1 or 2 disk system it is possible you are hitting limits of what your > >>>> hardware can do. > >>>> > >>>> Also note that the write-pipeline performance is ultimately defined in > >>>> bytes/sec, not just 'rows/sec'... thus my rows were small, and if > >>>> yours are big then you might be hitting a lower 'row/sec' limit even > >>>> though the amount of bytes you are writing is higher than what i might > >>>> have been doing. > >>>> > >>>> > >>>> > >>>> On Wed, Jun 9, 2010 at 3:59 PM, Jinsong Hu <[email protected]> > >>>> wrote: > >>>>> > >>>>> I still get lots of repetition of > >>>>> > >>>>> 2010-06-09 22:54:38,428 WARN > >>>>> org.apache.hadoop.hbase.regionserver.MemStoreFlushe > >>>>> r: Region Spam_MsgEventTable,2010-06-09 > >>>>> 20:05:20\x0905860d4bf1cb268ef69391cf97de > >>>>> 9f64,1276121160527 has too many store files, putting it back at the > end > >>>>> of > >>>>> the f > >>>>> lush queue. > >>>>> 2010-06-09 22:54:38,428 DEBUG > >>>>> org.apache.hadoop.hbase.regionserver.CompactSplitT > >>>>> hread: Compaction requested for region Spam_MsgEventTable,2010-06-09 > >>>>> 20:05:20\x0 > >>>>> 905860d4bf1cb268ef69391cf97de9f64,1276121160527/1537478401 because: > >>>>> regionserver > >>>>> /10.110.8.88:60020.cacheFlusher > >>>>> > >>>>> > >>>>> I also saw lots of > >>>>> > >>>>> 2010-06-09 22:50:12,527 INFO > >>>>> org.apache.hadoop.hbase.regionserver.HRegion: > >>>>> Block > >>>>> ing updates for 'IPC Server handler 1 on 60020' on region > >>>>> Spam_MsgEventTable,201 > >>>>> 0-06-09 20:05:20\x0905860d4bf1cb268ef69391cf97de9f64,1276121160527: > >>>>> memstore > >>>>> siz > >>>>> e 512.0m is >= than blocking 512.0m size > >>>>> 2010-06-09 22:50:12,598 INFO > >>>>> org.apache.hadoop.hbase.regionserver.HRegion: > >>>>> Block > >>>>> ing updates for 'IPC Server handler 5 on 60020' on region > >>>>> Spam_MsgEventTable,201 > >>>>> 0-06-09 20:05:20\x0905860d4bf1cb268ef69391cf97de9f64,1276121160527: > >>>>> memstore > >>>>> siz > >>>>> e 512.0m is >= than blocking 512.0m size > >>>>> > >>>>> even with the changed config. the regionserver has 4G ram. what else > >>>>> can > >>>>> be > >>>>> wrong ? > >>>>> > >>>>> The insertion rate is still not good. > >>>>> > >>>>> Jimmy. > >>>>> > >>>>> > >>>>> -------------------------------------------------- > >>>>> From: "Jinsong Hu" <[email protected]> > >>>>> Sent: Wednesday, June 09, 2010 1:59 PM > >>>>> To: <[email protected]> > >>>>> Subject: Re: ideas to improve throughput of the base writting > >>>>> > >>>>>> Thanks. I will make this change: > >>>>>> > >>>>>> <property> > >>>>>> <name>hbase.hregion.memstore.block.multiplier</name> > >>>>>> <value>8</value> > >>>>>> </property> > >>>>>> > >>>>>> <property> > >>>>>> <name>hbase.regionserver.msginterval</name> > >>>>>> <value>10000</value> > >>>>>> </property> > >>>>>> > >>>>>> <property> > >>>>>> <name>hbase.hstore.compactionThreshold</name> > >>>>>> <value>6</value> > >>>>>> </property> > >>>>>> > >>>>>> > >>>>>> <property> > >>>>>> <name>hbase.hstore.blockingStoreFiles</name> > >>>>>> <value>18</value> > >>>>>> </property> > >>>>>> > >>>>>> > >>>>>> and see how it goes. > >>>>>> > >>>>>> > >>>>>> Jimmy. > >>>>>> > >>>>>> -------------------------------------------------- > >>>>>> From: "Ryan Rawson" <[email protected]> > >>>>>> Sent: Wednesday, June 09, 2010 1:49 PM > >>>>>> To: <[email protected]> > >>>>>> Subject: Re: ideas to improve throughput of the base writting > >>>>>> > >>>>>>> More background here... you are running into a situation where the > >>>>>>> regionserver cannot flush fast enough and the size of the region's > >>>>>>> memstore has climbed too high and thus you get that error message. > >>>>>>> HBase attempts to protect itself by holding up clients (thus > causing > >>>>>>> the low performance you see). By expanding how big a memstore can > >>>>>>> get > >>>>>>> during times of stress you can improve performance, at the cost of > >>>>>>> memory usage. That is what that setting is about. > >>>>>>> > >>>>>>> As for the 1.5 minute setting, that is the maximal amount of time a > >>>>>>> handler thread will block for. You shouldn't need to tweak that > >>>>>>> value, and reducing it could cause issues. > >>>>>>> > >>>>>>> Now, as for compacting, HBase will compact small files into larger > >>>>>>> files, and on a massive upload you can expect to see this happen > >>>>>>> constantly, thus tying up 1 cpu worth on your regionserver. You > >>>>>>> could > >>>>>>> potentially reduce that by increasing the value: > >>>>>>> > >>>>>>> <property> > >>>>>>> <name>hbase.hstore.compactionThreshold</name> > >>>>>>> <value>3</value> > >>>>>>> > >>>>>>> the value is interpreted as "if there are more than 3 files for a > >>>>>>> region then run the compaction check". By raising this limit you > can > >>>>>>> accumulate more files before compacting them, thus reducing the > >>>>>>> frequency of compactions but also potentially increasing the > >>>>>>> performance of reads (more files to read = more seeks = slower). > I'd > >>>>>>> consider setting it to 5-7 or so in concert with setting > >>>>>>> "hbase.hstore.blockingStoreFiles" to a value at least 2x that. > >>>>>>> > >>>>>>> All of these settings increase the amount of ram your regionserver > >>>>>>> may > >>>>>>> need, so you will want to ensure you have at least 4000m of ram set > >>>>>>> in > >>>>>>> hbase-env.sh. This is why they are set so conservatively in the > >>>>>>> default shipping config. > >>>>>>> > >>>>>>> These are the 3 important settings that control how often > compactions > >>>>>>> occur and how RPC threads get blocked. Try tweaking all of them > and > >>>>>>> let me know if you are doing better. > >>>>>>> > >>>>>>> -ryan > >>>>>>> > >>>>>>> > >>>>>>> On Wed, Jun 9, 2010 at 1:37 PM, Ryan Rawson <[email protected]> > >>>>>>> wrote: > >>>>>>>> > >>>>>>>> you also want this config: > >>>>>>>> > >>>>>>>> <property> > >>>>>>>> <name>hbase.hregion.memstore.block.multiplier</name> > >>>>>>>> <value>8</value> > >>>>>>>> </property> > >>>>>>>> > >>>>>>>> > >>>>>>>> that should hopefully clear things up. > >>>>>>>> > >>>>>>>> -ryan > >>>>>>>> > >>>>>>>> On Wed, Jun 9, 2010 at 1:34 PM, Jinsong Hu < > [email protected]> > >>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>> I checked the log, there are lots of > >>>>>>>>> > >>>>>>>>> e 128.1m is >= than blocking 128.0m size > >>>>>>>>> 2010-06-09 17:26:36,736 INFO > >>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion: > >>>>>>>>> Block > >>>>>>>>> ing updates for 'IPC Server handler 8 on 60020' on region > >>>>>>>>> Spam_MsgEventTable,201 > >>>>>>>>> 0-06-09 > 05:25:32\x09c873847edf6e5390477494956ec04729,1276104002262: > >>>>>>>>> memstore > >>>>>>>>> siz > >>>>>>>>> e 128.1m is >= than blocking 128.0m size > >>>>>>>>> > >>>>>>>>> then after that there are lots of > >>>>>>>>> > >>>>>>>>> 2010-06-09 17:26:36,800 DEBUG > >>>>>>>>> org.apache.hadoop.hbase.regionserver.Store: > >>>>>>>>> Added > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> hdfs:// > namenodes1.cloud.ppops.net:8020/hbase/Spam_MsgEventTable/376337880/messag > >>>>>>>>> e_compound_terms/7606939244559826252, entries=30869, > >>>>>>>>> sequenceid=8350447892, > >>>>>>>>> mems > >>>>>>>>> ize=7.2m, filesize=3.4m to Spam_MsgEventTable,2010-06-09 > >>>>>>>>> 05:25:32\x09c873847edf6 > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> then lots of > >>>>>>>>> > >>>>>>>>> 2010-06-09 17:26:39,005 INFO > >>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion: > >>>>>>>>> Unblo > >>>>>>>>> cking updates for region Spam_MsgEventTable,2010-06-09 > >>>>>>>>> 05:25:32\x09c873847edf6e5 > >>>>>>>>> 390477494956ec04729,1276104002262 'IPC Server handler 8 on 60020' > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> This cycle happens again and again in the log. What can I do in > >>>>>>>>> this > >>>>>>>>> case > >>>>>>>>> to speed up writing ? > >>>>>>>>> right now the writing speed is really slow. close to 4 > rows/second > >>>>>>>>> for > >>>>>>>>> a > >>>>>>>>> regionserver. > >>>>>>>>> > >>>>>>>>> I checked the code and try to find out why there are so many > store > >>>>>>>>> files, > >>>>>>>>> and I noticed each second > >>>>>>>>> the regionserver reports to master, it calls the memstore flush > and > >>>>>>>>> write a > >>>>>>>>> store file. > >>>>>>>>> > >>>>>>>>> the parameter hbase.regionserver.msginterval default value is 1 > >>>>>>>>> second. > >>>>>>>>> I am > >>>>>>>>> thinking to change to 10 second. > >>>>>>>>> can that help ? I am also thinking to change > >>>>>>>>> hbase.hstore.blockingStoreFiles > >>>>>>>>> to 1000. I noticed that there is a parameter > >>>>>>>>> hbase.hstore.blockingWaitTime with default value of 1.5 minutes. > as > >>>>>>>>> long as > >>>>>>>>> the 1.5 minutes is reached, > >>>>>>>>> the compaction is executed. I am fine with running compaction > every > >>>>>>>>> 1.5 > >>>>>>>>> minutes, but running compaction every second > >>>>>>>>> and causing CPU consistently higher than 100% is not wanted. > >>>>>>>>> > >>>>>>>>> Any suggestion what kind of parameters to change to improve my > >>>>>>>>> writing > >>>>>>>>> speed > >>>>>>>>> ? > >>>>>>>>> > >>>>>>>>> Jimmy > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -------------------------------------------------- > >>>>>>>>> From: "Ryan Rawson" <[email protected]> > >>>>>>>>> Sent: Wednesday, June 09, 2010 1:01 PM > >>>>>>>>> To: <[email protected]> > >>>>>>>>> Subject: Re: ideas to improve throughput of the base writting > >>>>>>>>> > >>>>>>>>>> The log will say something like "blocking updates to..." when > you > >>>>>>>>>> hit > >>>>>>>>>> a limit. That log you indicate is just the regionserver > >>>>>>>>>> attempting > >>>>>>>>>> to > >>>>>>>>>> compact a region, but shouldn't prevent updates. > >>>>>>>>>> > >>>>>>>>>> what else does your logfile say? Search for the string (case > >>>>>>>>>> insensitive) "blocking updates"... > >>>>>>>>>> > >>>>>>>>>> -ryan > >>>>>>>>>> > >>>>>>>>>> On Wed, Jun 9, 2010 at 11:52 AM, Jinsong Hu > >>>>>>>>>> <[email protected]> > >>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> I made this change > >>>>>>>>>>> <property> > >>>>>>>>>>> <name>hbase.hstore.blockingStoreFiles</name> > >>>>>>>>>>> <value>15</value> > >>>>>>>>>>> </property> > >>>>>>>>>>> > >>>>>>>>>>> the system is still slow. > >>>>>>>>>>> > >>>>>>>>>>> Here is the most recent value for the region : > >>>>>>>>>>> stores=21, storefiles=186, storefileSizeMB=9681, > >>>>>>>>>>> memstoreSizeMB=128, > >>>>>>>>>>> storefileIndexSizeMB=12 > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> And the same log still happens: > >>>>>>>>>>> > >>>>>>>>>>> 2010-06-09 18:36:40,577 WARN org.apache.h > >>>>>>>>>>> adoop.hbase.regionserver.MemStoreFlusher: Region > >>>>>>>>>>> SOME_ABCEventTable,2010-06-09 0 > >>>>>>>>>>> 9:56:56\x093dc01b4d2c4872963717d80d8b5c74b1,1276107447570 has > too > >>>>>>>>>>> many > >>>>>>>>>>> store > >>>>>>>>>>> fil > >>>>>>>>>>> es, putting it back at the end of the flush queue. > >>>>>>>>>>> > >>>>>>>>>>> One idea that I have now is to further increase the > >>>>>>>>>>> hbase.hstore.blockingStoreFiles to a very high > >>>>>>>>>>> Number, such as 1000. What is the negative impact of this > change > >>>>>>>>>>> ? > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Jimmy > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> -------------------------------------------------- > >>>>>>>>>>> From: "Ryan Rawson" <[email protected]> > >>>>>>>>>>> Sent: Monday, June 07, 2010 3:58 PM > >>>>>>>>>>> To: <[email protected]> > >>>>>>>>>>> Subject: Re: ideas to improve throughput of the base writting > >>>>>>>>>>> > >>>>>>>>>>>> Try setting this config value: > >>>>>>>>>>>> > >>>>>>>>>>>> <property> > >>>>>>>>>>>> <name>hbase.hstore.blockingStoreFiles</name> > >>>>>>>>>>>> <value>15</value> > >>>>>>>>>>>> </property> > >>>>>>>>>>>> > >>>>>>>>>>>> and see if that helps. > >>>>>>>>>>>> > >>>>>>>>>>>> The thing about the 1 compact thread is the scarce resources > >>>>>>>>>>>> being > >>>>>>>>>>>> preserved in this case is cluster IO. People have had issues > >>>>>>>>>>>> with > >>>>>>>>>>>> compaction IO being too heavy. > >>>>>>>>>>>> > >>>>>>>>>>>> in your case, this setting can let the regionserver build up > >>>>>>>>>>>> more > >>>>>>>>>>>> store files without pausing your import. > >>>>>>>>>>>> > >>>>>>>>>>>> -ryan > >>>>>>>>>>>> > >>>>>>>>>>>> On Mon, Jun 7, 2010 at 3:52 PM, Jinsong Hu > >>>>>>>>>>>> <[email protected]> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> Hi, There: > >>>>>>>>>>>>> While saving lots of data to on hbase, I noticed that the > >>>>>>>>>>>>> regionserver > >>>>>>>>>>>>> CPU > >>>>>>>>>>>>> went to more than 100%. examination shows that the hbase > >>>>>>>>>>>>> CompactSplit > >>>>>>>>>>>>> is > >>>>>>>>>>>>> spending full time working on compacting/splitting hbase > store > >>>>>>>>>>>>> files. > >>>>>>>>>>>>> The > >>>>>>>>>>>>> machine I have is an 8 core machine. because there is only > one > >>>>>>>>>>>>> comact/split > >>>>>>>>>>>>> thread in hbase, only one core is fully used. > >>>>>>>>>>>>> I continue to submit map/reduce job to insert records to > >>>>>>>>>>>>> hbase. > >>>>>>>>>>>>> most > >>>>>>>>>>>>> of > >>>>>>>>>>>>> the time, the job runs very fast, around 1-5 minutes. But > >>>>>>>>>>>>> occasionally, > >>>>>>>>>>>>> it > >>>>>>>>>>>>> can take 2 hours. That is very bad to me. I highly suspect > that > >>>>>>>>>>>>> the > >>>>>>>>>>>>> occasional slow insertion is related to the > >>>>>>>>>>>>> insufficient speed compactsplit thread. > >>>>>>>>>>>>> I am thinking that I should parallize the compactsplit > thread, > >>>>>>>>>>>>> the > >>>>>>>>>>>>> code > >>>>>>>>>>>>> has > >>>>>>>>>>>>> this : the for loop "for (Store store: stores.values()) " > can > >>>>>>>>>>>>> be > >>>>>>>>>>>>> parallized via java 5's threadpool , thus multiple cores are > >>>>>>>>>>>>> used > >>>>>>>>>>>>> instead > >>>>>>>>>>>>> only one core is used. I wonder if this will help to increase > >>>>>>>>>>>>> the > >>>>>>>>>>>>> throughput. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Somebody mentioned that I can increase the regionsize to > that > >>>>>>>>>>>>> I > >>>>>>>>>>>>> don't > >>>>>>>>>>>>> do > >>>>>>>>>>>>> so > >>>>>>>>>>>>> many compaction. Under heavy writing situation. > >>>>>>>>>>>>> does anybody have experience showing it helps ? > >>>>>>>>>>>>> > >>>>>>>>>>>>> Jimmy. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> byte [] compactStores(final boolean majorCompaction) > >>>>>>>>>>>>> > >>>>>>>>>>>>> throws IOException { > >>>>>>>>>>>>> > >>>>>>>>>>>>> if (this.closing.get() || this.closed.get()) { > >>>>>>>>>>>>> > >>>>>>>>>>>>> LOG.debug("Skipping compaction on " + this + " because > >>>>>>>>>>>>> closing/closed"); > >>>>>>>>>>>>> > >>>>>>>>>>>>> return null; > >>>>>>>>>>>>> > >>>>>>>>>>>>> } > >>>>>>>>>>>>> > >>>>>>>>>>>>> splitsAndClosesLock.readLock().lock(); > >>>>>>>>>>>>> > >>>>>>>>>>>>> try { > >>>>>>>>>>>>> > >>>>>>>>>>>>> byte [] splitRow = null; > >>>>>>>>>>>>> > >>>>>>>>>>>>> if (this.closed.get()) { > >>>>>>>>>>>>> > >>>>>>>>>>>>> return splitRow; > >>>>>>>>>>>>> > >>>>>>>>>>>>> } > >>>>>>>>>>>>> > >>>>>>>>>>>>> try { > >>>>>>>>>>>>> > >> > > >
