These are my current configs: hbase.regionserver.handler.count 100 (Default was a much smaller number like 25 or something) hbase.hregion.memstore.block.multiplier 4 hbase.hstore.blockingStoreFiles 16 (default was 4.. Could this be the reason? But I don't see any IOExceptions in my log) hbase.hregion.majorcompaction 691200000 (major compactions off) hfile.block.cache.size 0.5 (default was 0.2)
As for the logs, I do see a lot of 2010-05-26 05:46:05,938 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction requested for region DocData,32016328,1274284525421/910285966 because: Region has too many store files But no IOExceptions.. And these compaction requests were turned off since they hadnt crossed the ttl.. I will post the more relevant pieces of the log.. Vidhya On 5/26/10 10:19 AM, "Jonathan Gray" <[email protected]> wrote: If you can post the logs somewhere that would be very helpful. At 2000 regions/node you probably need to continue to increase the ulimit. You might also need more handlers in the RS and DN. > -----Original Message----- > From: Vidhyashankar Venkataraman [mailto:[email protected]] > Sent: Wednesday, May 26, 2010 10:09 AM > To: [email protected] > Subject: Re: Effect of turning major compactions off.. > > No OOME or HDFS errors that I can see in the logs.. > I turned major compaction on and restarted Hbase : now the RS's arent > shutting down: Compactions are happening.. > > I had set the ulimit to 8000 a while back.. Should I increase it more > then? (With the current setting, each region can have a max of around 4 > open files if there are 2000 regions per node)... > > Let me also check the logs a little more carefully and get back to the > forum.. > > Thank you > Vidhya > > > On 5/26/10 9:38 AM, "Jean-Daniel Cryans" <[email protected]> wrote: > > I'm pretty sure something else is going on. > > 1) What does it log when it shuts down? Zookeeper session timeout? > OOME? HDFS errors? > > 2) Is your cluster meeting all the requirements? Especially the last > bullet point? See > http://hadoop.apache.org/hbase/docs/r0.20.4/api/overview- > summary.html#requirements > > J-D > > On Wed, May 26, 2010 at 9:07 AM, Vidhyashankar Venkataraman > <[email protected]> wrote: > > Are there any side effects to turning major compactions off, other > than just a hit in the read performance? > > > > I was trying to merge a 120 Gig update (modify/insert/delete > operations) into a 2 TB fully compacted Hbase table with 5 region > servers using a map reduce job.. Each RS was serving around 2000 > regions (256 MB max size)... Major compactions were turned off before > the job started (by setting the compaction period very high to around 4 > or 5 days).. > > > > As the job was going on, the region servers just shut down after the > table reached near-100% fragmentation (as shown in the web interface).. > On looking at the RS logs, I saw that there were compaction checks for > each region which obviously didn't clear, and the RS's shut down soon > after the checks.. I tried restarting the database after killing the > map reduce job (still, with major compactions turned off).. The RS's > shut down soon after booting up.. > > > > Is this expected? Even if the update files (the additional > StoreFiles) per region get huge, won't the region get split on its own? > > > > Thank you > > Vidhya > >
