Re: Major Compaction in 0.90.6
On Mon, Aug 19, 2013 at 11:52 PM, Monish r monishs...@gmail.com wrote: Hi Jean, s/Jean/Jean-Daniel ;) Thanks for the explanation. Just a clarification on the third answer, In our current cluster ( 0.90.6 ) , i find that irrespective of whether TTL is set or not , Major compaction compaction rewrites hfile for the region ( there is only one hfile for that region ) on every manual major compaction trigger. Can you enable DEBUG logs? You'd see why the major compaction is triggered. log : 2013-08-19 14:15:29,926 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 1 file(s), new file=hdfs://x.x.x.x:9000/hbase/NOTIFICATION_HISTORY/b00086bca62ee55796a960002291aca4/n/4754838096619480671 i find a new file is created for every major compaction triggger. Regards, R.Monish On Mon, Aug 19, 2013 at 11:52 PM, Jean-Daniel Cryans jdcry...@apache.org wrote: Inline. J-D On Mon, Aug 19, 2013 at 2:48 AM, Monish r monishs...@gmail.com wrote: Hi guys, I have the following questions in HBASE 0.90.6 1. Does hbase use only one compaction thread to handle both major and minor compaction? Yes, look at CompactSplitThread 2. If hbase uses multiple compaction threads, which configuration parameter defines the number of compaction threads? It doesn't in 0.90.6 but CompactSplitThread lists those for 0.92+ hbase.regionserver.thread.compaction.large hbase.regionserver.thread.compaction.small 3. After hbase.majorcompaction.interval from last major compaction ,if major compaction is executed on a table already major compacted Does hbase skip all the table regions from major compaction? Determining if something is major-compacted is definitely not at the table-level. In 0.90.6, MajorCompactionChecker will ask HRegion.isMajorCompaction() to check if it needs to major compact again, which in turns checks every Store. FWW if you have TTL turned on it will still major compact a major compacted file, HFiles don't have an index of what's deleted or TTL'd and it doesn't do a full read of each files to check. Regards, R.Monish
Major Compaction in 0.90.6
Hi guys, I have the following questions in HBASE 0.90.6 1. Does hbase use only one compaction thread to handle both major and minor compaction? 2. If hbase uses multiple compaction threads, which configuration parameter defines the number of compaction threads? 3. After hbase.majorcompaction.interval from last major compaction ,if major compaction is executed on a table already major compacted Does hbase skip all the table regions from major compaction? Regards, R.Monish
Re: Major Compaction in 0.90.6
Hi Monish, Any reason for you to use 0.90.6? It's a pretty old version. Have you considered moving to a more recent one? JM 2013/8/19 Monish r monishs...@gmail.com Hi guys, I have the following questions in HBASE 0.90.6 1. Does hbase use only one compaction thread to handle both major and minor compaction? 2. If hbase uses multiple compaction threads, which configuration parameter defines the number of compaction threads? 3. After hbase.majorcompaction.interval from last major compaction ,if major compaction is executed on a table already major compacted Does hbase skip all the table regions from major compaction? Regards, R.Monish
Re: Major Compaction in 0.90.6
Hi Jean, We are thinking of moving to 0.92 or down the line , but for now we are using this version. If anyone can share what the behavior of hbase in 0.90.x for the above questions will be great. One more question to add, 1.Major compaction rewrites a region hfile , even if it has only one hfile for that region and contains no expired cells ( both by delete or by TTL ) . Doesnt hbase major compaction have any intellegence to determine whether a rewrite is necessary? Regards, R.Monish On Mon, Aug 19, 2013 at 7:19 PM, Jean-Marc Spaggiari jean-m...@spaggiari.org wrote: Hi Monish, Any reason for you to use 0.90.6? It's a pretty old version. Have you considered moving to a more recent one? JM 2013/8/19 Monish r monishs...@gmail.com Hi guys, I have the following questions in HBASE 0.90.6 1. Does hbase use only one compaction thread to handle both major and minor compaction? 2. If hbase uses multiple compaction threads, which configuration parameter defines the number of compaction threads? 3. After hbase.majorcompaction.interval from last major compaction ,if major compaction is executed on a table already major compacted Does hbase skip all the table regions from major compaction? Regards, R.Monish
