[
https://issues.apache.org/jira/browse/HBASE-834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12625638#action_12625638
]
stack commented on HBASE-834:
-----------------------------
I tried the patch. Here's a filtered extract from the logs that just shows the
new messages around the major compaction test:
{code}
2008-08-26 04:31:02,419 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Hours sense last major compaction: 0
2008-08-26 04:31:02,420 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Major compaction triggered on store: 1028785192/historian
2008-08-26 04:31:02,877 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Hours sense last major compaction: 0
2008-08-26 04:31:02,878 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Major compaction triggered on store: 1028785192/info
2008-08-26 04:31:08,588 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Hours sense last major compaction: 4
2008-08-26 04:31:08,588 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Major compaction triggered on store: 652690253/info
2008-08-26 04:31:37,237 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Hours sense last major compaction: 4
2008-08-26 04:31:37,237 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Major compaction triggered on store: 1648250611/info
2008-08-26 04:31:59,721 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Hours sense last major compaction: 4
2008-08-26 04:31:59,721 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Major compaction triggered on store: 433766857/info
2008-08-26 04:32:23,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Hours sense last major compaction: 4
2008-08-26 04:32:23,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Major compaction triggered on store: 532635319/info
2008-08-26 04:32:40,876 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Hours sense last major compaction: 4
2008-08-26 04:32:40,876 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Major compaction triggered on store: 478968074/info
2008-08-26 04:33:16,252 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Hours sense last major compaction: 4
2008-08-26 04:33:16,252 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Major compaction triggered on store: 305918941/info
2008-08-26 04:33:32,483 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Hours sense last major compaction: 4
2008-08-26 04:33:32,483 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Major compaction triggered on store: 629593107/info
2008-08-26 04:49:28,735 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Hours sense last major compaction: 0
2008-08-26 04:49:28,735 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Major compaction triggered on store: 1028785192/historian
2008-08-26 04:49:29,218 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Hours sense last major compaction: 0
2008-08-26 04:49:29,218 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Major compaction triggered on store: 1028785192/info
2008-08-26 04:57:20,395 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Hours sense last major compaction: 0
2008-08-26 04:57:32,731 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Hours sense last major compaction: 0
2008-08-26 04:58:23,362 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Hours sense last major compaction: 0
2008-08-26 04:58:23,362 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Major compaction triggered on store: 1720995599/info
2008-08-26 04:58:44,441 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Hours sense last major compaction: 0
2008-08-26 04:58:56,754 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Hours sense last major compaction: 0
2008-08-26 04:59:41,982 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Hours sense last major compaction: 0
...
{code}
We seem to be major compacting too much -- even though I'd set major compaction
time down to 30 minutes instead of 24 so I could test (Its probably this test
'if (lowTimestamp < System.currentTimeMillis() - majorCompactionTime){
' -- if lowTimestamp is zero, then we'll major compact).
We probably shouldn't log if we're returning a zero out of getLowTimestamp
method.
Would also suggest that getLowTimestamp be renamed getLowestTimestamp and moved
into HStore from HRegion since its only used there (make it private too?).
Did you mean to do the below in HRegion Billy?
{code}
@@ -867,7 +889,7 @@
* @throws IOException
*/
public byte [] compactStores() throws IOException {
- return compactStores(false);
+ return compactStores(false);
{code}
Make the above fixes and I'll try it again Billy. We need this patch.
> Upper bound on files we compact at any one time
> -----------------------------------------------
>
> Key: HBASE-834
> URL: https://issues.apache.org/jira/browse/HBASE-834
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.2.1, 0.18.0
> Reporter: stack
> Assignee: Billy Pearson
> Priority: Minor
> Fix For: 0.2.1, 0.19.0
>
> Attachments: 834-0.2.1-patch.txt, 834-patch.txt
>
>
> From Billy in HBASE-64, which we closed because it got pulled all over the
> place:
> {code}
> Currently we do compaction on a region when the
> hbase.hstore.compactionThreshold is reached - default 3
> I thank we should configure a max number of mapfiles to compact at one time
> simulator to doing a minor compaction in bigtable. This keep compaction's
> form getting tied up in one region to long letting other regions get way to
> many memcache flushes making compaction take longer and longer for each region
> If we did that when a regions updates start to slack off the max number will
> eventuly include all mapfiles causeing a major compaction on that region.
> Unlike big table this would leave the master out of the process and letting
> the region server handle the major compaction when it has time.
> When doing a minor compaction on a few files I thank we should compact the
> newest mapfiles first leave the larger/older ones for when we have low
> updates to a region.
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
