[
https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925960#action_12925960
]
HBase Review Board commented on HBASE-3160:
-------------------------------------------
Message from: "Nicolas" <[email protected]>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1103/
-----------------------------------------------------------
(Updated 2010-10-28 14:48:04.897593)
Review request for hbase.
Changes
-------
Updates after internal review.
1. Removed PRIORITY_BLOCKING. It was a little confusing because blocking is <=
0 and not necessarily the value 0 itself.
2. Cleaned up compaction downgrade conditional.
Questions resolved:
1q. Wondering if we want to go ahead with a split if the compaction priority is
PRIORITY_BLOCKING?
1a. Counter-example: storefiles could be very small and compaction will reduce
it to one medium file. It was starved because a major compaction occurred
immediately beforehand and took forever to finish.
2q. Should we add in logic to interrupt a running compaction if it is lower
priority (e.g. Pri >= 5) and we have a PRIORITY_BLOCKING request? We can then
re-enqueue the compaction to run at a less pressured time.
2a. Save this problem for another JIRA. Since the master has threadpools, we
should give the CompactSplitThread prioritized threadpools and dedicate one
thread only to blocking requests.
Summary
-------
Switch to more intelligent priority metric: blockingSize -
max(len(s.storefiles) for s in region.stores) . This will allow us to better
prioritize, give us faster responsiveness to users, and feel more cavalier
about issuing new compaction requests. Note that we also found/fixed a major
compaction downgrade bug while writing this code.
This addresses bug HBASE-3160.
http://issues.apache.org/jira/browse/HBASE-3160
Diffs (updated)
-----
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionRequestor.java
1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/PriorityCompactionQueue.java
1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1027787
trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestPriorityCompactionQueue.java
1027787
Diff: http://review.cloudera.org/r/1103/diff
Testing
-------
mvn clean install -Dtest=TestPriorityCompaction
dev cluster tests (on 0.89)
Thanks,
Nicolas
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
>
> One of the problems with the current compaction queue is that we have a very
> low granularity on the importance of the various compactions in the queue.
> If a StoreFile count exceeds 15 files, only then do we bump via enum change.
> We should instead look into more intelligent, granular priority metrics for
> choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
