[
https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635925#comment-13635925
]
Elliott Clark commented on HBASE-8299:
--------------------------------------
Then running the perf tests gets this result:
h2.Data
||Name||Max Files||Min Files||Ratio||IO||File Diff||IO Per File||
|EverythingPolicy|10|3|1.2|566916287|-113090|5012.97|
|RandomPolicy|10|3|1.2|457296661|-109033|4194.11|
|NoStuckLargestPolicy|10|3|1.2|30827212|-111271|277.05|
|NoStuckBestRatioPolicy|10|3|1.2|25519685|-111439|229.00|
|RatioBasedCompactionPolicy|10|3|1.2|25242372|-111622|226.14|
|NoStuckSmallestPolicy|10|3|1.2|24200357|-111339|217.36|
|ExploringCompactionPolicy|10|3|1.2|23526926|-110921|212.11|
h2.Explanations
* Random Policy just illustrates what the worst case is
* Everything policy compacts all files all the time.
* The No Stuck Smallest policy seems like the best all around pick for this.
It won't get stuck because when it hits the storefile blocking limit it ignores
the ratio. It then picks the smallest set of files to compact.
* The No Stuck Best Ratio does similar except instead of trying to pick the
smallest set of files it chooses a set of files that are close to being in
ratio. This works really well on spiky load but really badly on sinusoidal
work loads.
h2.Testing
[~jdcryans] put this patch up on a 0.94 cluster. He then started a random
gets, while looping a bulk load job. For the old compaction algorithm we ended
up with max average latency of 136ms. For the new algorithm that doesn't get
stuck we ended up with a max average latency of 20ms. Lots of compactions were
skipped. The compactions that were chosen all were what I would have expected.
No major compactions were needed or triggered.
h2.Plan
So I'll clean up the code and make the NoStuckSmallestPolicy default.
> ExploringCompactionPolicy can get stuck in rare cases.
> ------------------------------------------------------
>
> Key: HBASE-8299
> URL: https://issues.apache.org/jira/browse/HBASE-8299
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.95.1
> Reporter: Elliott Clark
> Assignee: Elliott Clark
> Attachments: HBASE-8299-0.patch, HBASE-8299-1.patch
>
>
> If the files are very oddly sized then it's possible that
> ExploringCompactionPolicy can get stuck.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
