[
https://issues.apache.org/jira/browse/HBASE-7842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578081#comment-13578081
]
Sergey Shelukhin commented on HBASE-7842:
-----------------------------------------
what I don't understand is the first new condition for every file. E.g. (assume
ratio 1) in order 10 7 4 5 is good to compact but 7 10 4 5 is not (if reordered
by size while preserving contiguousness, some other example can probably be
invented.
Then, maximizing ratio without regard for number of files can be bad, e.g.
files 0..3 with sizes 100 3 2 2 - max files 3, min files 2, assuming "2 2"
passes ratio test - we can compact 2..3 for 2/4, or 1..3 for 3/7; 3/7 is less,
but we probably want to do it anyway to minimize I/O amplification later.
Just thinking aloud: what if we use ratio/size filter *after* we find best set
out of all permutations (# of files/sum of sizes)? Presumably, good
permutations already won't rewrite a lot of data needlessly, because otherwise
they would have large sum of sizes and thus lower preference for being the
"best set".
For example, 10 2 2 2 2 3 2, max files 6 - if we choose size 6 sets, 0..5 gets
us 6/21, and 1..6 gets us 6/13, we don't need ratio test for each set to
establish that.
So the only thing we need to do is find if the chosen set itself satisfies some
ratio test... probably ratio for biggest file (without regard to ordering of
files, e.g. 6 10 6 is good). If it can then ratio is good, it's the best
compaction we can do (according to this criteria), if not the best compaction
we can do is not good enough.
> Add compaction policy that explores more storefile groups
> ---------------------------------------------------------
>
> Key: HBASE-7842
> URL: https://issues.apache.org/jira/browse/HBASE-7842
> Project: HBase
> Issue Type: New Feature
> Components: Compaction
> Affects Versions: 0.96.0
> Reporter: Elliott Clark
> Assignee: Elliott Clark
>
> Some workloads that are not as stable can have compactions that are too large
> or too small using the current storefile selection algorithm.
> Currently:
> * Find the first file that Size(fi) <= Sum(0, i-1, FileSize(fx))
> * Ensure that there are the min number of files (if there aren't then bail
> out)
> * If there are too many files keep the larger ones.
> I would propose something like:
> * Find all sets of storefiles where every file satisfies
> ** FileSize(fi) <= Sum(0, i-1, FileSize(fx))
> ** Num files in set =< max
> ** Num Files in set >= min
> * Then pick the set of files that maximizes ((# storefiles in set) /
> Sum(FileSize(fx)))
> The thinking is that the above algorithm is pretty easy reason about, all
> files satisfy the ratio, and should rewrite the least amount of data to get
> the biggest impact in seeks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
