[
https://issues.apache.org/jira/browse/ACCUMULO-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13758464#comment-13758464
]
John Vines commented on ACCUMULO-902:
-------------------------------------
So, I put some thought into this and I think the easiest way to configure it is
to define a minimum and maximum amount of MinC and MajC. Then the overall
maximum amount of *Compactions is the max(minimum MinC + maximum MajC, minimum
MajC + maximum MinC) [Though I would think that the latter argument is dominant
due to suggested configurations]. Then the weights of minor vs. major can be
determined by the comparing the difference of max and min of each type of
compaction.
The tricky part, aside from setting up the thread pool(s) to use and managing
it, is determining which task to go for next. Of course, if there are < minimum
of a type running, those should jump to the front of the queue. But depending
on your configuration, there is the possibility that Majors get blocked off
because minors keep taking up slots (they wrap and then no major can fit in
that slot so another minor runs, etc. so Majors get bound by the minimum,
though this could be the case of a bad configuration).
An optional component is to allow some customization in the order of
compactions to allow some sort of per-table QoS (which may be a refactoring of
the !METADATA/!!ROOT table code), but that's another conversation.
There also lends the case of a cost of merging minor compactions vs. normal
minor compactions, but I think its' really not worth segregating them.
> Have a common resource pool for minor and major compactions
> -----------------------------------------------------------
>
> Key: ACCUMULO-902
> URL: https://issues.apache.org/jira/browse/ACCUMULO-902
> Project: Accumulo
> Issue Type: Improvement
> Components: tserver
> Reporter: John Vines
> Assignee: Keith Turner
> Fix For: 1.6.0
>
>
> Currently we have a defined threadpool for minor and major compactions,
> independent of one another. However, there are situations where a system may
> be minor compaction heavy with no major, or vice versa. I would like to see a
> common threadpool which is accessible to both operations for work to be done,
> with guarantees for certain resources to be available to the other type of
> work. That is, it should be a defined pool size with a (configurable) minimum
> of resources maintained for the other to maintain a certain QoS. Of course,
> major is heavier than minor, so some weighting of operations needs to be done
> to keep workloads reasonable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
