[
https://issues.apache.org/jira/browse/HBASE-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liyin Tang reassigned HBASE-6361:
---------------------------------
Assignee: Akashnil
> Change the compaction queue to a round robin scheduler
> ------------------------------------------------------
>
> Key: HBASE-6361
> URL: https://issues.apache.org/jira/browse/HBASE-6361
> Project: HBase
> Issue Type: Improvement
> Reporter: Akashnil
> Assignee: Akashnil
>
> Currently the compaction requests are submitted to the minor/major compaction
> queue of a region-server from every column-family/region belonging to it. The
> requests are processed from the queue in FIFO order (First in First out). We
> want to make a lazy scheduler in place of the current queue-based one. The
> idea of lazy scheduling is that, it is always better to make a decision
> (compaction selection) later if the decision is relevant later only.
> Presently, when the queue gets bottle-necked, there is a delay between
> compaction selection of a request and its execution. Rather than that, we can
> postpone the compaction selection until the queue is empty when we will have
> more information and choices (new flush files will have arrived by then) to
> make a better decision.
> Removing the queue, we propose to implement a round-robin scheduler. All the
> column families in their regions will be visited in sequence periodically. In
> each visit, if the column family generates a valid compaction request, the
> request is executed before moving to the next one. We do not plan to change
> the current compaction algorithm for now. We expect that it will
> automatically make a better decision when doing just-in-time selection due to
> the new change. How do we know that? Let us consider an example.
> Suppose there is a short term bottleneck in the queue so that it is blocked
> for a period of time. (Let the min-files for compaction = 4). For an active
> column-family, when new flushes are written, new compaction requests, each of
> size 4, will be added to the queue continuously until the queue starts
> processing them.
> Now consider a round-robin scheduler. The effect of a bottle-neck due to the
> IO rate of compaction results in a longer latency to visit the same column
> family again. When the same active column family is visited following a long
> delay, suppose 16 new flush files have been written there. The compaction
> selection algorithm will select one compaction request of size 16, as opposed
> to 4 compaction requests of size 4 that would have been generated in the
> previous case.
> A compaction request with 16 flush files is more IOPs-efficient than the same
> set of files being compacted 4 at a time. This is because both consume the
> same total amount of reads and writes while producing a file of size 16
> compared to 4 files of size 4. So, in the second case, we obtained a free
> compaction 4*4->16 without paying for it. In case of the queue, those smaller
> 4 sized files would have consumed more IOPs to become bigger later.
> On my simulator, I did some experiments on how a bottle-neck of the queue
> affects the compaction selections in the current system. It appears that, a
> filled up queue actually makes all future compaction selections less and less
> efficient in terms of IOPs, resulting in a runway positive feedback loop
> which can potentially explode the compaction queue. (This was also observed
> in production recently). The main effect of this change should be to deal
> with bursty loads. When a bottleneck occurs, the compaction selection will
> become more IOPs-efficient rather than less efficient, resulting in negative
> feedback and restoration to stability more easily. As for monitoring, the
> compaction queue size will not be present as a metric. However, the number of
> files in each compaction will indicate if a bottleneck has occurred.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
