[
https://issues.apache.org/jira/browse/HBASE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642673#comment-16642673
]
Andrew Purtell commented on HBASE-21000:
----------------------------------------
I put up two patches, one for branch-1, and one for master and branch-2.
I'd like to see this committed to branch-1 so 1.5 releases do not present a
regression over 1.4 or earlier releases.
Compaction has changed quite a bit in master or branch-2, and although this
settings change will affect them the same way as branch-1, I have no strong
opinion either way if anything need be changed there. Making a change on a
releasing branch-2 would warrant a release note.
> Default limits for PressureAwareCompactionThroughputController are too low
> --------------------------------------------------------------------------
>
> Key: HBASE-21000
> URL: https://issues.apache.org/jira/browse/HBASE-21000
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 1.5.0
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-21000-branch-1.patch, HBASE-21000.patch
>
>
> In PressureAwareCompactionThroughputController:
> {code:java}
> /**
>
>
> * A throughput controller which uses the follow schema to limit throughput
>
>
> * <ul>
>
>
> * <li>If compaction pressure is greater than 1.0, no limitation.</li>
>
>
> * <li>In off peak hours, use a fixed throughput limitation
>
>
> * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_OFFPEAK}</li>
>
>
> * <li>In normal hours, the max throughput is tuned between
>
>
> * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND} and
>
>
> * {@value #HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND}, using the
> formula "lower +
>
> * (higher - lower) * compactionPressure", where compactionPressure is
> in range [0.0, 1.0]</li>
>
> * </ul>
>
>
> */
> {code}
> The lower and upper bounds are 10MB/sec and 20MB/sec, respectively:
> {code:java}
> public static final String
> HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND =
> "hbase.hstore.compaction.throughput.higher.bound";
> private static final long
> DEFAULT_HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_HIGHER_BOUND =
> 20L * 1024 * 1024;
> public static final String
> HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND =
> "hbase.hstore.compaction.throughput.lower.bound";
> private static final long
> DEFAULT_HBASE_HSTORE_COMPACTION_MAX_THROUGHPUT_LOWER_BOUND =
> 10L * 1024 * 1024;
> {code}
> (In contrast, in PressureAwareFlushThroughputController the lower and upper
> bounds are 10x of those limits, at 100MB/sec and 200MB/sec, respectively.)
> In fairly light load scenarios we see compaction quickly falls behind and
> write clients are backed off or failing due to RegionTooBusy exceptions.
> Although compaction throughput becomes unbounded after the store reaches the
> blocking file count, in the lead up to this the default settings do not
> provide enough bandwidth to stave off blocking. The defaults should be
> increased.
> I'm not sure what new good defaults make sense. We could start by doubling
> them to 20MB/sec and 40MB/sec respectively. Might need to be doubled again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
