[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209725#comment-13209725
]
[email protected] commented on HBASE-5162:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3930/
-----------------------------------------------------------
Review request for hbase, Michael Stack, Jean-Daniel Cryans, and Lars Hofhansl.
Summary
-------
Under heavy write load, HBase will create a saw-tooth pattern in accepting
writes. This is due to the I/O in minor compactions not being able to keep up
with the write load. Specifically, the memstore is attempting to flush while we
are attempting to do a minor compaction, leading to blocking _all_ writes.
Instead, we need to have the option of graceful degradation mechanism.
This patch supports both a short-term,adjustable server-side write blocking as
well as client-side back-off to help alleviate temporary memstore pressure.
This addresses bug HBASE-5162.
https://issues.apache.org/jira/browse/HBASE-5162
Diffs
-----
src/main/java/org/apache/hadoop/hbase/HConstants.java 763fe89
src/main/java/org/apache/hadoop/hbase/client/BackoffPolicy.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/HTable.java 57605e6
src/main/java/org/apache/hadoop/hbase/client/MonitoredResult.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/NoServerBackoffPolicy.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 25cb31d
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7d7be3c
src/main/java/org/apache/hadoop/hbase/regionserver/MemstorePressureMonitor.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/OperationStatus.java
1b94ab5
src/main/java/org/apache/hadoop/hbase/regionserver/PressureMonitor.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/SimpleMemStorePressureMonitor.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/StoreUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/client/TestClientBackOff.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStorePressureMonitor.java
PRE-CREATION
Diff: https://reviews.apache.org/r/3930/diff
Testing
-------
Thanks,
Jesse
> Basic client pushback mechanism
> -------------------------------
>
> Key: HBASE-5162
> URL: https://issues.apache.org/jira/browse/HBASE-5162
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: java_HBASE-5162.patch
>
>
> The current blocking we do when we are close to some limits (memstores over
> the multiplier factor, too many store files, global memstore memory) is bad,
> too coarse and confusing. After hitting HBASE-5161, it really becomes obvious
> that we need something better.
> I did a little brainstorm with Stack, we came up quickly with two solutions:
> - Send some exception to the client, like OverloadedException, that's thrown
> when some situation happens like getting past the low memory barrier. It
> would be thrown when the client gets a handler and does some check while
> putting or deleting. The client would treat this a retryable exception but
> ideally wouldn't check .META. for a new location. It could be fancy and have
> multiple levels of pushback, like send the exception to 25% of the clients,
> and then go up if the situation persists. Should be "easy" to implement but
> we'll be using a lot more IO to send the payload over and over again (but at
> least it wouldn't sit in the RS's memory).
> - Send a message alongside a successful put or delete to tell the client to
> slow down a little, this way we don't have to do back and forth with the
> payload between the client and the server. It's a cleaner (I think) but more
> involved solution.
> In every case the RS should do very obvious things to notify the operators of
> this situation, through logs, web UI, metrics, etc.
> Other ideas?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
