[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207095#comment-13207095
]
Jean-Daniel Cryans commented on HBASE-5162:
-------------------------------------------
Yeah to me the most interesting/challenging part is figuring out what's
considered a "heavy load" plus all the nuances for the phases.
Right now we have a pattern of "soft" and "hard" limits:
- memstore multiplier, they can get by default up to 2x their configured size
by default before blocking.
- compaction threshold and blocking values, the former is when we should kick
a compaction and the latter is when we are in a "too many store files" mode.
- global memstore barriers, the lower one starts force flushing the bigger
regions and the upper one blocks inserts.
I may be missing some others.
Do we want to keep them or piggy back? In all three cases we don't tell the
client anything until it blocks and instead we try to just take action ASAP.
One way could be to send resets once we're past the soft limit... but I'm not
sure if it's really going to help.
> Basic client pushback mechanism
> -------------------------------
>
> Key: HBASE-5162
> URL: https://issues.apache.org/jira/browse/HBASE-5162
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Fix For: 0.94.0
>
>
> The current blocking we do when we are close to some limits (memstores over
> the multiplier factor, too many store files, global memstore memory) is bad,
> too coarse and confusing. After hitting HBASE-5161, it really becomes obvious
> that we need something better.
> I did a little brainstorm with Stack, we came up quickly with two solutions:
> - Send some exception to the client, like OverloadedException, that's thrown
> when some situation happens like getting past the low memory barrier. It
> would be thrown when the client gets a handler and does some check while
> putting or deleting. The client would treat this a retryable exception but
> ideally wouldn't check .META. for a new location. It could be fancy and have
> multiple levels of pushback, like send the exception to 25% of the clients,
> and then go up if the situation persists. Should be "easy" to implement but
> we'll be using a lot more IO to send the payload over and over again (but at
> least it wouldn't sit in the RS's memory).
> - Send a message alongside a successful put or delete to tell the client to
> slow down a little, this way we don't have to do back and forth with the
> payload between the client and the server. It's a cleaner (I think) but more
> involved solution.
> In every case the RS should do very obvious things to notify the operators of
> this situation, through logs, web UI, metrics, etc.
> Other ideas?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
