[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesse Yates updated HBASE-5162:
-------------------------------
Attachment: java_HBASE-5162.patch
Worked up an initial implementation that doesn't actually break the RPC (I
think, even if client or the server has monitoring turned on), but avoids doing
the exception passing back to the client, which IMHO is the least clean way to
handle this as exceptions are for exceptional cases, not the common case.
Right now the implementation only covers Puts, but could definitely be extended
to cover other writes; I wanted to get some feedback on general style, etc.
before going for the full blown implementation (and all the general cleanup
associated with a 'real' patch).
As far as how it is put together...
Client has a backoff policy to take into consideration what the server is
saying.
This policy can take into account the current, max, and growth size of the
write buffer for figuring out how long to sleep. The server pressure is
unwrapped from the server in the Result as a MonitoredResult and then updated
on the client. The next put will then take into account that pressure when
attempting a put. The problem here is that the server can't tell all clients
that the pressure has gone down, but that trade-off is common given
traditional collision/backoff situations.
The client has also been given permission to grow to a multiplier of its
writeBufferSize, similar to the memstoremultiplier, allowing it to buffer more
writes. If a write is within the expansion range, we want to allow the client
to accept more writes while waiting/backing-off, so we launch a flusher thread
that after waiting the backoff time will flush the writes (singleton). This
gives us back-off as well as some flexiblilty on the client as to how much we
buffer. To disable the backoff behavior, its as simple as setting the
multiplier to 1, so the expansion = max.
Similarly, on the RS, we make the policy for monitoring pluggable and
optional to enable the blocking for a given threshold. The policy here can take
into account the current number of store files, the number where we will get
blocking and the current % full of the memstore (total size and max size). This
allows us to ensure that we only slow down based on a given policy (yet to be
implemented) but do get protection for the RS from malicious/over-eager clients.
This impl may be a little over the top, but it does cover both sides of the
on-the-wire story. Only things left to do are to extend this to other writes as
well as creating at least one (if not two or three) tunable policies for
figuring out blocking.
Thoughts?
> Basic client pushback mechanism
> -------------------------------
>
> Key: HBASE-5162
> URL: https://issues.apache.org/jira/browse/HBASE-5162
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Fix For: 0.94.0
>
> Attachments: java_HBASE-5162.patch
>
>
> The current blocking we do when we are close to some limits (memstores over
> the multiplier factor, too many store files, global memstore memory) is bad,
> too coarse and confusing. After hitting HBASE-5161, it really becomes obvious
> that we need something better.
> I did a little brainstorm with Stack, we came up quickly with two solutions:
> - Send some exception to the client, like OverloadedException, that's thrown
> when some situation happens like getting past the low memory barrier. It
> would be thrown when the client gets a handler and does some check while
> putting or deleting. The client would treat this a retryable exception but
> ideally wouldn't check .META. for a new location. It could be fancy and have
> multiple levels of pushback, like send the exception to 25% of the clients,
> and then go up if the situation persists. Should be "easy" to implement but
> we'll be using a lot more IO to send the payload over and over again (but at
> least it wouldn't sit in the RS's memory).
> - Send a message alongside a successful put or delete to tell the client to
> slow down a little, this way we don't have to do back and forth with the
> payload between the client and the server. It's a cleaner (I think) but more
> involved solution.
> In every case the RS should do very obvious things to notify the operators of
> this situation, through logs, web UI, metrics, etc.
> Other ideas?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
