[
https://issues.apache.org/jira/browse/HBASE-23744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17033872#comment-17033872
]
Geoffrey Jacoby commented on HBASE-23744:
-----------------------------------------
[~anoop.hbase][~xucang][~ndimiduk] - as far as I know, the "shut down writes by
setting the call queue length to 0" behavior is undocumented. And I agree with
Xu's point that it's a very subtle, less obvious way to do it.
It is, however, tested for in
TestSimpleRpcScehduler.testSoftAndHardQueueLimits:451-4. This test just happens
to pass because in practice the FastPathBalancedQueueRpcExecutor is always
getting lucky and never has a fast path handler in the stack when the test is
run. I came across the bug because an internal fork of HBase I use happened to
not always be lucky.
An alternative fix would be to declare the 0 call queue behavior undefined and
remove the 0 test logic from testSoftAndHardQueueLimits, but I don't think it's
a good idea to leave the test logic as-is without the fix in this patch making
the Fast executor work similarly to the other executor types.
And regardless of which path we take in this JIRA, a supported, well-documented
kill switch for writes is definitely something that I as an operator have
wished for before. :-) (For example, to block all writes before running a risky
hbck operation.)
> FastPathBalancedQueueRpcExecutor should enforce queue length of 0
> -----------------------------------------------------------------
>
> Key: HBASE-23744
> URL: https://issues.apache.org/jira/browse/HBASE-23744
> Project: HBase
> Issue Type: Bug
> Reporter: Geoffrey Jacoby
> Assignee: Geoffrey Jacoby
> Priority: Minor
>
> FastPathBalancedQueueRpcExecutor allows RPC requests to skip the RPC queue
> and get worked by an available handler under certain circumstances.
> Relatedly, the hbase.ipc.server.max.callqueue.length parameter can be set to
> 0, including dynamically. This can be useful to temporarily prevent writes on
> a cluster.
> When this is the case the executor is supposed to block all dispatching.
> However, the FastPathBalancedQueueRpcExecutor will still dispatch the request
> if one of the "fast path" handlers is available on its stack. This both isn't
> the desired effect, and also makes
> TestSimpleRpcScheduler.testSoftAndHardQueueLimits unstable when it checks the
> queue length 0 behavior.
> A simple fix is just to check max queue length > 0 before
> FastPathBalancedQueueRpcExecutor pops the fast handler off the stack.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
