[jira] [Commented] (HBASE-23744) FastPathBalancedQueueRpcExecutor should enforce queue length of 0
[
https://issues.apache.org/jira/browse/HBASE-23744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157073#comment-17157073
]
Hudson commented on HBASE-23744:
Results for branch branch-2
[build #2743 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2743/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2743/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2743/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2743/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2743/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> FastPathBalancedQueueRpcExecutor should enforce queue length of 0
> -
>
> Key: HBASE-23744
> URL: https://issues.apache.org/jira/browse/HBASE-23744
> Project: HBase
> Issue Type: Bug
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0
>
>
> FastPathBalancedQueueRpcExecutor allows RPC requests to skip the RPC queue
> and get worked by an available handler under certain circumstances.
> Relatedly, the hbase.ipc.server.max.callqueue.length parameter can be set to
> 0, including dynamically. This can be useful to temporarily prevent writes on
> a cluster.
> When this is the case the executor is supposed to block all dispatching.
> However, the FastPathBalancedQueueRpcExecutor will still dispatch the request
> if one of the "fast path" handlers is available on its stack. This both isn't
> the desired effect, and also makes
> TestSimpleRpcScheduler.testSoftAndHardQueueLimits unstable when it checks the
> queue length 0 behavior.
> A simple fix is just to check max queue length > 0 before
> FastPathBalancedQueueRpcExecutor pops the fast handler off the stack.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23744) FastPathBalancedQueueRpcExecutor should enforce queue length of 0
[
https://issues.apache.org/jira/browse/HBASE-23744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17156964#comment-17156964
]
Hudson commented on HBASE-23744:
Results for branch master
[build #1784 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1784/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1784/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1663//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1784/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1784/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> FastPathBalancedQueueRpcExecutor should enforce queue length of 0
> -
>
> Key: HBASE-23744
> URL: https://issues.apache.org/jira/browse/HBASE-23744
> Project: HBase
> Issue Type: Bug
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0
>
>
> FastPathBalancedQueueRpcExecutor allows RPC requests to skip the RPC queue
> and get worked by an available handler under certain circumstances.
> Relatedly, the hbase.ipc.server.max.callqueue.length parameter can be set to
> 0, including dynamically. This can be useful to temporarily prevent writes on
> a cluster.
> When this is the case the executor is supposed to block all dispatching.
> However, the FastPathBalancedQueueRpcExecutor will still dispatch the request
> if one of the "fast path" handlers is available on its stack. This both isn't
> the desired effect, and also makes
> TestSimpleRpcScheduler.testSoftAndHardQueueLimits unstable when it checks the
> queue length 0 behavior.
> A simple fix is just to check max queue length > 0 before
> FastPathBalancedQueueRpcExecutor pops the fast handler off the stack.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23744) FastPathBalancedQueueRpcExecutor should enforce queue length of 0
[
https://issues.apache.org/jira/browse/HBASE-23744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17156723#comment-17156723
]
Hudson commented on HBASE-23744:
Results for branch branch-1
[build #1324 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1324/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1324//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1324//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1324//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> FastPathBalancedQueueRpcExecutor should enforce queue length of 0
> -
>
> Key: HBASE-23744
> URL: https://issues.apache.org/jira/browse/HBASE-23744
> Project: HBase
> Issue Type: Bug
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0
>
>
> FastPathBalancedQueueRpcExecutor allows RPC requests to skip the RPC queue
> and get worked by an available handler under certain circumstances.
> Relatedly, the hbase.ipc.server.max.callqueue.length parameter can be set to
> 0, including dynamically. This can be useful to temporarily prevent writes on
> a cluster.
> When this is the case the executor is supposed to block all dispatching.
> However, the FastPathBalancedQueueRpcExecutor will still dispatch the request
> if one of the "fast path" handlers is available on its stack. This both isn't
> the desired effect, and also makes
> TestSimpleRpcScheduler.testSoftAndHardQueueLimits unstable when it checks the
> queue length 0 behavior.
> A simple fix is just to check max queue length > 0 before
> FastPathBalancedQueueRpcExecutor pops the fast handler off the stack.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23744) FastPathBalancedQueueRpcExecutor should enforce queue length of 0
[ https://issues.apache.org/jira/browse/HBASE-23744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17156529#comment-17156529 ] Viraj Jasani commented on HBASE-23744: -- Merged changes to master, branch-2 and branch-1. Since github is down, have not yet confirmed commit in UI but I was able to push the changes manually. > FastPathBalancedQueueRpcExecutor should enforce queue length of 0 > - > > Key: HBASE-23744 > URL: https://issues.apache.org/jira/browse/HBASE-23744 > Project: HBase > Issue Type: Bug >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Minor > > FastPathBalancedQueueRpcExecutor allows RPC requests to skip the RPC queue > and get worked by an available handler under certain circumstances. > Relatedly, the hbase.ipc.server.max.callqueue.length parameter can be set to > 0, including dynamically. This can be useful to temporarily prevent writes on > a cluster. > When this is the case the executor is supposed to block all dispatching. > However, the FastPathBalancedQueueRpcExecutor will still dispatch the request > if one of the "fast path" handlers is available on its stack. This both isn't > the desired effect, and also makes > TestSimpleRpcScheduler.testSoftAndHardQueueLimits unstable when it checks the > queue length 0 behavior. > A simple fix is just to check max queue length > 0 before > FastPathBalancedQueueRpcExecutor pops the fast handler off the stack. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23744) FastPathBalancedQueueRpcExecutor should enforce queue length of 0
[ https://issues.apache.org/jira/browse/HBASE-23744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033967#comment-17033967 ] Nick Dimiduk commented on HBASE-23744: -- bq. And regardless of which path we take in this JIRA, a supported, well-documented kill switch for writes is definitely something that I as an operator have wished for before. How about filing a ticket requesting a read-only mode? You can outline exactly what you're after there, we can has out requirements, > FastPathBalancedQueueRpcExecutor should enforce queue length of 0 > - > > Key: HBASE-23744 > URL: https://issues.apache.org/jira/browse/HBASE-23744 > Project: HBase > Issue Type: Bug >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Minor > > FastPathBalancedQueueRpcExecutor allows RPC requests to skip the RPC queue > and get worked by an available handler under certain circumstances. > Relatedly, the hbase.ipc.server.max.callqueue.length parameter can be set to > 0, including dynamically. This can be useful to temporarily prevent writes on > a cluster. > When this is the case the executor is supposed to block all dispatching. > However, the FastPathBalancedQueueRpcExecutor will still dispatch the request > if one of the "fast path" handlers is available on its stack. This both isn't > the desired effect, and also makes > TestSimpleRpcScheduler.testSoftAndHardQueueLimits unstable when it checks the > queue length 0 behavior. > A simple fix is just to check max queue length > 0 before > FastPathBalancedQueueRpcExecutor pops the fast handler off the stack. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23744) FastPathBalancedQueueRpcExecutor should enforce queue length of 0
[ https://issues.apache.org/jira/browse/HBASE-23744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033872#comment-17033872 ] Geoffrey Jacoby commented on HBASE-23744: - [~anoop.hbase][~xucang][~ndimiduk] - as far as I know, the "shut down writes by setting the call queue length to 0" behavior is undocumented. And I agree with Xu's point that it's a very subtle, less obvious way to do it. It is, however, tested for in TestSimpleRpcScehduler.testSoftAndHardQueueLimits:451-4. This test just happens to pass because in practice the FastPathBalancedQueueRpcExecutor is always getting lucky and never has a fast path handler in the stack when the test is run. I came across the bug because an internal fork of HBase I use happened to not always be lucky. An alternative fix would be to declare the 0 call queue behavior undefined and remove the 0 test logic from testSoftAndHardQueueLimits, but I don't think it's a good idea to leave the test logic as-is without the fix in this patch making the Fast executor work similarly to the other executor types. And regardless of which path we take in this JIRA, a supported, well-documented kill switch for writes is definitely something that I as an operator have wished for before. :-) (For example, to block all writes before running a risky hbck operation.) > FastPathBalancedQueueRpcExecutor should enforce queue length of 0 > - > > Key: HBASE-23744 > URL: https://issues.apache.org/jira/browse/HBASE-23744 > Project: HBase > Issue Type: Bug >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Minor > > FastPathBalancedQueueRpcExecutor allows RPC requests to skip the RPC queue > and get worked by an available handler under certain circumstances. > Relatedly, the hbase.ipc.server.max.callqueue.length parameter can be set to > 0, including dynamically. This can be useful to temporarily prevent writes on > a cluster. > When this is the case the executor is supposed to block all dispatching. > However, the FastPathBalancedQueueRpcExecutor will still dispatch the request > if one of the "fast path" handlers is available on its stack. This both isn't > the desired effect, and also makes > TestSimpleRpcScheduler.testSoftAndHardQueueLimits unstable when it checks the > queue length 0 behavior. > A simple fix is just to check max queue length > 0 before > FastPathBalancedQueueRpcExecutor pops the fast handler off the stack. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23744) FastPathBalancedQueueRpcExecutor should enforce queue length of 0
[ https://issues.apache.org/jira/browse/HBASE-23744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033355#comment-17033355 ] Anoop Sam John commented on HBASE-23744: bq. hbase.ipc.server.max.callqueue.length parameter can be set to 0, including dynamically. This can be useful to temporarily prevent writes on a cluster. Any pointer in the doc or so where we say this? > FastPathBalancedQueueRpcExecutor should enforce queue length of 0 > - > > Key: HBASE-23744 > URL: https://issues.apache.org/jira/browse/HBASE-23744 > Project: HBase > Issue Type: Bug >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Minor > > FastPathBalancedQueueRpcExecutor allows RPC requests to skip the RPC queue > and get worked by an available handler under certain circumstances. > Relatedly, the hbase.ipc.server.max.callqueue.length parameter can be set to > 0, including dynamically. This can be useful to temporarily prevent writes on > a cluster. > When this is the case the executor is supposed to block all dispatching. > However, the FastPathBalancedQueueRpcExecutor will still dispatch the request > if one of the "fast path" handlers is available on its stack. This both isn't > the desired effect, and also makes > TestSimpleRpcScheduler.testSoftAndHardQueueLimits unstable when it checks the > queue length 0 behavior. > A simple fix is just to check max queue length > 0 before > FastPathBalancedQueueRpcExecutor pops the fast handler off the stack. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23744) FastPathBalancedQueueRpcExecutor should enforce queue length of 0
[ https://issues.apache.org/jira/browse/HBASE-23744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17025408#comment-17025408 ] Xu Cang commented on HBASE-23744: - The idea you showed in PR makes sense to me. Just wondering, is there another way to properly "temporarily prevent writes on cluster" such as disabling PRC handling ? Setting the callqueue.length to 0 is a bit subtle to indicate the fact we want to disable writes. If we do so, could you please add a one sentence comment in FastPathBalancedQueueRpcExecutor class? thanks. [~gjacoby] > FastPathBalancedQueueRpcExecutor should enforce queue length of 0 > - > > Key: HBASE-23744 > URL: https://issues.apache.org/jira/browse/HBASE-23744 > Project: HBase > Issue Type: Bug >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Minor > > FastPathBalancedQueueRpcExecutor allows RPC requests to skip the RPC queue > and get worked by an available handler under certain circumstances. > Relatedly, the hbase.ipc.server.max.callqueue.length parameter can be set to > 0, including dynamically. This can be useful to temporarily prevent writes on > a cluster. > When this is the case the executor is supposed to block all dispatching. > However, the FastPathBalancedQueueRpcExecutor will still dispatch the request > if one of the "fast path" handlers is available on its stack. This both isn't > the desired effect, and also makes > TestSimpleRpcScheduler.testSoftAndHardQueueLimits unstable when it checks the > queue length 0 behavior. > A simple fix is just to check max queue length > 0 before > FastPathBalancedQueueRpcExecutor pops the fast handler off the stack. -- This message was sent by Atlassian Jira (v8.3.4#803005)
