[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348432#comment-15348432 ] Sean Busbey commented on HBASE-15146: - [~ikeda] please have this discussion either on dev@hbase or in a new JIRA. this one has been resolved for ~6 months, so it isn't likely to get a continued audience. > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15146-v7.patch, HBASE-15146-v8.patch, > HBASE-15146-v8.patch, HBASE-15146.0.patch, HBASE-15146.1.patch, > HBASE-15146.2.patch, HBASE-15146.3.patch, HBASE-15146.4.patch, > HBASE-15146.5.patch, HBASE-15146.6.patch > > > Blocking on the epoll thread is awful. The new rpc scheduler can have lots of > different queues. Those queues have different capacity limits. Currently the > dispatch method can block trying to add the the blocking queue in any of the > schedulers. > This causes readers to block, tcp acks are delayed, and everything slows down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348059#comment-15348059 ] Hiroshi Ikeda commented on HBASE-15146: --- If all clients are well-behaved and patiently wait their response, blocking reader threads doesn't become a vicious circle. The latency becomes bad depending on how congestion is there, and that is expected as a part of gradually reducing performance. It would be true that the latency becomes unacceptable one in heavy load, but who knows whether blocked tasks can't be answered in a reasonable time? The preceding tasks might be quite light. These days there is a plan to adopt AdaptiveLifoCoDelCallQueue, and the task might be immediately executed after release. When I think about a full-queue error response, I don't image what a client should do when it receives the error response, even if the latency is much low. Such clients probably send again and again their request until they draw a winning ticket, and the whole latency to get a fruitful result will be unpredictably longer. The reader threads continue to rob worker threads of CPU time, with excessive overhead of context switches, and the worker threads would hardly execute tasks and resolve the full-queue condition. Ironically that becomes harder when the response is quicker. BTW, in the first place, I'm now doubting the queue should be bounded. If all clients are well-behaved and wait their response, the number of requests are naturally bounded under the number of connections, and the native resources of sockets will be run out in front of running out heap. AdaptiveLifoCoDelCallQueue also seems under the assumption that all requests can be queued. There are some cheat clients that don't wait their response and send multiple requests to simultaneously execute. I think we can count the number of simultaneous requests for each connection, and roughly cap it probably depending on the number of the connections (excluding idle connections who have no request?) and the number of the queued requests with some threshold, etc. > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15146-v7.patch, HBASE-15146-v8.patch, > HBASE-15146-v8.patch, HBASE-15146.0.patch, HBASE-15146.1.patch, > HBASE-15146.2.patch, HBASE-15146.3.patch, HBASE-15146.4.patch, > HBASE-15146.5.patch, HBASE-15146.6.patch > > > Blocking on the epoll thread is awful. The new rpc scheduler can have lots of > different queues. Those queues have different capacity limits. Currently the > dispatch method can block trying to add the the blocking queue in any of the > schedulers. > This causes readers to block, tcp acks are delayed, and everything slows down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343958#comment-15343958 ] Elliott Clark commented on HBASE-15146: --- bq.That seems based on the fact that there are some clients which don't wait the response of the previously sent request Nope, it's all about pushing back before blocking on queueing things that can't be answered in a reasonable time. You can get the queue full with enough single clients. The reader is non-blocking. The fact that we were blocking and not acking tcp streams until after full requests are completed was not a preformance benefit in any way. It was just wrong. It lost perf in multiple different ways. To see this in the extreme set the handlers to one, and the call queue length to one. Then tcpdump and see what happens. > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15146-v7.patch, HBASE-15146-v8.patch, > HBASE-15146-v8.patch, HBASE-15146.0.patch, HBASE-15146.1.patch, > HBASE-15146.2.patch, HBASE-15146.3.patch, HBASE-15146.4.patch, > HBASE-15146.5.patch, HBASE-15146.6.patch > > > Blocking on the epoll thread is awful. The new rpc scheduler can have lots of > different queues. Those queues have different capacity limits. Currently the > dispatch method can block trying to add the the blocking queue in any of the > schedulers. > This causes readers to block, tcp acks are delayed, and everything slows down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[
https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343934#comment-15343934
]
Hiroshi Ikeda commented on HBASE-15146:
---
That seems based on the fact that there are some clients which don't wait the
response of the previously sent request. Even though a server sends the busy
signal to clients, that is less effective unless such badly behaved clients
stop sending their requests.
{quote}
bq.Selector.select immediately causes a context switch when an event occurs,
Yes it does, and you want to get the reader threads back to the calling select
as fast as possible.
{quote}
I meant Select.select has the possibility to greatly reduce throughput by
overhead of context switches.
> Don't block on Reader threads queueing to a scheduler queue
> ---
>
> Key: HBASE-15146
> URL: https://issues.apache.org/jira/browse/HBASE-15146
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Blocker
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-15146-v7.patch, HBASE-15146-v8.patch,
> HBASE-15146-v8.patch, HBASE-15146.0.patch, HBASE-15146.1.patch,
> HBASE-15146.2.patch, HBASE-15146.3.patch, HBASE-15146.4.patch,
> HBASE-15146.5.patch, HBASE-15146.6.patch
>
>
> Blocking on the epoll thread is awful. The new rpc scheduler can have lots of
> different queues. Those queues have different capacity limits. Currently the
> dispatch method can block trying to add the the blocking queue in any of the
> schedulers.
> This causes readers to block, tcp acks are delayed, and everything slows down.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343790#comment-15343790 ] Elliott Clark commented on HBASE-15146: --- bq.In general, gradually reducing performance is rather preferable in heavy load. We've found the exact opposite many many times. Pushing back on the client is a well know and understood load shedding mechanism. That allows the server to take what it can handle and no more. By contrast every time the server promises to do work that it can't handle things get worse. GC gets worse, queue call times get worse, and it becomes a cycle. That continues until a regionserver is in-operable. Removing threads that can call select leads to multiple seconds where no tcp acks are sent. On loaded servers we saw all reader threads completely stop any network selects at all. bq.Selector.select immediately causes a context switch when an event occurs, and this patch might make worse performance in such subtle heavy congestion. Yes it does, and you want to get the reader threads back to the calling select as fast as possible. That's the most basic tenant of an event loop. What was happening was that the threads would stop for multiple seconds because the queues were full. That meant the event loop is stopped. > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15146-v7.patch, HBASE-15146-v8.patch, > HBASE-15146-v8.patch, HBASE-15146.0.patch, HBASE-15146.1.patch, > HBASE-15146.2.patch, HBASE-15146.3.patch, HBASE-15146.4.patch, > HBASE-15146.5.patch, HBASE-15146.6.patch > > > Blocking on the epoll thread is awful. The new rpc scheduler can have lots of > different queues. Those queues have different capacity limits. Currently the > dispatch method can block trying to add the the blocking queue in any of the > schedulers. > This causes readers to block, tcp acks are delayed, and everything slows down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343753#comment-15343753 ] Hiroshi Ikeda commented on HBASE-15146: --- Oh, I just realized I misunderstood classification annotations. Thanks. > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15146-v7.patch, HBASE-15146-v8.patch, > HBASE-15146-v8.patch, HBASE-15146.0.patch, HBASE-15146.1.patch, > HBASE-15146.2.patch, HBASE-15146.3.patch, HBASE-15146.4.patch, > HBASE-15146.5.patch, HBASE-15146.6.patch > > > Blocking on the epoll thread is awful. The new rpc scheduler can have lots of > different queues. Those queues have different capacity limits. Currently the > dispatch method can block trying to add the the blocking queue in any of the > schedulers. > This causes readers to block, tcp acks are delayed, and everything slows down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343720#comment-15343720 ] Sean Busbey commented on HBASE-15146: - RWQueueRpcExecutor is IA.LimitedPrivate, which is allowed to change on minor releases. AFAICT, that's what happened here. Since this change has already gone into releases, please open a new JIRA if you think something needs to change in the solution here. > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15146-v7.patch, HBASE-15146-v8.patch, > HBASE-15146-v8.patch, HBASE-15146.0.patch, HBASE-15146.1.patch, > HBASE-15146.2.patch, HBASE-15146.3.patch, HBASE-15146.4.patch, > HBASE-15146.5.patch, HBASE-15146.6.patch > > > Blocking on the epoll thread is awful. The new rpc scheduler can have lots of > different queues. Those queues have different capacity limits. Currently the > dispatch method can block trying to add the the blocking queue in any of the > schedulers. > This causes readers to block, tcp acks are delayed, and everything slows down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343718#comment-15343718 ] Hiroshi Ikeda commented on HBASE-15146: --- Changing the published method RWQueueRpcExecutor.dispatch etc. breaks compatibility against explicitly declaring the contract by InterfaceAudience. It doesn't seem to make sense that reader threads hold the initiative from worker threads and continue to just reject incoming requests. Moreover, in my old experience, Selector.select immediately causes a context switch when an event occurs, and this patch might make worse performance in such subtle heavy congestion. In general, gradually reducing performance is rather preferable in heavy load. > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15146-v7.patch, HBASE-15146-v8.patch, > HBASE-15146-v8.patch, HBASE-15146.0.patch, HBASE-15146.1.patch, > HBASE-15146.2.patch, HBASE-15146.3.patch, HBASE-15146.4.patch, > HBASE-15146.5.patch, HBASE-15146.6.patch > > > Blocking on the epoll thread is awful. The new rpc scheduler can have lots of > different queues. Those queues have different capacity limits. Currently the > dispatch method can block trying to add the the blocking queue in any of the > schedulers. > This causes readers to block, tcp acks are delayed, and everything slows down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121674#comment-15121674 ] Hudson commented on HBASE-15146: FAILURE: Integrated in HBase-Trunk_matrix #664 (See [https://builds.apache.org/job/HBase-Trunk_matrix/664/]) HBASE-15146 Don't block on Reader threads queueing to a scheduler queue (eclark: rev 138b754671d51d3f494adc250ab0cb9e085c858a) * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RWQueueRpcExecutor.java * hbase-client/src/test/java/org/apache/hadoop/hbase/exceptions/TestClientExceptionsUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/BalancedQueueRpcExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcScheduler.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java * hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/ClientExceptionsUtil.java * hbase-client/src/main/java/org/apache/hadoop/hbase/CallQueueTooBigException.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/FifoRpcScheduler.java * hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15146-v7.patch, HBASE-15146-v8.patch, > HBASE-15146-v8.patch, HBASE-15146.0.patch, HBASE-15146.1.patch, > HBASE-15146.2.patch, HBASE-15146.3.patch, HBASE-15146.4.patch, > HBASE-15146.5.patch, HBASE-15146.6.patch > > > Blocking on the epoll thread is awful. The new rpc scheduler can have lots of > different queues. Those queues have different capacity limits. Currently the > dispatch method can block trying to add the the blocking queue in any of the > schedulers. > This causes readers to block, tcp acks are delayed, and everything slows down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122113#comment-15122113 ] Hudson commented on HBASE-15146: SUCCESS: Integrated in HBase-1.2 #522 (See [https://builds.apache.org/job/HBase-1.2/522/]) HBASE-15146 Don't block on Reader threads queueing to a scheduler queue (eclark: rev 51998b9eb5c97265c93a83047d897eb17c7a58ca) * hbase-client/src/main/java/org/apache/hadoop/hbase/CallQueueTooBigException.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcScheduler.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/BalancedQueueRpcExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RWQueueRpcExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java * hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java * hbase-client/src/test/java/org/apache/hadoop/hbase/exceptions/TestClientExceptionsUtil.java * hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/ClientExceptionsUtil.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/FifoRpcScheduler.java > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15146-v7.patch, HBASE-15146-v8.patch, > HBASE-15146-v8.patch, HBASE-15146.0.patch, HBASE-15146.1.patch, > HBASE-15146.2.patch, HBASE-15146.3.patch, HBASE-15146.4.patch, > HBASE-15146.5.patch, HBASE-15146.6.patch > > > Blocking on the epoll thread is awful. The new rpc scheduler can have lots of > different queues. Those queues have different capacity limits. Currently the > dispatch method can block trying to add the the blocking queue in any of the > schedulers. > This causes readers to block, tcp acks are delayed, and everything slows down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122140#comment-15122140 ] Hudson commented on HBASE-15146: SUCCESS: Integrated in HBase-1.3-IT #466 (See [https://builds.apache.org/job/HBase-1.3-IT/466/]) HBASE-15146 Don't block on Reader threads queueing to a scheduler queue (eclark: rev 421fe24e9bb925e6199cc02118a5314458caeb38) * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RWQueueRpcExecutor.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java * hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/ClientExceptionsUtil.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java * hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/BalancedQueueRpcExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcScheduler.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java * hbase-client/src/test/java/org/apache/hadoop/hbase/exceptions/TestClientExceptionsUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/FifoRpcScheduler.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-client/src/main/java/org/apache/hadoop/hbase/CallQueueTooBigException.java > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15146-v7.patch, HBASE-15146-v8.patch, > HBASE-15146-v8.patch, HBASE-15146.0.patch, HBASE-15146.1.patch, > HBASE-15146.2.patch, HBASE-15146.3.patch, HBASE-15146.4.patch, > HBASE-15146.5.patch, HBASE-15146.6.patch > > > Blocking on the epoll thread is awful. The new rpc scheduler can have lots of > different queues. Those queues have different capacity limits. Currently the > dispatch method can block trying to add the the blocking queue in any of the > schedulers. > This causes readers to block, tcp acks are delayed, and everything slows down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121959#comment-15121959 ] Hudson commented on HBASE-15146: SUCCESS: Integrated in HBase-1.3 #519 (See [https://builds.apache.org/job/HBase-1.3/519/]) HBASE-15146 Don't block on Reader threads queueing to a scheduler queue (eclark: rev 421fe24e9bb925e6199cc02118a5314458caeb38) * hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RWQueueRpcExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java * hbase-client/src/main/java/org/apache/hadoop/hbase/CallQueueTooBigException.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/FifoRpcScheduler.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/BalancedQueueRpcExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcScheduler.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java * hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/ClientExceptionsUtil.java * hbase-client/src/test/java/org/apache/hadoop/hbase/exceptions/TestClientExceptionsUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15146-v7.patch, HBASE-15146-v8.patch, > HBASE-15146-v8.patch, HBASE-15146.0.patch, HBASE-15146.1.patch, > HBASE-15146.2.patch, HBASE-15146.3.patch, HBASE-15146.4.patch, > HBASE-15146.5.patch, HBASE-15146.6.patch > > > Blocking on the epoll thread is awful. The new rpc scheduler can have lots of > different queues. Those queues have different capacity limits. Currently the > dispatch method can block trying to add the the blocking queue in any of the > schedulers. > This causes readers to block, tcp acks are delayed, and everything slows down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121769#comment-15121769 ] Hudson commented on HBASE-15146: SUCCESS: Integrated in HBase-1.2-IT #412 (See [https://builds.apache.org/job/HBase-1.2-IT/412/]) HBASE-15146 Don't block on Reader threads queueing to a scheduler queue (eclark: rev 51998b9eb5c97265c93a83047d897eb17c7a58ca) * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RWQueueRpcExecutor.java * hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/ClientExceptionsUtil.java * hbase-client/src/test/java/org/apache/hadoop/hbase/exceptions/TestClientExceptionsUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcScheduler.java * hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/FifoRpcScheduler.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/BalancedQueueRpcExecutor.java * hbase-client/src/main/java/org/apache/hadoop/hbase/CallQueueTooBigException.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15146-v7.patch, HBASE-15146-v8.patch, > HBASE-15146-v8.patch, HBASE-15146.0.patch, HBASE-15146.1.patch, > HBASE-15146.2.patch, HBASE-15146.3.patch, HBASE-15146.4.patch, > HBASE-15146.5.patch, HBASE-15146.6.patch > > > Blocking on the epoll thread is awful. The new rpc scheduler can have lots of > different queues. Those queues have different capacity limits. Currently the > dispatch method can block trying to add the the blocking queue in any of the > schedulers. > This causes readers to block, tcp acks are delayed, and everything slows down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119628#comment-15119628 ] stack commented on HBASE-15146: --- bq. Not a single unit test repeats between the sets. I'll do a weeding again (unless folks are up for fixing the flakies). I put back the stochastic test but it is flakey. Will purge again. > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 1.2.0 > > Attachments: HBASE-15146-v7.patch, HBASE-15146-v8.patch, > HBASE-15146-v8.patch, HBASE-15146.0.patch, HBASE-15146.1.patch, > HBASE-15146.2.patch, HBASE-15146.3.patch, HBASE-15146.4.patch, > HBASE-15146.5.patch, HBASE-15146.6.patch > > > Blocking on the epoll thread is awful. The new rpc scheduler can have lots of > different queues. Those queues have different capacity limits. Currently the > dispatch method can block trying to add the the blocking queue in any of the > schedulers. > This causes readers to block, tcp acks are delayed, and everything slows down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[
https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118871#comment-15118871
]
Hadoop QA commented on HBASE-15146:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 28s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 27s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 33s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 17s
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 149m 14s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 15s
{color} | {color:green} hbase-client in the patch passed with JDK v1.7.0_91.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 131m 54s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 2m
52s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 354m 33s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.hbase.mapreduce.TestImportExport |
| | hadoop.hbase.master.balancer.TestStochasticLoadBalancer |
| JDK v1.8.0_66 Timed out junit tests
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119321#comment-15119321 ] Elliott Clark commented on HBASE-15146: --- Not a single unit test repeats between the sets. :-/ Looks like these are flakey. I'll run this on a couple of different machines. If everything passes on all the places I can test I'll commit. > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 1.2.0 > > Attachments: HBASE-15146-v7.patch, HBASE-15146-v8.patch, > HBASE-15146-v8.patch, HBASE-15146.0.patch, HBASE-15146.1.patch, > HBASE-15146.2.patch, HBASE-15146.3.patch, HBASE-15146.4.patch, > HBASE-15146.5.patch, HBASE-15146.6.patch > > > Blocking on the epoll thread is awful. The new rpc scheduler can have lots of > different queues. Those queues have different capacity limits. Currently the > dispatch method can block trying to add the the blocking queue in any of the > schedulers. > This causes readers to block, tcp acks are delayed, and everything slows down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118156#comment-15118156 ] Gary Helmling commented on HBASE-15146: --- +1 on v8 patch. > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 1.2.0 > > Attachments: HBASE-15146-v7.patch, HBASE-15146-v8.patch, > HBASE-15146.0.patch, HBASE-15146.1.patch, HBASE-15146.2.patch, > HBASE-15146.3.patch, HBASE-15146.4.patch, HBASE-15146.5.patch, > HBASE-15146.6.patch > > > Blocking on the epoll thread is awful. The new rpc scheduler can have lots of > different queues. Those queues have different capacity limits. Currently the > dispatch method can block trying to add the the blocking queue in any of the > schedulers. > This causes readers to block, tcp acks are delayed, and everything slows down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[
https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118365#comment-15118365
]
Hadoop QA commented on HBASE-15146:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 26s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 19s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 32s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 31s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 16s
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 110m 15s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 38s
{color} | {color:green} hbase-client in the patch passed with JDK v1.7.0_91.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 25s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 183m 43s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Timed out junit tests |
org.apache.hadoop.hbase.client.TestFromClientSide |
| | org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor |
| |
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[
https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15116595#comment-15116595
]
Hadoop QA commented on HBASE-15146:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 28s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 38s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 36s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 18s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 53s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 5s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 57s
{color} | {color:red} Patch generated 3 new checkstyle issues in hbase-client
(total was 44, now 47). {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 10s
{color} | {color:red} Patch generated 12 new checkstyle issues in hbase-server
(total was 105, now 117). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 48s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 16s
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 172m 26s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 37s
{color} | {color:green} hbase-client in the patch passed with JDK v1.7.0_91.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 161m 9s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 42s
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 413m 31s {color}
|
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113175#comment-15113175 ] Elliott Clark commented on HBASE-15146: --- Working on a new version of the patch to address the issues that Gary was talking about. > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 1.2.0 > > Attachments: HBASE-15146.0.patch, HBASE-15146.1.patch, > HBASE-15146.2.patch, HBASE-15146.3.patch, HBASE-15146.4.patch, > HBASE-15146.5.patch > > > Blocking on the epoll thread is awful. The new rpc scheduler can have lots of > different queues. Those queues have different capacity limits. Currently the > dispatch method can block trying to add the the blocking queue in any of the > schedulers. > This causes readers to block, tcp acks are delayed, and everything slows down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[
https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113393#comment-15113393
]
Hadoop QA commented on HBASE-15146:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 49s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 51s
{color} | {color:red} Patch generated 7 new checkstyle issues in hbase-client
(total was 44, now 51). {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 3m 59s
{color} | {color:red} Patch generated 12 new checkstyle issues in hbase-server
(total was 105, now 117). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
21m 54s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 44s {color}
| {color:red} hbase-client in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 51s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 58s {color}
| {color:red} hbase-client in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 59s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 19s
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 53s {color}
| {color:black} {color}
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110673#comment-15110673 ] Sean Busbey commented on HBASE-15146: - please ping me when this goes in. > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 1.2.0 > > Attachments: HBASE-15146.0.patch, HBASE-15146.1.patch, > HBASE-15146.2.patch > > > Blocking on the epoll thread is awful. The new rpc scheduler can have lots of > different queues. Those queues have different capacity limits. Currently the > dispatch method can block trying to add the the blocking queue in any of the > schedulers. > This causes readers to block, tcp acks are delayed, and everything slows down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[
https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110653#comment-15110653
]
Hadoop QA commented on HBASE-15146:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 46s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 6s
{color} | {color:red} Patch generated 12 new checkstyle issues in hbase-server
(total was 105, now 117). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
20m 45s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 12s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 3s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 72m 29s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0 Failed junit tests | hadoop.hbase.ipc.TestAsyncIPC |
| | hadoop.hbase.ipc.TestIPC |
| JDK v1.7.0_79 Failed junit tests | hadoop.hbase.ipc.TestAsyncIPC |
| | hadoop.hbase.ipc.TestIPC |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12783577/HBASE-15146.2.patch |
| JIRA Issue | HBASE-15146 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110929#comment-15110929 ] stack commented on HBASE-15146: --- You have thread dumps showing how awful this is [~eclark]? If queue is 'too big', we go back to the client. Client retries? You have it running over there? How does it look when we hit the limit? Patch LGTM > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 1.2.0 > > Attachments: HBASE-15146.0.patch, HBASE-15146.1.patch, > HBASE-15146.2.patch > > > Blocking on the epoll thread is awful. The new rpc scheduler can have lots of > different queues. Those queues have different capacity limits. Currently the > dispatch method can block trying to add the the blocking queue in any of the > schedulers. > This causes readers to block, tcp acks are delayed, and everything slows down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110975#comment-15110975 ] Elliott Clark commented on HBASE-15146: --- I have some thread dumps (I'll get them when I get to work). They are a little deceptive since they are only blocking 5 threads. However since those are the threads feeding everything it's like everything is slow. bq.If queue is 'too big', we go back to the client. Client retries? Yeah client retries. bq.You have it running over there? How does it look when we hit the limit? Yeah have it running with the first version. However a co-worker found the bug about not reducing queue size. So need to get that up. > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 1.2.0 > > Attachments: HBASE-15146.0.patch, HBASE-15146.1.patch, > HBASE-15146.2.patch, HBASE-15146.3.patch > > > Blocking on the epoll thread is awful. The new rpc scheduler can have lots of > different queues. Those queues have different capacity limits. Currently the > dispatch method can block trying to add the the blocking queue in any of the > schedulers. > This causes readers to block, tcp acks are delayed, and everything slows down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111216#comment-15111216 ] Elliott Clark commented on HBASE-15146: --- bq.please ping me when this goes in. Will do > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 1.2.0 > > Attachments: HBASE-15146.0.patch, HBASE-15146.1.patch, > HBASE-15146.2.patch, HBASE-15146.3.patch > > > Blocking on the epoll thread is awful. The new rpc scheduler can have lots of > different queues. Those queues have different capacity limits. Currently the > dispatch method can block trying to add the the blocking queue in any of the > schedulers. > This causes readers to block, tcp acks are delayed, and everything slows down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[
https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111397#comment-15111397
]
Hadoop QA commented on HBASE-15146:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 49s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 11s
{color} | {color:red} Patch generated 12 new checkstyle issues in hbase-server
(total was 105, now 117). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
20m 12s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 103m 42s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color}
|
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 39s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 232m 41s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0 Failed junit tests |
hadoop.hbase.security.token.TestGenerateDelegationToken |
| | hadoop.hbase.ipc.TestDelayedRpc |
| JDK v1.7.0_79 Failed junit tests | hadoop.hbase.ipc.TestDelayedRpc |
| JDK v1.7.0_79 Timed out junit tests | org.apache.hadoop.hbase.TestZooKeeper |
| | org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
| |
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111785#comment-15111785 ] Gary Helmling commented on HBASE-15146: --- In AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(), we don't pass along the Throwable when calling updateCachedLocations(). Still looking into where receiveGlobalFailure() is called from, but I'm concerned this could be another hole for clearing meta cache, while could cause some unnecessary load re-scanning meta. Seems like we need closer checking of the multi-action results, so that if we _only_ get non-cache busting exception(s), then we pass that along to updateCacheLocations so we don't unnecessarily clear. We can drop the IllegalStateExceptions that were previously added to method signatures, since we're now using offer() on the queue. FifoRpcScheduler needs to decrement queueSize in the runnable that is submitted. > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 1.2.0 > > Attachments: HBASE-15146.0.patch, HBASE-15146.1.patch, > HBASE-15146.2.patch, HBASE-15146.3.patch, HBASE-15146.4.patch, > HBASE-15146.5.patch > > > Blocking on the epoll thread is awful. The new rpc scheduler can have lots of > different queues. Those queues have different capacity limits. Currently the > dispatch method can block trying to add the the blocking queue in any of the > schedulers. > This causes readers to block, tcp acks are delayed, and everything slows down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[
https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111788#comment-15111788
]
Hadoop QA commented on HBASE-15146:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 3s {color}
| {color:red} HBASE-15146 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/latest/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12783735/HBASE-15146.5.patch |
| JIRA Issue | HBASE-15146 |
| Powered by | Apache Yetus 0.1.0 http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/244/console |
This message was automatically generated.
> Don't block on Reader threads queueing to a scheduler queue
> ---
>
> Key: HBASE-15146
> URL: https://issues.apache.org/jira/browse/HBASE-15146
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Blocker
> Fix For: 1.2.0
>
> Attachments: HBASE-15146.0.patch, HBASE-15146.1.patch,
> HBASE-15146.2.patch, HBASE-15146.3.patch, HBASE-15146.4.patch,
> HBASE-15146.5.patch
>
>
> Blocking on the epoll thread is awful. The new rpc scheduler can have lots of
> different queues. Those queues have different capacity limits. Currently the
> dispatch method can block trying to add the the blocking queue in any of the
> schedulers.
> This causes readers to block, tcp acks are delayed, and everything slows down.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[
https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111752#comment-15111752
]
Hadoop QA commented on HBASE-15146:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s {color}
| {color:red} HBASE-15146 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/latest/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12783727/HBASE-15146.4.patch |
| JIRA Issue | HBASE-15146 |
| Powered by | Apache Yetus 0.1.0 http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/243/console |
This message was automatically generated.
> Don't block on Reader threads queueing to a scheduler queue
> ---
>
> Key: HBASE-15146
> URL: https://issues.apache.org/jira/browse/HBASE-15146
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Blocker
> Fix For: 1.2.0
>
> Attachments: HBASE-15146.0.patch, HBASE-15146.1.patch,
> HBASE-15146.2.patch, HBASE-15146.3.patch, HBASE-15146.4.patch
>
>
> Blocking on the epoll thread is awful. The new rpc scheduler can have lots of
> different queues. Those queues have different capacity limits. Currently the
> dispatch method can block trying to add the the blocking queue in any of the
> schedulers.
> This causes readers to block, tcp acks are delayed, and everything slows down.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111753#comment-15111753 ] stack commented on HBASE-15146: --- Patch LGTM. We only decrement when we get to max queue size. Were you going to add another place to decrement? > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 1.2.0 > > Attachments: HBASE-15146.0.patch, HBASE-15146.1.patch, > HBASE-15146.2.patch, HBASE-15146.3.patch, HBASE-15146.4.patch > > > Blocking on the epoll thread is awful. The new rpc scheduler can have lots of > different queues. Those queues have different capacity limits. Currently the > dispatch method can block trying to add the the blocking queue in any of the > schedulers. > This causes readers to block, tcp acks are delayed, and everything slows down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15109883#comment-15109883 ] Gary Helmling commented on HBASE-15146: --- We should also modify RWQueueRpcExecutor. > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Priority: Blocker > Attachments: HBASE-15146.0.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[ https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15109887#comment-15109887 ] Jean-Marc Spaggiari commented on HBASE-15146: - when logging the exception saying the queue might be too small, is it possible to print its current size? > Don't block on Reader threads queueing to a scheduler queue > --- > > Key: HBASE-15146 > URL: https://issues.apache.org/jira/browse/HBASE-15146 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Priority: Blocker > Attachments: HBASE-15146.0.patch, HBASE-15146.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue
[
https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15109902#comment-15109902
]
Enis Soztutar commented on HBASE-15146:
---
Why not {{BlockingQ.offer()}} instead of {{add()}}. Throwing exception is
costly.
> Don't block on Reader threads queueing to a scheduler queue
> ---
>
> Key: HBASE-15146
> URL: https://issues.apache.org/jira/browse/HBASE-15146
> Project: HBase
> Issue Type: Bug
>Reporter: Elliott Clark
>Priority: Blocker
> Attachments: HBASE-15146.0.patch, HBASE-15146.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
