[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842302#comment-16842302
]
HBase QA commented on HBASE-14331:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HBASE-14331 does not apply to master. Rebase required? Wrong
Branch? See
https://yetus.apache.org/documentation/in-progress/precommit-patchnames for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-14331 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12767759/HBASE-14331-V6.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/347/console |
| Powered by | Apache Yetus 0.9.0 http://yetus.apache.org |
This message was automatically generated.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance, Scheduler
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Major
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331-V4.patch,
> HBASE-14331-V5.patch, HBASE-14331-V6.patch, HBASE-14331-V6.patch,
> HBASE-14331.patch, HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331507#comment-15331507
]
Hadoop QA commented on HBASE-14331:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} HBASE-14331 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.2.1/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12767759/HBASE-14331-V6.patch |
| JIRA Issue | HBASE-14331 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/2224/console |
| Powered by | Apache Yetus 0.2.1 http://yetus.apache.org |
This message was automatically generated.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331-V4.patch,
> HBASE-14331-V5.patch, HBASE-14331-V6.patch, HBASE-14331-V6.patch,
> HBASE-14331.patch, HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331253#comment-15331253
]
Hiroshi Ikeda commented on HBASE-14331:
---
bq. Do you want to close this issue then in favor of HBASE-14479?
I think these issues focus on different concerns (even though they would be
related). A big issue is hard to tackle.
bq. So, where would we do scheduling if the Reader threads ran the request
Hiroshi Ikeda?
Even if we adopt the strategy of making reader threads tend to directly execute
tasks, we can use queues to store excessive tasks waiting for their turn
(putting it aside whether we should do so).
Passing tasks between different threads will actually reduce latency (and that
would be justified in heavy congestion), but throughput will be harmed by
context switches and using CAS based queues might be not so harmful.
Rather, I worry how compatibility issues around callQueue in the way to make
reader directly execute tasks.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331-V4.patch,
> HBASE-14331-V5.patch, HBASE-14331-V6.patch, HBASE-14331-V6.patch,
> HBASE-14331.patch, HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331034#comment-15331034
]
stack commented on HBASE-14331:
---
Do you want to close this issue then in favor of HBASE-14479?
The handoff from Reader to Handler is a perf killer. Running more threads than
there are CPUs, at least in the read case where all is from cache, only makes
us slower.
But the handoff of Reader to Handler is where we do request scheduling.
Currently we have a scheduling that will try to background requests that come
from long-running Scans in favor of other request types ('deadline'). It could
be made do more sophisticated scheduling but we should probably put this aside
and just take on the recent FB addition of AdaptiveLifoCoDelCallQueue as our
default policy (See http://queue.acm.org/detail.cfm?id=2839461). It is a
general heuristic that works well for a broad set of loadings.
So, where would we do scheduling if the Reader threads ran the request
[~ikeda]? Thanks.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331-V4.patch,
> HBASE-14331-V5.patch, HBASE-14331-V6.patch, HBASE-14331-V6.patch,
> HBASE-14331.patch, HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331015#comment-15331015
]
stack commented on HBASE-14331:
---
I'd commit these [~ikeda]
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331-V4.patch,
> HBASE-14331-V5.patch, HBASE-14331-V6.patch, HBASE-14331-V6.patch,
> HBASE-14331.patch, HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14989030#comment-14989030
]
Hiroshi Ikeda commented on HBASE-14331:
---
I'm now thinking about directly executing tasks when the Leader/Followers
pattern is applied to RpcServer (HBASE-14479).
By removing threads in RpcExecutor with increasing the reader threads in
RpcServer, we expect the non-trivial overhead of park/unpark will be reduced.
Also bypassing the call-queues in a low load will remove the overhead of the
call-queues themselves.
In this case, the call-queue would be required to have methods offer and poll,
instead of put and take, which might be not compatible with the way I have done
in this issue.
Pseudo code I'm thinking:
{code}
final AtomicLong stateRef = new ...
void dispatch (task) {
boolean taskQueued = false;
while (true) {
long state = stateRef.get();
int taskThreads = toTaskThreads(state);
int queuedTasks = toQueuedTasks(state);
if (! taskQueued && (taskThreads > 20 || queuedTasks > 0)) {
callQueue.offer(task);
taskQueued = true;
}
boolean join = taskThreads <= 20 ||
(taskQueued ? queuedTasks + 1 : queuedTasks) > 1000;
if (join) {
if (stateRef.compareAndSet(toState(taskThreads + 1,
queuedTasks)) {
break;
}
} else {
assert taskQueued;
if (stateRef.compareAndSet(toState(taskThreads,
queuedTasks + 1)) {
return;
}
}
}
execute(taskQueued ? callQueue.poll() : task);
while (true) {
long state = stateRef.get();
int taskThreads = toTaskThreads(state);
int queuedTasks = toQueuedTasks(state);
if (queuedTasks > 1000 || queuedTasks > 0 && taskThreads <= 20)
{
if (stateRef.compareAndSet(toState(taskThreads,
queuedTasks - 1))) {
execute(callQueue.poll());
}
} else {
if (stateRef.compareAndSet(toState(taskThreads - 1,
queuedTasks))) {
return;
}
}
}
}
{code}
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331-V4.patch,
> HBASE-14331-V5.patch, HBASE-14331-V6.patch, HBASE-14331-V6.patch,
> HBASE-14331.patch, HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14973705#comment-14973705
]
ramkrishna.s.vasudevan commented on HBASE-14331:
+1 on creating sub tasks where ever possible.
bq. Avoid calculating the deadline value each time.
+1 on this. I noticed this and hence tried to create a wrapper and pre
calculate the deadline and priority and just use that in the comparison instead
of doing it every time.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331-V4.patch,
> HBASE-14331-V5.patch, HBASE-14331-V6.patch, HBASE-14331-V6.patch,
> HBASE-14331.patch, HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14966569#comment-14966569
]
Hiroshi Ikeda commented on HBASE-14331:
---
I have realized I can create a series of small patches:
1. Fix bugs about RWQueueRpcExecutor.
2. Use a high-scale counter for a frequently accessed counter in RpcExecutor.
3. Introduce CallQueue and make RpcExecutor use it.
4. Introduce CallQueueFacotry and introduce alternative RPC executors.
5. Avoid calculating the deadline value each time.
6. Introduce semaphore based implementation classes of CallQueue.
But some patches are still not so trivial despite the separation...
Is that meaningful to create sub tasks?
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331-V4.patch,
> HBASE-14331-V5.patch, HBASE-14331-V6.patch, HBASE-14331-V6.patch,
> HBASE-14331.patch, HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14966732#comment-14966732
]
Hadoop QA commented on HBASE-14331:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12767759/HBASE-14331-V6.patch
against master branch at commit 6e3b7af0efced7351163a5011a44bfc18851fce0.
ATTACHMENT ID: 12767759
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1751 checkstyle errors (more than the master's current 1747 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16138//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16138//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16138//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16138//console
This message is automatically generated.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331-V4.patch,
> HBASE-14331-V5.patch, HBASE-14331-V6.patch, HBASE-14331-V6.patch,
> HBASE-14331.patch, HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964791#comment-14964791
]
Hadoop QA commented on HBASE-14331:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12767545/HBASE-14331-V6.patch
against master branch at commit c1f0442045f44fcbb3935f9244794929a5d0caea.
ATTACHMENT ID: 12767545
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1751 checkstyle errors (more than the master's current 1747 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16113//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16113//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16113//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16113//console
This message is automatically generated.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331-V4.patch,
> HBASE-14331-V5.patch, HBASE-14331-V6.patch, HBASE-14331.patch,
> HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14902311#comment-14902311
]
ramkrishna.s.vasudevan commented on HBASE-14331:
Any updates on the single node testing or the server side testing. I think if
this has good improvement it would be a good add on.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331-V4.patch,
> HBASE-14331-V5.patch, HBASE-14331.patch, HBASE-14331.patch,
> SemaphoreBasedBlockingQueue.java, SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743369#comment-14743369
]
Hadoop QA commented on HBASE-14331:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12755703/HBASE-14331-V4.patch
against master branch at commit ceff3e242bc71130a7c0dea9828e6763fc0c9fec.
ATTACHMENT ID: 12755703
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:red}-1 javac{color}. The patch appears to cause mvn compile goal to
fail with Hadoop version 2.4.0.
Compilation errors resume:
[ERROR] COMPILATION ERROR :
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java:[58,64]
cannot find symbol
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallQueueFactories.java:[25,53]
package org.apache.hadoop.hbase.ipc.DeadlineCallQueue does not exist
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallQueueFactories.java:[33,21]
cannot find symbol
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java:[69,5]
method does not override or implement a method from a supertype
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallQueueFactories.java:[48,35]
cannot find symbol
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallQueueFactories.java:[52,22]
cannot find symbol
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallQueueFactories.java:[53,51]
cannot find symbol
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallQueueFactories.java:[73,35]
cannot find symbol
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallQueueFactories.java:[77,22]
cannot find symbol
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallQueueFactories.java:[78,36]
cannot find symbol
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallQueueFactories.java:[79,52]
cannot find symbol
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallQueueFactories.java:[80,33]
cannot find symbol
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on
project hbase-server: Compilation failure: Compilation failure:
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java:[58,64]
cannot find symbol
[ERROR] symbol: class DeadlineCalculator
[ERROR] location: class org.apache.hadoop.hbase.ipc.SimpleRpcScheduler
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallQueueFactories.java:[25,53]
package org.apache.hadoop.hbase.ipc.DeadlineCallQueue does not exist
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallQueueFactories.java:[33,21]
cannot find symbol
[ERROR] symbol: class DeadlineCalculator
[ERROR] location: class org.apache.hadoop.hbase.ipc.CallQueueFactories
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java:[69,5]
method does not override or implement a method from a supertype
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallQueueFactories.java:[48,35]
cannot find symbol
[ERROR] symbol: class DeadlineCalculator
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallQueueFactories.java:[52,22]
cannot find symbol
[ERROR] symbol: class DeadlineCallQueue
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallQueueFactories.java:[53,51]
cannot find symbol
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744805#comment-14744805
]
Hadoop QA commented on HBASE-14331:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12755879/HBASE-14331-V5.patch
against master branch at commit 63a2fb5143ab41d84b6b44001b63806534b26bdd.
ATTACHMENT ID: 12755879
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1838 checkstyle errors (more than the master's current 1834 errors).
{color:red}-1 findbugs{color}. The patch appears to cause Findbugs
(version 2.0.3) to fail.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 4 zombie test(s):
at
org.apache.hadoop.hbase.replication.regionserver.TestReplicationWALReaderManager.test(TestReplicationWALReaderManager.java:184)
at
org.apache.hadoop.hdfs.TestBlockReaderLocalLegacy.testBlockReaderLocalLegacyWithAppend(TestBlockReaderLocalLegacy.java:218)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15591//testReport/
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15591//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15591//console
This message is automatically generated.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331-V4.patch,
> HBASE-14331-V5.patch, HBASE-14331.patch, HBASE-14331.patch,
> SemaphoreBasedBlockingQueue.java, SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743314#comment-14743314
]
Hiroshi Ikeda commented on HBASE-14331:
---
{quote}
Personally I would prefer a series of smaller patches. I.e. can't we just
optionally replace the LinkedBlockingQueues used with with the SemaphoreBased*
based ones, and then add the nice factories in a 2nd patch?
{quote}
{{BlockingQueue}} has too many methods and its contracts. I found I just
created broken implementations. Fortunately, a few methods were required in the
logic, and I extracted them into a small interface {{CallQueue}}.
{quote}
Do you think that with this patch we can get rid of the multi call queue logic?
I.e. go back have a single queue (or one for reader and one for writers).
{quote}
I think there is no reason to use multi queues. One should use a single queue
until there is truly no choice to use multi queues.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331.patch,
> HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14742781#comment-14742781
]
Hiroshi Ikeda commented on HBASE-14331:
---
No I don't have a cluster I can freely use for tests, but HBASE-11355 showed a
problem of practical overhead around a call queue with accessing multiple
threads, achieving centain improvement with striping a lock. One should have
noticed that the call queue used a single lock for each put/take in
{{LinkingBlockingQueue}}.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331.patch,
> HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14742911#comment-14742911
]
Lars Hofhansl commented on HBASE-14331:
---
Thanks for looking into this [~ikeda]. I usually test these things on just a
single machine (I set up a single node HDFS, and a single node HBase on top of
that). Especially for region server optimization such as this one, that would
be sufficient.
Personally I would prefer a series of smaller patches. I.e. can't we just
optionally replace the LinkedBlockingQueues used with with the SemaphoreBased*
based ones, and then add the nice factories in a 2nd patch? But I see
[~ram_krish] seems to like the other code you added. That's good enough :)
Questions:
Do you think that with this patch we can get rid of the multi call queue logic?
I.e. go back have a single queue (or one for reader and one for writers).
I also really like your suggestion about not waiting when we reached the queue
limit, but return a failure code to the client. We can easily do that with your
semaphore approach.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331.patch,
> HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740280#comment-14740280
]
Hiroshi Ikeda commented on HBASE-14331:
---
That seems a mere bug.
AnnotationReadingPriorityFunction.getDeadline
{code}
public long getDeadline(RequestHeader header, Message param) {
String methodName = header.getMethodName();
if (methodName.equalsIgnoreCase("scan")) {
ScanRequest request = (ScanRequest)param;
if (!request.hasScannerId()) {
return 0;
}
// get the 'virtual time' of the scanner, and applies sqrt() to get a
// nice curve for the delay. More a scanner is used the less priority it
gets.
// The weight is used to have more control on the delay.
long vtime = rpcServices.getScannerVirtualTime(request.getScannerId());
return Math.round(Math.sqrt(vtime * scanVirtualTimeWeight));
}
return 0;
}
{code}
> long vtime = rpcServices.getScannerVirtualTime(request.getScannerId());
doesn't give the 'virtual time' for the request. That should be:
long vtime = request.getNextCallSeq();
And It is subtle but seems not to be changed. (But I also feel that method is
not so trivial to be frequently used in a comparator...)
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331.patch,
> HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740304#comment-14740304
]
Hiroshi Ikeda commented on HBASE-14331:
---
Sorry I referred a old blanch, and string comparison has been replaced to type
check.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331.patch,
> HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740547#comment-14740547
]
ramkrishna.s.vasudevan commented on HBASE-14331:
Have you run some performance numbers? Just to see if there is any
improvements with the Semaphore based queue? I can see the stand alone reports
but I am asking in terms of cluster set up and incoming requests? Lets wait for
some more additional comments.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331.patch,
> HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738350#comment-14738350
]
Hiroshi Ikeda commented on HBASE-14331:
---
{quote}
In SemaphoreBasedPriorityCallQueue, do you really need the private static
abstract class Container? Can the comparator directly work on the Element
and the ConcurrentkSkipListMap use that comparator which operates on the
element E directly?
{quote}
{{Container}} is needed, and you can add the same object twice or more, which
is expected for a queue.
{{Container}} is not needed to be abstract. That is a relic of trying to
implement the {{BlockingQueue}}, and another concrete class was used to extract
a sub map to implement the method {{contains}} etc.
{quote}
If the value part of ConcurrentkSkipListMap is not used can we use
ConcurrentSkipListSet.
{quote}
Using {{ConcurrentSkipListSet}} would be better if it had a method
{{addIfAbsent}} or something. It is possible that two containers for the same
element have the same hash code and adding a new container is failed.
{quote}
private static int compareAddition(Container c1, Container c2) {
Is this really needed? Can we just allow the natural ordering to happen here?
{quote}
Comparison on {{Container}} depends on the given comparator on contained
elements. It is possible to change {{Container}} to be a non-static class and
implement {{Comparable}} which refers the outside instance variable of the
given comparator, but I think that is more confusing.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331.patch,
> HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738372#comment-14738372
]
Hiroshi Ikeda commented on HBASE-14331:
---
{quote}
Container is needed, and you can add the same object twice or more, which is
expected for a queue.
{quote}
That is not only for the same object, but also for the objects which have the
same priority on the given comparator.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331.patch,
> HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738634#comment-14738634
]
Hiroshi Ikeda commented on HBASE-14331:
---
I don't know how to interpret the QA test result.
Its "zombies jstack extract" shows process which are not applied by my patch.
Does that not extract but show all of the simultaneously java process testing
other patches? I give up :-(
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331.patch,
> HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740080#comment-14740080
]
Hiroshi Ikeda commented on HBASE-14331:
---
One never expects that the ordering of a comparator is dynamically changed,
which causes inconsistent for collections based on a comparator.
{{SimpleRpcScheduler}} seems to use such a bad comparator.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331.patch,
> HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738244#comment-14738244
]
ramkrishna.s.vasudevan commented on HBASE-14331:
Went thro the patch. The CallQueueFactories are really nice and the way they
work seemlessly with the existing mode.
Few questions,
In SemaphoreBasedPriorityCallQueue, do you really need the private static
abstract class Container? Can the comparator directly work on the Element
and the ConcurrentkSkipListMap use that comparator which operates on the
element E directly?
If the value part of ConcurrentkSkipListMap is not used can we use
ConcurrentSkipListSet. I think even its constructor uses a Comparator.
{code}
private static int compareAddition(Container c1, Container c2) {
{code}
Is this really needed? Can we just allow the natural ordering to happen here?
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331.patch,
> HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736363#comment-14736363
]
Hadoop QA commented on HBASE-14331:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12754820/HBASE-14331.patch
against master branch at commit e95358a7fc3f554dcbb351c8b7295cafc01e8c23.
ATTACHMENT ID: 12754820
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1837 checkstyle errors (more than the master's current 1834 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.util.TestMiniClusterLoadEncoded
org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel
{color:red}-1 core zombie tests{color}. There are 7 zombie test(s):
at
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat2.testMRIncrementalLoadWithLocality(TestHFileOutputFormat2.java:400)
at
org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat.testWithMapReduceImpl(TestTableSnapshotInputFormat.java:225)
at
org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatTestBase.testWithMapReduce(TableSnapshotInputFormatTestBase.java:161)
at
org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat.testWithMapReduceAndOfflineHBaseMultiRegion(TestTableSnapshotInputFormat.java:148)
at
org.apache.hadoop.hbase.regionserver.wal.TestWALReplay.testReplayEditsAfterRegionMovedWithMultiCF(TestWALReplay.java:218)
at
org.apache.hadoop.hbase.TestZooKeeper.testLogSplittingAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:626)
at
org.apache.hadoop.hbase.mapreduce.MultiTableInputFormatTestBase.testScan(MultiTableInputFormatTestBase.java:255)
at
org.apache.hadoop.hbase.mapreduce.MultiTableInputFormatTestBase.testScanEmptyToEmpty(MultiTableInputFormatTestBase.java:196)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15490//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15490//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15490//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15490//console
This message is automatically generated.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331.patch, HBASE-14331.patch,
> SemaphoreBasedBlockingQueue.java, SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736406#comment-14736406
]
Hiroshi Ikeda commented on HBASE-14331:
---
About this patch.
This patch uses {{Semaphore}} based queue instead of {{BlockingQueue}}, unless
{{org.apache.hadoop.hbase.ipc.CallQueueFactories.useBlockingQueue=true}} in the
configuration, which uses adapters from {{BlockingQueue}}, and the property is
just intended for tests.
Also, this patch extends the behavior when a single queue is used:
(1) Even if you uses a single queue, additional single queues for read and scan
also can be used by {{hbase.ipc.server.callqueue.read.ratio}} (default 0,
indicating non-use) and {{hbase.ipc.server.callqueue.scan.ratio}} (default 0),
with distributing handlers by the ratios.
(2) When using a single queue, high priority handlers to deal with admin
requests, etc. also uses a just single queue. (You are using a single queue
because you think it is enough, don't you ?)
How many queues you uses is multiplied
{{hbase.ipc.server.callqueue.handler.factor=}} (default 0) by
{{hbase.regionserver.handler.count=}} (default 30) and if the result is
less than 2, a single queue is used for ordinary requests.
Calculating the distributed counts of queues and handlers for write/read/scan
is a little changed and it may a bit increase the sum of them as necessary
(instead of adjusting between write/read/scan. An existing class actually
contained bugs and unexpected logic).
{{org.apache.hadoop.hbase.util.Counter}} is used for counting the handlers to
simultaneously executing tasks. That is because incrementing and decrementing
the counter for each task may cause overhead (while I have been trying to
reduce such overhead by this issue !)
In order to shut out {{BlockingQueue}} from {{RpcExecutor}} I added a new
abstract package private class {{BlockingQueueRpcExecutor}}. One cannot easily
remove the protected methods because they are exposed to Coprocessor and
Phoenix via {{RWQueueRpcExecutor}} and {{BalancedQueueRpcExecutor}}.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331.patch, HBASE-14331.patch,
> SemaphoreBasedBlockingQueue.java, SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736678#comment-14736678
]
ramkrishna.s.vasudevan commented on HBASE-14331:
Great work. Going thro the Semaphore based code still.
Currently there is a RpcSchedulerFactory which creates a SimpleRpcScheduler
using the SimpleRpcSchedulerFactory. Now if i need to have a different type of
RpcScheduler which implements a new round robin way of scheduling then also I
need to support both FIFO and DEADLINe mode of the CallQueue. So will this
SempahoreBasedQueue allow me to extend its implementation to have a round robin
mode?
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331.patch, HBASE-14331.patch,
> SemaphoreBasedBlockingQueue.java, SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738022#comment-14738022
]
Hiroshi Ikeda commented on HBASE-14331:
---
bq. So will this SempahoreBasedQueue allow me to extend its implementation to
have a round robin mode?
Do you mean SemaphoreBasedQueue should be public?
The interface {{CallQueue}} is extracted from the current implementation
classes in order to replace the internal queue, and the methods of the
interface are just prepared for it. I think this interface is not enough for
other uses and its naming is a bit rough and vague, and I created it as
package-private as you can change it as necessary.
It would be better to create CAS based bounded FIFO/priority {{BlockingQueue}},
but it is difficult for me. Actually Java7 introduces the class
{{LinkedTransferQueue}}, which is CAS based *unbounded* FIFO {{BlockingQueue}},
and it might be possible to create other ones by imitating its logic, or in
future Java might introduce them.
With respect to RpcScheduler, do you mean it would be useful to use round robin
for multiplexed queues instead of random selection? I believe that it is the
best to use a single CAS queue (unless you change the logic of handling
requests by their priority). HBASE-11355 introduced multiplexed queue and
improved performance, but I think it is just sort of lock striping for lock
based queues, and it went the wrong way.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331.patch, HBASE-14331.patch,
> SemaphoreBasedBlockingQueue.java, SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738203#comment-14738203
]
Hadoop QA commented on HBASE-14331:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12755055/HBASE-14331-V3.patch
against master branch at commit a11bb2a933ad799546e7179fdf6ce75e3e22d44b.
ATTACHMENT ID: 12755055
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1837 checkstyle errors (more than the master's current 1834 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplicationSmallTests
{color:red}-1 core zombie tests{color}. There are 13 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testCompareStoreFileCount(TestPerColumnFamilyFlush.java:600)
at
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.testSSHCleanupDaugtherRegionsOfAbortedSplit(TestSplitTransactionOnCluster.java:1372)
at
org.apache.hadoop.hbase.client.TestReplicasClient.testSmallScanWithReplicas(TestReplicasClient.java:606)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15518//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15518//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15518//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15518//console
This message is automatically generated.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331.patch,
> HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14734507#comment-14734507
]
Hiroshi Ikeda commented on HBASE-14331:
---
Sorry, it has just struck me that my implementation of the {{remove(Object)}}
method is inconsistent with the other methods.
The method cannot throw {{UnsupportedOperationException}} in the contract of
{{BlockingQueue}}, and it would be better to define a new interface which has
just {{take}} and {{put}}, but {{BalancedQueueRpcExecutor}} and
{{RWQueueRpcExecutor}} are published to "Coprocesssor" and "Phoenix" with
unbelievably messy constructors so that it is quite difficult.
For now, I'll append the broken classes in package private.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> SemaphoreBasedBlockingQueue.java, SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735504#comment-14735504
]
Hadoop QA commented on HBASE-14331:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12754687/HBASE-14331.patch
against master branch at commit e95358a7fc3f554dcbb351c8b7295cafc01e8c23.
ATTACHMENT ID: 12754687
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1837 checkstyle errors (more than the master's current 1834 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15468//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15468//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15468//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15468//console
This message is automatically generated.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736169#comment-14736169
]
stack commented on HBASE-14331:
---
These are the tests that went zombie
kalashnikov:hbase.git stack$ python dev-support/findHangingTests.py
https://builds.apache.org/job/PreCommit-HBASE-Build/15468//consoleText
Fetching the console output from the URL
Printing hanging tests
Hanging test : org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot
Hanging test : org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot
Hanging test : org.apache.hadoop.hbase.mob.compactions.TestMobCompactor
Hanging test : org.apache.hadoop.hbase.mob.mapreduce.TestMobSweepJob
Hanging test : org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel
Hanging test : org.apache.hadoop.hbase.util.TestHBaseFsck
Hanging test : org.apache.hadoop.hbase.util.TestMiniClusterLoadEncoded
Hanging test : org.apache.hadoop.hbase.snapshot.TestExportSnapshot
Printing Failing tests
Failing test : org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential
Let me retry patch.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724142#comment-14724142
]
Ted Yu commented on HBASE-14331:
Can you put the three new Java classes in a single patch with proper Apache
License header ?
Looks like SemaphoreBasedLinkedBlockingQueue is better than
SemaphoreBasedPriorityBlockingQueue when waked read threads - tasks are high.
What's your interpretation ?
Thanks
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> SemaphoreBasedBlockingQueue.java, SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724156#comment-14724156
]
Ted Yu commented on HBASE-14331:
For the two BlockingQueue implementations, please add unit tests.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> SemaphoreBasedBlockingQueue.java, SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724631#comment-14724631
]
Hiroshi Ikeda commented on HBASE-14331:
---
{quote}
Can you put the three new Java classes in a single patch with proper Apache
License header ?
{quote}
{quote}
For the two BlockingQueue implementations, please add unit tests.
{quote}
The added classes are yet trial and they even have no package declaration. I'll
make them into a single patch with tests when I find how I should inject these
classes into the call-queue logic. It is the easiest to just replace existing
classes with the new classes, but it may be too rough.
{quote}
Looks like SemaphoreBasedLinkedBlockingQueue is better than
SemaphoreBasedPriorityBlockingQueue when waked read threads - tasks are high.
What's your interpretation ?
{quote}
The priority queue is not FIFO. Added elements in the queue are sorted with a
comparator, and the least element are given by polling. This process is
apparently heavier than just adding a element to the tail of the queue, and
comparing such queues is not fair.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> SemaphoreBasedBlockingQueue.java, SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14722832#comment-14722832
]
Hiroshi Ikeda commented on HBASE-14331:
---
{{BoundedConcurrentLinkedQueue}} is a class which just extends
{{ConcurrentLinkedQueue}} and adds the boundary (but it seems to have bugs
about {{Iterator.remove}} and methods internally using the
{{Iterator.remove]}), and doesn't implement BlockingQueue. That means
{{Semaphore}} or something is required for exclusive control after all.
a single callQueue related improvements
---
Key: HBASE-14331
URL: https://issues.apache.org/jira/browse/HBASE-14331
Project: HBase
Issue Type: Improvement
Components: IPC/RPC, Performance
Reporter: Hiroshi Ikeda
Priority: Minor
Attachments: CallQueuePerformanceTestApp.java
{{LinkedBlockingQueue}} well separates locks between the {{take}} method and
the {{put}} method, but not between takers, and not between putters. These
methods are implemented to take locks at the almost beginning of their logic.
HBASE-11355 introduces multiple call-queues to reduce such possible
congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
There are the other shortcomings of using {{BlockingQueue}}. When using
multiple queues, since {{BlockingQueue}} blocks threads it is required to
prepare enough threads for each queue. It is possible that there is a queue
starving for threads while there is another queue where threads are idle.
Even if you can tune parameters to avoid such situations, the tuning is not
so trivial.
I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14720048#comment-14720048
]
Ted Yu commented on HBASE-14331:
In HBase, there is BoundedConcurrentLinkedQueue
Can you expand your program by including BoundedConcurrentLinkedQueue ?
Thanks
a single callQueue related improvements
---
Key: HBASE-14331
URL: https://issues.apache.org/jira/browse/HBASE-14331
Project: HBase
Issue Type: Improvement
Components: IPC/RPC, Performance
Reporter: Hiroshi Ikeda
Priority: Minor
Attachments: CallQueuePerformanceTestApp.java
{{LinkedBlockingQueue}} well separates locks between the {{take}} method and
the {{put}} method, but not between takers, and not between putters. These
methods are implemented to take locks at the almost beginning of their logic.
HBASE-11355 introduces multiple call-queues to reduce such possible
congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
There are the other shortcomings of using {{BlockingQueue}}. When using
multiple queues, since {{BlockingQueue}} blocks threads it is required to
prepare enough threads for each queue. It is possible that there is a queue
starving for threads while there is another queue where threads are idle.
Even if you can tune parameters to avoid such situations, the tuning is not
so trivial.
I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
