[
https://issues.apache.org/jira/browse/CASSANDRA-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13629415#comment-13629415
]
Nitsan Wakart commented on CASSANDRA-4718:
------------------------------------------
OK, off the bat without knowing anything about nothing (following on from
twitter chat with Jake Luciani @tjake):
I understand your process to be a pipeline of work being performed in stages
such that tasks have to complete one stage to go to the next. You need some
fast queue mechanism to deliver tasks between stages with minimum contention.
Two approaches come to mind:
1. To build on existing frameworks you will probably want to look at the
Disruptor (https://github.com/LMAX-Exchange/disruptor) and thread affinity
(https://github.com/peter-lawrey/Java-Thread-Affinity) this is going to work
well for rigid processes with an abundance of CPUs to play with, but with
planning can also work well for more restrictive environments.
2. Go wild and build something new!!! the typical work pool model will have all
the workers contending for work, this will create contention over the head of
the queue. We can try for a new algorithm where we have some way of removing
this contention for example by having an work arbitration thread deliver work
to the least busy worker or some other crazy scheme. I plan to give it a go and
see what happens :)
Option 1 is sure to deliver improved performance over your JDK variety of
collections. Thread affinity will help deliver more predictable results but
will require some layout algorithm for the available resources(i.e. make a
different plan for a dual socket 8 core than a 4 socket 4 core machine).
Affinity is likely to prove important in any case, but I am not aware of any
public framework making use of it.
An overlooked aspect of the Disruptor is the fact that it is a queue and an
object pool merged into one. To make the most of it you will need to have your
pipelined events form some super structure which will cycle through the process
and then get reused.
In any case, here's an initial brain dump, happy to get involved in the
solution if this sounds interesting.
> More-efficient ExecutorService for improved throughput
> ------------------------------------------------------
>
> Key: CASSANDRA-4718
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4718
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Jonathan Ellis
> Priority: Minor
> Attachments: baq vs trunk.png, PerThreadQueue.java
>
>
> Currently all our execution stages dequeue tasks one at a time. This can
> result in contention between producers and consumers (although we do our best
> to minimize this by using LinkedBlockingQueue).
> One approach to mitigating this would be to make consumer threads do more
> work in "bulk" instead of just one task per dequeue. (Producer threads tend
> to be single-task oriented by nature, so I don't see an equivalent
> opportunity there.)
> BlockingQueue has a drainTo(collection, int) method that would be perfect for
> this. However, no ExecutorService in the jdk supports using drainTo, nor
> could I google one.
> What I would like to do here is create just such a beast and wire it into (at
> least) the write and read stages. (Other possible candidates for such an
> optimization, such as the CommitLog and OutboundTCPConnection, are not
> ExecutorService-based and will need to be one-offs.)
> AbstractExecutorService may be useful. The implementations of
> ICommitLogExecutorService may also be useful. (Despite the name these are not
> actual ExecutorServices, although they share the most important properties of
> one.)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
