[
https://issues.apache.org/jira/browse/ZOOKEEPER-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jay Shrauner updated ZOOKEEPER-1505:
------------------------------------
Description:
CommitProcessor has a single thread that both pulls requests off its queues and
runs all downstream processors. This is noticeably inefficient for
read-intensive workloads, which could be run concurrently. The trick is
handling write transactions. I propose multi-threading this code according to
the following two constraints
- each session must see its requests responded to in order
- all committed transactions must be handled in zxid order, across all
sessions
I believe these cover the only constraints we need to honor. In particular, I
believe we can relax the following:
- it does not matter if the read request in one session happens before or
after the write request in another session
With these constraints, I propose the following threads
- 1 primary queue servicing/work dispatching thread
- 0-N assignable worker threads, where a given session is always assigned to
the same worker thread
By assigning sessions always to the same worker thread (using a simple
sessionId mod number of worker threads), we guarantee the first constraint--
requests we push onto the thread queue are processed in order. The way we
guarantee the second constraint is we only allow a single commit transaction to
be in flight at a time--the queue servicing thread blocks while a commit
transaction is in flight, and when the transaction completes it clears the flag.
On a 32 core machine running Linux 2.6.38, achieved best performance with 32
worker threads for a 56% +/- 5% improvement in throughput (this improvement was
measured on top of that for ZOOKEEPER-1504, not in isolation).
New classes introduced in this patch are:
WorkerService (also in ZOOKEEPER-1504): ExecutorService wrapper that makes
worker threads daemon threads and names then in an easily debuggable manner.
Supports assignable threads (as used here) and non-assignable threads (as used
by NIOServerCnxnFactory).
was:
CommitProcessor has a single thread that both pulls requests off its queues and
runs all downstream processors. This is noticeably inefficient for
read-intensive workloads, which could be run concurrently. The trick is
handling write transactions. I propose multi-threading this code according to
the following two constraints
- each session must see its requests responded to in order
- all committed transactions must be handled in zxid order, across all
sessions
I believe these cover the only constraints we need to honor. In particular, I
believe we can relax the following:
- it does not matter if the read request in one session happens before or
after the write request in another session
With these constraints, I propose the following threads
- 1 primary queue servicing/work dispatching thread
- 0-N assignable worker threads, where a given session is always assigned to
the same worker thread
By assigning sessions always to the same worker thread (using a simple
sessionId mod number of worker threads), we guarantee the first
constraint--requests we push onto the thread queue are processed in order. The
way we guarantee the second constraint is we only allow a single commit
transaction to be in flight at a time--the queue servicing thread blocks while
a commit transaction is in flight, and when the transaction completes it clears
the flag.
On a 32 core machine running Linux 2.6.38, achieved best performance with 32
worker threads for a 56% +/- 5% improvement in throughput (this improvement was
measured on top of that for ZOOKEEPER-1504, not in isolation).
New classes introduced in this patch are:
WorkerService (also in ZOOKEEPER-1504): ExecutorService wrapper that makes
worker threads daemon threads and names then in an easily debuggable manner.
Supports assignable threads (as used here) and non-assignable threads (as used
by NIOServerCnxnFactory).
> Multi-thread CommitProcessor
> ----------------------------
>
> Key: ZOOKEEPER-1505
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1505
> Project: ZooKeeper
> Issue Type: Improvement
> Components: server
> Reporter: Jay Shrauner
> Assignee: Jay Shrauner
> Labels: performance, scaling
> Attachments: ZOOKEEPER-1505.patch
>
>
> CommitProcessor has a single thread that both pulls requests off its queues
> and runs all downstream processors. This is noticeably inefficient for
> read-intensive workloads, which could be run concurrently. The trick is
> handling write transactions. I propose multi-threading this code according to
> the following two constraints
> - each session must see its requests responded to in order
> - all committed transactions must be handled in zxid order, across all
> sessions
> I believe these cover the only constraints we need to honor. In particular, I
> believe we can relax the following:
> - it does not matter if the read request in one session happens before or
> after the write request in another session
> With these constraints, I propose the following threads
> - 1 primary queue servicing/work dispatching thread
> - 0-N assignable worker threads, where a given session is always assigned
> to the same worker thread
> By assigning sessions always to the same worker thread (using a simple
> sessionId mod number of worker threads), we guarantee the first constraint--
> requests we push onto the thread queue are processed in order. The way we
> guarantee the second constraint is we only allow a single commit transaction
> to be in flight at a time--the queue servicing thread blocks while a commit
> transaction is in flight, and when the transaction completes it clears the
> flag.
> On a 32 core machine running Linux 2.6.38, achieved best performance with 32
> worker threads for a 56% +/- 5% improvement in throughput (this improvement
> was measured on top of that for ZOOKEEPER-1504, not in isolation).
> New classes introduced in this patch are:
> WorkerService (also in ZOOKEEPER-1504): ExecutorService wrapper that
> makes worker threads daemon threads and names then in an easily debuggable
> manner. Supports assignable threads (as used here) and non-assignable threads
> (as used by NIOServerCnxnFactory).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
