Hello Binglin Chang, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/2938
to look at the new patch set (#5).
Change subject: rpc: LIFO service queue
......................................................................
rpc: LIFO service queue
This adds a service queue implementation which differs from the previous
one in a few ways:
1) when the worker threads want to dequeue work and there is none available,
they push themselves onto a stack and wait for work on their own independent
condition variable rather than a central one. Because it uses a stack,
the wake-up behavior is LIFO rather than FIFO, which is preferable to
reduce context switches and increase cache locality (hot threads stay
hot).
2) when enqueueing a call, if there is already a waiting worker thread,
then the call is assigned directly to that thread without going
through the queue implementation. So, even if the queue
implementation is heavy-weight (such as our priority queue) it does
not affect the performance of workloads which have enough worker
threads allocated.
This improves rpc-bench throughput fairly substantially (~47% for sync,
66% for async):
./bin/rpc-bench -server_reactors 24 -async_call_concurrency 100
-worker_threads 40 -client_threads 24 -run_seconds 10
original:
------------------------
Mode: Sync
Client threads: 24
Worker threads: 40
Server reactors: 24
----------------------------------
Reqs/sec: 189006
User CPU per req: 21.9328us
Sys CPU per req: 31.633us
Ctx Sw. per req: 5.08355
Mode: Async
Client reactors: 24
Call concurrency: 100
Worker threads: 40
Server reactors: 24
----------------------------------
Reqs/sec: 202874
User CPU per req: 14.2749us
Sys CPU per req: 23.538us
Ctx Sw. per req: 2.7318
lifo queue without hash change:
-----------------------
Reqs/sec: 278730
User CPU per req: 19.8051us
Sys CPU per req: 21.9387us
Ctx Sw. per req: 4.04089
Mode: Async
Client reactors: 24
Call concurrency: 100
Worker threads: 40
Server reactors: 24
----------------------------------
Reqs/sec: 336000
User CPU per req: 11.0189us
Sys CPU per req: 13.6012us
Ctx Sw. per req: 1.48036
Change-Id: Ibd1eb677dd52f89683eb648b42918fcf51437215
---
M src/kudu/rpc/CMakeLists.txt
M src/kudu/rpc/service_pool.cc
M src/kudu/rpc/service_pool.h
A src/kudu/rpc/service_queue-test.cc
A src/kudu/rpc/service_queue.cc
M src/kudu/rpc/service_queue.h
6 files changed, 407 insertions(+), 84 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/38/2938/5
--
To view, visit http://gerrit.cloudera.org:8080/2938
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibd1eb677dd52f89683eb648b42918fcf51437215
Gerrit-PatchSet: 5
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <[email protected]>
Gerrit-Reviewer: Binglin Chang <[email protected]>
Gerrit-Reviewer: Henry Robinson <[email protected]>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <[email protected]>
