Hello Kudu Jenkins, Abhishek Chennaka,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24305
to look at the new patch set (#2).
Change subject: KUDU-1865 reduce cross-thread allocations in RPC hot paths
(part 1)
......................................................................
KUDU-1865 reduce cross-thread allocations in RPC hot paths (part 1)
This patch was originally posted by Todd [1]. I'm re-posting it
with updates -- that's the best path forward, otherwise there are too
many conflicts if trying to re-base and revv the original patch
in gerrit [1].
To verify that the number of cross-thread allocations is reduced with
this patch, I ran the target rpc-bench test scenario under customized
environment to trace tcmalloc allocations/deallocations, and then used
the Python script attached to KUDU-1865 ticket (with modifications)
to track cross-thread allocations/deallocations in tcmalloc:
env TCMALLOC_TRACE=1 \
TCMALLOC_RECLAIM_MEMORY=0 \
TCMALLOC_STACKTRACE_METHOD=generic_fp \
./bin/rpc-bench \
--server_reactors=1 \
--client_threads=1 \
--run_seconds=1 \
--gtest_filter='*BenchmarkCalls'
The results confirm that all the malloc/free asymmetry related to
ReactorTask objects is now gone. I'm planning to take care of
the rest of allocation/de-allocation asymmetry in hot RPC paths
in follow-up patches.
I ran the same test scenario with different parameters under customized
environment to evaluate the total amount of lock contention and number
of I/O wait syscalls: the TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES env
variable was customized to induce more activity on tcmalloc's central
free list.
The result of running rpc-bench under the perf utility on 8 CPU-core
machine are below. While the numbers have shown some amount of
variation, the two samples below reflect the picture seen after running
the test scenario many times. Since I didn't specify the address of
the speenlock in tcmalloc, this provides quite a fuzzy picture on
the actual improvement in libtcmalloc's contention in particular,
but it shows overall improvement.
export TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=2097152
perf stat \
-e 'syscalls:sys_enter_futex' \
-e 'syscalls:sys_enter_epoll_wait*' \
./bin/rpc-bench \
--gtest_filter='*BenchmarkCalls' \
--server_reactors=2 \
--client_threads=4 \
--worker_threads=2 \
--run_seconds=60
before:
11,907,775 syscalls:sys_enter_futex
8,983,211 syscalls:sys_enter_epoll_wait
60.014426125 seconds time elapsed
41.389530000 seconds user
128.630758000 seconds sys
after:
9,951,308 syscalls:sys_enter_futex
7,910,928 syscalls:sys_enter_epoll_wait
60.014275152 seconds time elapsed
37.602754000 seconds user
116.237592000 seconds sys
[1] https://gerrit.cloudera.org/#/c/5905/
Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
---
M src/kudu/rpc/connection.cc
M src/kudu/rpc/connection.h
M src/kudu/rpc/messenger.cc
M src/kudu/rpc/messenger.h
M src/kudu/rpc/negotiation.cc
M src/kudu/rpc/reactor.cc
M src/kudu/rpc/reactor.h
7 files changed, 189 insertions(+), 269 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/05/24305/2
--
To view, visit http://gerrit.cloudera.org:8080/24305
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Gerrit-Change-Number: 24305
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Abhishek Chennaka <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
