Hello Kudu Jenkins, Abhishek Chennaka,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24305
to look at the new patch set (#3).
Change subject: KUDU-1865 reduce cross-thread allocations in RPC (part 1)
......................................................................
KUDU-1865 reduce cross-thread allocations in RPC (part 1)
This patch was originally posted by Todd [1]. I'm re-posting it
re-based and updated -- that's the best path forward; otherwise,
there are too many conflicts if trying to re-base and revv the
original patch in gerrit [1].
To verify that the number of cross-thread allocations is reduced with
this patch, I ran the target rpc-bench test scenario under customized
environment to trace tcmalloc allocations/deallocations, and then used
the Python script attached to KUDU-1865 ticket (updated a bit) to track
cross-thread allocations/deallocations in tcmalloc:
env TCMALLOC_TRACE=1 \
TCMALLOC_RECLAIM_MEMORY=0 \
TCMALLOC_STACKTRACE_METHOD=generic_fp \
./bin/rpc-bench \
--server_reactors=1 \
--client_threads=1 \
--run_seconds=1 \
--gtest_filter='*BenchmarkCalls'
The results confirm that all the malloc/free asymmetry related to
ReactorTask objects is now gone. I'm planning to take care of
the rest of allocation/de-allocation asymmetry in hot RPC paths
in follow-up patches.
I ran the same test scenario with different parameters under a custom
environment to evaluate the total number of calls that lead to acquiring
locks guarding the per-size-classes' central free lists in tcmalloc.
I did so with the default and customized setting for the
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES environment variable. The
customization allowed to induce more activity on tcmalloc's central
free lists since the sizes of the free lists were shorter than with the
default setting.
The result of running rpc-bench under the perf utility on 8 CPU-core
machine are below. While the numbers have shown some amount of
variation from run to run, the two samples below reflect the picture
seen after running the test scenario many times. Since I didn't specify
the address of the corresponding spinlock in tcmalloc, I used an ad-hoc
filtering via grep on the stacks output by 'perf script'.
There is an improvement, indeed:
* ~1.37x times reduction in the total number of calls to the free lists
with the default setting of TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES
* ~1.06x times reduction in the total number of calls to the free lists
with the custom setting of TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES
export TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=1048576
perf record -g -e 'syscalls:sys_enter_futex' \
./bin/rpc-bench \
--gtest_filter='*BenchmarkCalls' \
--server_reactors=2 \
--client_threads=4 \
--worker_threads=2 \
--run_seconds=60
perf script | \
grep -E 'tcmalloc::ThreadCache::(FetchFrom|ReleaseTo)CentralCache' | \
wc -l
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES unset (using default tcmalloc setting):
before: 499 488 491
after: 373 328 376
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=1048576
before: 13794 12727 12215
after: 12695 12137 11857
[1] https://gerrit.cloudera.org/#/c/5905/
Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
---
M src/kudu/rpc/connection.cc
M src/kudu/rpc/connection.h
M src/kudu/rpc/messenger.cc
M src/kudu/rpc/messenger.h
M src/kudu/rpc/negotiation.cc
M src/kudu/rpc/reactor.cc
M src/kudu/rpc/reactor.h
7 files changed, 189 insertions(+), 269 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/05/24305/3
--
To view, visit http://gerrit.cloudera.org:8080/24305
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Gerrit-Change-Number: 24305
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Abhishek Chennaka <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
