Alexey Serbin has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/24305 )
Change subject: KUDU-1865 reduce cross-thread allocations in RPC (part 1)
......................................................................
KUDU-1865 reduce cross-thread allocations in RPC (part 1)
This patch was originally posted by Todd [1]. I'm re-posting it
re-based and updated -- that's the best path forward; otherwise,
there are too many conflicts if trying to re-base and revv the
original patch in gerrit [1].
To verify that the number of cross-thread allocations is reduced with
this patch, I ran the target rpc-bench test scenario under customized
environment to trace tcmalloc allocations/deallocations, and then used
the Python script attached to KUDU-1865 ticket (updated a bit) to track
cross-thread allocations/deallocations in tcmalloc:
env TCMALLOC_TRACE=1 \
TCMALLOC_STACKTRACE_METHOD=generic_fp \
./bin/rpc-bench \
--server_reactors=1 \
--client_threads=1 \
--run_seconds=1 \
--gtest_filter='*BenchmarkCalls'
The results confirm that all the malloc/free asymmetry related to
ReactorTask objects is now gone. I'm planning to take care of
the rest of allocation/de-allocation asymmetry in hot RPC paths
in follow-up patches.
I ran the same test scenario with different parameters under a custom
environment to evaluate the total number of calls that lead to acquiring
locks guarding the per-size-classes' central free lists in tcmalloc.
I did so with the default and customized setting for the
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES environment variable. The
customization allowed to induce more activity on tcmalloc's central
free lists since the sizes of the free lists were shorter than with the
default setting.
The results of running rpc-bench under the perf utility on 8 CPU-core
machine are below. There is an improvement:
* ~1.37x times reduction in the total number of calls to the free lists
with the default setting of TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES
* ~1.06x times reduction in the total number of calls to the free lists
with the custom setting of TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES
As for requests-per-second performance, the improvement is minuscule.
The target test scenario doesn't create enough memory pressure and
concurrency to get to the point where too much of lock contention in
tcmalloc significantly affects RPC performance.
export TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=1048576
perf record -g -e 'syscalls:sys_enter_futex' \
./bin/rpc-bench \
--gtest_filter='*BenchmarkCalls' \
--server_reactors=2 \
--client_threads=4 \
--worker_threads=2 \
--run_seconds=60
perf script | \
grep -E 'tcmalloc::ThreadCache::(FetchFrom|ReleaseTo)CentralCache' | \
wc -l
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES unset (using default tcmalloc setting):
before: 499 488 491
after: 373 328 376
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=1048576
before: 13794 12727 12215
after: 12695 12137 11857
[1] https://gerrit.cloudera.org/#/c/5905/
Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Reviewed-on: http://gerrit.cloudera.org:8080/24305
Tested-by: Alexey Serbin <[email protected]>
Reviewed-by: Michael Smith <[email protected]>
Reviewed-by: Zoltan Martonka <[email protected]>
Reviewed-by: Marton Greber <[email protected]>
---
M src/kudu/rpc/connection.cc
M src/kudu/rpc/connection.h
M src/kudu/rpc/messenger.cc
M src/kudu/rpc/messenger.h
M src/kudu/rpc/negotiation.cc
M src/kudu/rpc/reactor.cc
M src/kudu/rpc/reactor.h
7 files changed, 190 insertions(+), 269 deletions(-)
Approvals:
Alexey Serbin: Verified
Michael Smith: Looks good to me, but someone else must approve
Zoltan Martonka: Looks good to me, but someone else must approve
Marton Greber: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/24305
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Gerrit-Change-Number: 24305
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Abhishek Chennaka <[email protected]>
Gerrit-Reviewer: Abhishek Rawat <[email protected]>
Gerrit-Reviewer: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Anonymous Coward (763)
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: David Rorke <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Kurt Deschler <[email protected]>
Gerrit-Reviewer: Marton Greber <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Zoltan Martonka <[email protected]>
