[
https://issues.apache.org/jira/browse/KUDU-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18082454#comment-18082454
]
ASF subversion and git services commented on KUDU-1865:
-------------------------------------------------------
Commit 6ab30abe247627112ff04a64b5a6481277dc607a in kudu's branch
refs/heads/master from Alexey Serbin
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=6ab30abe2 ]
KUDU-1865 reduce cross-thread allocations in RPC (part 1)
This patch was originally posted by Todd [1]. I'm re-posting it
re-based and updated -- that's the best path forward; otherwise,
there are too many conflicts if trying to re-base and revv the
original patch in gerrit [1].
To verify that the number of cross-thread allocations is reduced with
this patch, I ran the target rpc-bench test scenario under customized
environment to trace tcmalloc allocations/deallocations, and then used
the Python script attached to KUDU-1865 ticket (updated a bit) to track
cross-thread allocations/deallocations in tcmalloc:
env TCMALLOC_TRACE=1 \
TCMALLOC_STACKTRACE_METHOD=generic_fp \
./bin/rpc-bench \
--server_reactors=1 \
--client_threads=1 \
--run_seconds=1 \
--gtest_filter='*BenchmarkCalls'
The results confirm that all the malloc/free asymmetry related to
ReactorTask objects is now gone. I'm planning to take care of
the rest of allocation/de-allocation asymmetry in hot RPC paths
in follow-up patches.
I ran the same test scenario with different parameters under a custom
environment to evaluate the total number of calls that lead to acquiring
locks guarding the per-size-classes' central free lists in tcmalloc.
I did so with the default and customized setting for the
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES environment variable. The
customization allowed to induce more activity on tcmalloc's central
free lists since the sizes of the free lists were shorter than with the
default setting.
The results of running rpc-bench under the perf utility on 8 CPU-core
machine are below. There is an improvement:
* ~1.37x times reduction in the total number of calls to the free lists
with the default setting of TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES
* ~1.06x times reduction in the total number of calls to the free lists
with the custom setting of TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES
As for requests-per-second performance, the improvement is minuscule.
The target test scenario doesn't create enough memory pressure and
concurrency to get to the point where too much of lock contention in
tcmalloc significantly affects RPC performance.
export TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=1048576
perf record -g -e 'syscalls:sys_enter_futex' \
./bin/rpc-bench \
--gtest_filter='*BenchmarkCalls' \
--server_reactors=2 \
--client_threads=4 \
--worker_threads=2 \
--run_seconds=60
perf script | \
grep -E 'tcmalloc::ThreadCache::(FetchFrom|ReleaseTo)CentralCache' | \
wc -l
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES unset (using default tcmalloc setting):
before: 499 488 491
after: 373 328 376
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=1048576
before: 13794 12727 12215
after: 12695 12137 11857
[1] https://gerrit.cloudera.org/#/c/5905/
Change-Id: I86a2acbd1d8cb724728034c4e91907c99cbfe32e
Reviewed-on: http://gerrit.cloudera.org:8080/24305
Tested-by: Alexey Serbin <[email protected]>
Reviewed-by: Michael Smith <[email protected]>
Reviewed-by: Zoltan Martonka <[email protected]>
Reviewed-by: Marton Greber <[email protected]>
> Create fast path for RespondSuccess() in KRPC
> ---------------------------------------------
>
> Key: KUDU-1865
> URL: https://issues.apache.org/jira/browse/KUDU-1865
> Project: Kudu
> Issue Type: Improvement
> Components: perf, rpc
> Reporter: Sailesh Mukil
> Priority: Major
> Labels: perfomance, rpc, scalability
> Attachments: alloc-pattern.py, cross-thread.txt
>
>
> A lot of RPCs just respond with RespondSuccess() which returns the exact
> payload every time. This takes the same path as any other response by
> ultimately calling Connection::QueueResponseForCall() which has a few small
> allocations. These small allocations (and their corresponding deallocations)
> are called quite frequently (once for every IncomingCall) and end up taking
> quite some time in the kernel (traversing the free list, spin locks etc.)
> This was found when [~mmokhtar] ran some profiles on Impala over KRPC on a 20
> node cluster and found the following:
> The exact % of time spent is hard to quantify from the profiles, but these
> were the among the top 5 of the slowest stacks:
> {code:java}
> impalad ! tcmalloc::CentralFreeList::ReleaseToSpans - [unknown source file]
> impalad ! tcmalloc::CentralFreeList::ReleaseListToSpans + 0x1a - [unknown
> source file]
> impalad ! tcmalloc::CentralFreeList::InsertRange + 0x3b - [unknown source
> file]
> impalad ! tcmalloc::ThreadCache::ReleaseToCentralCache + 0x103 - [unknown
> source file]
> impalad ! tcmalloc::ThreadCache::Scavenge + 0x3e - [unknown source file]
> impalad ! operator delete + 0x329 - [unknown source file]
> impalad ! __gnu_cxx::new_allocator<kudu::Slice>::deallocate + 0x4 -
> new_allocator.h:110
> impalad ! std::_Vector_base<kudu::Slice,
> std::allocator<kudu::Slice>>::_M_deallocate + 0x5 - stl_vector.h:178
> impalad ! ~_Vector_base + 0x4 - stl_vector.h:160
> impalad ! ~vector - stl_vector.h:425 <----Deleting
> 'slices' vector
> impalad ! kudu::rpc::Connection::QueueResponseForCall + 0xac -
> connection.cc:433
> impalad ! kudu::rpc::InboundCall::Respond + 0xfa - inbound_call.cc:133
> impalad ! kudu::rpc::InboundCall::RespondSuccess + 0x43 - inbound_call.cc:77
> impalad ! kudu::rpc::RpcContext::RespondSuccess + 0x1f7 - rpc_context.cc:66
> ..
> {code}
> {code:java}
> impalad ! tcmalloc::CentralFreeList::FetchFromOneSpans - [unknown source file]
> impalad ! tcmalloc::CentralFreeList::RemoveRange + 0xc0 - [unknown source
> file]
> impalad ! tcmalloc::ThreadCache::FetchFromCentralCache + 0x62 - [unknown
> source file]
> impalad ! operator new + 0x297 - [unknown source file] <--- Creating
> new 'OutboundTransferTask' object.
> impalad ! kudu::rpc::Connection::QueueResponseForCall + 0x76 -
> connection.cc:432
> impalad ! kudu::rpc::InboundCall::Respond + 0xfa - inbound_call.cc:133
> impalad ! kudu::rpc::InboundCall::RespondSuccess + 0x43 - inbound_call.cc:77
> impalad ! kudu::rpc::RpcContext::RespondSuccess + 0x1f7 - rpc_context.cc:66
> ...
> {code}
> Even creating and deleting the 'RpcContext' takes a lot of time:
> {code:java}
> impalad ! tcmalloc::CentralFreeList::ReleaseToSpans - [unknown source file]
> impalad ! tcmalloc::CentralFreeList::ReleaseListToSpans + 0x1a - [unknown
> source file]
> impalad ! tcmalloc::CentralFreeList::InsertRange + 0x3b - [unknown source
> file]
> impalad ! tcmalloc::ThreadCache::ReleaseToCentralCache + 0x103 - [unknown
> source file]
> impalad ! tcmalloc::ThreadCache::Scavenge + 0x3e - [unknown source file]
> impalad ! operator delete + 0x329 - [unknown source file]
> impalad ! impala::TransmitDataResponsePb::~TransmitDataResponsePb + 0x16 -
> impala_internal_service.pb.cc:1221
> impalad ! impala::TransmitDataResponsePb::~TransmitDataResponsePb + 0x8 -
> impala_internal_service.pb.cc:1222
> impalad ! kudu::DefaultDeleter<google::protobuf::Message>::operator() + 0x5 -
> gscoped_ptr.h:145
> impalad ! ~gscoped_ptr_impl + 0x9 - gscoped_ptr.h:228
> impalad ! ~gscoped_ptr - gscoped_ptr.h:318
> impalad ! kudu::rpc::RpcContext::~RpcContext + 0x1e - rpc_context.cc:53
> <-----
> impalad ! kudu::rpc::RpcContext::RespondSuccess + 0x1ff - rpc_context.cc:67
> {code}
> The above show that creating these small objects under moderately heavy load
> results in heavy contention in the kernel. We will benefit a lot if we create
> a fast path for 'RespondSuccess'.
> My suggestion is to create all these small objects at once along with the
> 'InboundCall' object while it is being created, in a 'RespondSuccess'
> structure, and just use that structure when we want to send 'success' back to
> the sender. This would already contain the 'OutboundTransferTask', a 'Slice'
> with 'success', etc. We would expect that most RPCs respond with 'success' a
> majority of the time.
> How this would benefit us is that we don't go back and forth every time to
> allocate and deallocate memory for these small objects, instead we do it all
> at once while creating the 'InboundCall' object.
> I just wanted to start a discussion about this, so even if what I suggested
> seems a little off, hopefully we can move forward with this on some level.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
