Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24176 )
Change subject: WIP [rpc] in-bulk memory recycling for Connection::ProcessOutboundTransfers() ...................................................................... Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/24176/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/24176/2//COMMIT_MSG@45 PS2, Line 45: #18 0x0000000002077de9 in kudu::rpc::Connectio > Trying to understand this callstack - what bugs me is that the change is ar This patch addresses the issue. This call stack is captured before this change. The issue was that the InboundCall destructors were invoked upon completing transfer for the InboundCall's response. Do you want me to put the complete stack in the commit message? The complete unredacted stack is below: Thread 119 (Thread 0x7eebde8b7700 (LWP 69646)): #0 sys_futex (v2=0, u2=0x0, t=0x7eebde8b6520, v=<optimized out>, o=128, u=0x52257c8 <tcmalloc::Static::pageheap_lock_>) at ../gutil/linux_syscall_support.h:3401 #1 base::internal::SpinLockDelay (loop=<optimized out>, value=2, w=0x52257c8 <tcmalloc::Static::pageheap_lock_>) at ../gutil/spinlock_linux-inl.h:81 #2 base::internal::SpinLockDelay (w=0x52257c8 <tcmalloc::Static::pageheap_lock_>, value=2, loop=<optimized out>) at ../gutil/spinlock_linux-inl.h:69 #3 0x000000000377dffd in SpinLock::SlowLock() () #4 0x00000000038b71e8 in tcmalloc::CentralFreeList::ReleaseToSpans(void*) () #5 0x00000000038b7226 in tcmalloc::CentralFreeList::ReleaseListToSpans(void*) () #6 0x00000000038b7509 in tcmalloc::CentralFreeList::InsertRange(void*, void*, int) () #7 0x00000000038c7ee1 in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned int, int) () #8 0x00000000038c8505 in tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned int) () #9 0x00000000038d9103 in google::protobuf::internal::ArenaImpl::~ArenaImpl() () #10 0x0000000002043eec in google::protobuf::Arena::~Arena (this=0xf7ae19a08, __in_chrg=<optimized out>) at ../../../../toolchain/toolchain-packages-gcc10.4.0/protobuf-3.14.0/include/google/protobuf/arena.h:250 #11 kudu::rpc::InboundCall::~InboundCall (this=0xf7ae19860, __in_chrg=<optimized out>) at inbound_call.cc:76 #12 0x000000000207a1e0 in std::default_delete<kudu::rpc::InboundCall>::operator() (__ptr=0xf7ae19860, this=0x5715ef408) at /grid/0/jenkins/workspace/workspace/CDH-parallel-redhat8/SOURCES/impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/include/c++/10.4.0/bits/unique_ptr.h:79 #13 std::unique_ptr<kudu::rpc::InboundCall, std::default_delete<kudu::rpc::InboundCall> >::~unique_ptr (this=0x5715ef408, __in_chrg=<optimized out>) at /grid/0/jenkins/workspace/workspace/CDH-parallel-redhat8/SOURCES/impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/include/c++/10.4.0/bits/unique_ptr.h:361 #14 kudu::rpc::ResponseTransferCallbacks::~ResponseTransferCallbacks (this=0x5715ef400, __in_chrg=<optimized out>) at connection.cc:558 #15 kudu::rpc::ResponseTransferCallbacks::~ResponseTransferCallbacks (this=0x5715ef400, __in_chrg=<optimized out>) at connection.cc:563 #16 kudu::rpc::ResponseTransferCallbacks::NotifyTransferFinished (this=0x5715ef400) at connection.cc:566 #17 0x00000000020748be in kudu::rpc::OutboundTransfer::SendBuffer (this=this@entry=0xc5cff9a80, socket=0xf1fd880) at transfer.cc:263 #18 0x0000000002077de9 in kudu::rpc::Connection::ProcessOutboundTransfers (this=0x13dd0000) at /grid/0/jenkins/workspace/workspace/CDH-parallel-redhat8/SOURCES/impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/include/c++/10.4.0/bits/unique_ptr.h:421 #19 0x0000000002078440 in kudu::rpc::Connection::QueueOutbound (this=0x13dd0000, transfer=...) at connection.cc:353 #20 0x000000000207ade1 in kudu::rpc::QueueTransferTask::Run (this=0x58c2465d0, thr=<optimized out>) at /grid/0/jenkins/workspace/workspace/CDH-parallel-redhat8/SOURCES/impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/include/c++/10.4.0/bits/unique_ptr.h:172 #21 0x0000000002052b40 in kudu::rpc::ReactorThread::AsyncHandler (this=0x126ebc58) at reactor.cc:342 #22 0x000000000376e063 in ev_invoke_pending () #23 0x000000000204f5a1 in kudu::rpc::ReactorThread::InvokePendingCb (loop=0x10e48880) at reactor.cc:202 #24 0x00000000037713b5 in ev_run () #25 0x00000000020508eb in ev::loop_ref::run (flags=0, this=0x126ebc60) at ../../../../toolchain/toolchain-packages-gcc10.4.0/libev-4.20-p1/include/ev++.h:211 #26 kudu::rpc::ReactorThread::RunThread (this=0x126ebc58) at reactor.cc:503 #27 0x00000000013b7643 in std::function<void ()>::operator()() const (this=0x137989b8) at /grid/0/jenkins/workspace/workspace/CDH-parallel-redhat8/SOURCES/impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/include/c++/10.4.0/bits/std_function.h:617 #28 kudu::Thread::SuperviseThread (arg=0x13798960) at thread.cc:695 #29 0x00007eff960d51ca in start_thread () from /lib64/libpthread.so.0 #30 0x00007eff92ca7953 in clone () from /lib64/libc.so.6 http://gerrit.cloudera.org:8080/#/c/24176/1/src/kudu/rpc/connection.cc File src/kudu/rpc/connection.cc: http://gerrit.cloudera.org:8080/#/c/24176/1/src/kudu/rpc/connection.cc@668 PS1, Line 668: boost::intrusive::list<OutboundTransfer> recycled_transfers; // NOLINT(*) > Let's be cautious about adding memory traffic from list<> operations here i What's the essence behind the concern of list resizing? Is that about dynamic memory allocation? Intrusive containers don’t allocate memory dynamically. A call to push_back() doesn’t lead to a dynamic allocation with new. https://theboostcpplibraries.com/boost.intrusive -- To view, visit http://gerrit.cloudera.org:8080/24176 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idf7ab105a851ef4d583efc2d1b33d57607810df0 Gerrit-Change-Number: 24176 Gerrit-PatchSet: 1 Gerrit-Owner: Alexey Serbin <[email protected]> Gerrit-Reviewer: Alexey Serbin <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Kurt Deschler <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Comment-Date: Thu, 09 Apr 2026 17:43:14 +0000 Gerrit-HasComments: Yes
