Hi all, Link for fig_1 and fig_2: https://drive.google.com/drive/folders/1w77EZTQp-vUcoYfIIaIBN3mZL4HS32M8?usp=sharing
Thank you, Regards *Sumudu Madushanka* On Fri, 10 Jun 2022 at 09:01, Quanlong Huang <huangquanl...@gmail.com> wrote: > Hi Sumudu, > > Thanks for sharing your findings! Unfortunately, the attachments (fig_1 and > fig_2) are missing. They were probably ignored by the mailing list. Could > you upload them to Google Drive and share the links? > > BTW, is it the same query you shared at this thread? > https://lists.apache.org/thread/dstobv61xjhm18vddv264s66sx2oh22w > > I think we need some time to dig into the details. Not sure if the issue > still occurs in 4.1.0. > > Thanks, > Quanlong > > On Tue, Jun 7, 2022 at 11:42 PM Sumudu Madushanka < > ngsmadushanka...@cse.mrt.ac.lk> wrote: > > > Hi all, > > > > We are using kudu 1.15.0, and impala 3.4.0-RELEASE > > > > In our impala select query, we are scanning *10 kudu partitions/tablets* > > per impala daemon host in *6 node cluster* with *16 cores* per machine ( > > *c5.4xlarge*). > > > > The kudu threads (around 9 threads) are working in parallel to do the > > scanning as in given in fig_2. But, in Impala *only one reactor thread > > (rpc reactor-227 thread)* is working and it’s at* 99.99%* which creates a > > bottleneck(fig_1). There are other reactor threads at idle. Please note > > that the query results are not cached to disk. > > > > Here are our questions. > > > > 1. Why only one rpc reactor thread is working and creating a bottleneck > > for multi-tablets scan query in an impala daemon host and why is it > always *rpc > > reactor-227*? (We also suspect that this could be the rpc reactor thread > > that runs inside kudu client inside the impala daemon) > > > > 2. How can we solve this issue for other rpc reactor threads to work as > > well? > > > > This is creating a huge bottleneck in the query scanning. Could there be > > other reasons why this behavior occurs? > > > > Appreciate your help! > > > > gstack 22788 > > Thread 1 (process 22788): > > #0 0x00007f15a5ebeaeb in recv () from /usr/lib64/libpthread.so.0 > > #1 0x00007f15a42763b4 in kudu::Socket::Recv (this=0xd5588f0, > > buf=0x17324004 "\f\b\325\300\b\020", amt=1185345, nread=0x7f14c81d9940) > at > > /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/util/net/socket.cc:517 > > #2 0x00007f15a419172e in kudu::rpc::InboundTransfer::ReceiveBuffer > > (this=0xd507840, socket=...) at > > /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/rpc/transfer.cc:144 > > #3 0x00007f15a4182fd0 in ReadHandler (revents=<optimized out>, > > watcher=..., this=0xdb56000) at > > /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/rpc/connection.cc:653 > > #4 ev::base<ev_io, ev::io>::method_thunk<kudu::rpc::Connection, > > &kudu::rpc::Connection::ReadHandler> (loop=<optimized out>, w=<optimized > > out>, revents=<optimized out>) at > > > /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/installed/uninstrumented/include/ev++.h:479 > > #5 0x00007f15a43c1f1b in ev_invoke_pending (loop=0xd552480) at > > /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/src/libev-4.20/ev.c:3155 > > #6 0x00007f15a41611dd in kudu::rpc::ReactorThread::InvokePendingCb > > (loop=0xd552480) at > > /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/rpc/reactor.cc:196 > > #7 0x00007f15a43c55f4 in ev_run (loop=0xd552480, flags=<optimized out>) > at > > /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/src/libev-4.20/ev.c:3555 > > #8 0x00007f15a41617ab in run (flags=0, this=0x9c03088) at > > > /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/installed/uninstrumented/include/ev++.h:211 > > #9 kudu::rpc::ReactorThread::RunThread (this=0x9c03080) at > > /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/rpc/reactor.cc:497 > > #10 0x00007f15a429df06 in operator() (this=0x9ffdb28) at > > > /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/installed/uninstrumented/include/boost/function/function_template.hpp:771 > > #11 kudu::Thread::SuperviseThread (arg=0x9ffdb00) at > > /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/util/thread.cc:675 > > #12 0x00007f15a5eb7ea5 in start_thread () from /usr/lib64/libpthread.so.0 > > #13 0x00007f15a252b9fd in clone () from /usr/lib64/libc.so.6 > > > > > > > > Thank you > > > > > > > > Best Regards, > > *Sumudu Madushanka* > > >
