Hi all, We are using kudu 1.15.0, and impala 3.4.0-RELEASE
In our impala select query, we are scanning *10 kudu partitions/tablets* per impala daemon host in *6 node cluster* with *16 cores* per machine ( *c5.4xlarge*). The kudu threads (around 9 threads) are working in parallel to do the scanning as in given in fig_2. But, in Impala *only one reactor thread (rpc reactor-227 thread)* is working and it’s at* 99.99%* which creates a bottleneck(fig_1). There are other reactor threads at idle. Please note that the query results are not cached to disk. Here are our questions. 1. Why only one rpc reactor thread is working and creating a bottleneck for multi-tablets scan query in an impala daemon host and why is it always *rpc reactor-227*? (We also suspect that this could be the rpc reactor thread that runs inside kudu client inside the impala daemon) 2. How can we solve this issue for other rpc reactor threads to work as well? This is creating a huge bottleneck in the query scanning. Could there be other reasons why this behavior occurs? Appreciate your help! gstack 22788 Thread 1 (process 22788): #0 0x00007f15a5ebeaeb in recv () from /usr/lib64/libpthread.so.0 #1 0x00007f15a42763b4 in kudu::Socket::Recv (this=0xd5588f0, buf=0x17324004 "\f\b\325\300\b\020", amt=1185345, nread=0x7f14c81d9940) at /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/util/net/socket.cc:517 #2 0x00007f15a419172e in kudu::rpc::InboundTransfer::ReceiveBuffer (this=0xd507840, socket=...) at /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/rpc/transfer.cc:144 #3 0x00007f15a4182fd0 in ReadHandler (revents=<optimized out>, watcher=..., this=0xdb56000) at /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/rpc/connection.cc:653 #4 ev::base<ev_io, ev::io>::method_thunk<kudu::rpc::Connection, &kudu::rpc::Connection::ReadHandler> (loop=<optimized out>, w=<optimized out>, revents=<optimized out>) at /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/installed/uninstrumented/include/ev++.h:479 #5 0x00007f15a43c1f1b in ev_invoke_pending (loop=0xd552480) at /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/src/libev-4.20/ev.c:3155 #6 0x00007f15a41611dd in kudu::rpc::ReactorThread::InvokePendingCb (loop=0xd552480) at /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/rpc/reactor.cc:196 #7 0x00007f15a43c55f4 in ev_run (loop=0xd552480, flags=<optimized out>) at /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/src/libev-4.20/ev.c:3555 #8 0x00007f15a41617ab in run (flags=0, this=0x9c03088) at /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/installed/uninstrumented/include/ev++.h:211 #9 kudu::rpc::ReactorThread::RunThread (this=0x9c03080) at /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/rpc/reactor.cc:497 #10 0x00007f15a429df06 in operator() (this=0x9ffdb28) at /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/installed/uninstrumented/include/boost/function/function_template.hpp:771 #11 kudu::Thread::SuperviseThread (arg=0x9ffdb00) at /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/util/thread.cc:675 #12 0x00007f15a5eb7ea5 in start_thread () from /usr/lib64/libpthread.so.0 #13 0x00007f15a252b9fd in clone () from /usr/lib64/libc.so.6 Thank you Best Regards, *Sumudu Madushanka*
