imay opened a new issue #1258: routine load submit task stall URL: https://github.com/apache/incubator-doris/issues/1258 **Describe the bug** When there are too many concurrent routine load task, all `submit_task` will wait for a lock when the thread pool is full. following is the stack This is which hold the `_lock`, and waiting to enter the thread pool ``` #0 0x00007f5686e39965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000026cf8ec in __gthread_cond_wait (__mutex=<optimized out>, __cond=__cond@entry=0x56b43c0) at /var/local/gcc/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x56b43c0, __lock=...) at ../../../.././libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000140f2df in blocking_put (val=..., this=0x56b4388) at /var/local/incubator-doris/be/src/util/blocking_queue.hpp:109 #4 offer (func=..., this=0x56b4388) at /var/local/incubator-doris/be/src/util/thread_pool.hpp:72 #5 doris::RoutineLoadTaskExecutor::submit_task (this=0x56b4380, task=...) at /var/local/incubator-doris/be/src/runtime/routine_load/routine_load_task_executor.cpp:96 #6 0x0000000000ec8dd7 in doris::BackendService::submit_routine_load_task (this=0x5793ce0, t_status=..., tasks=...) at /var/local/incubator-doris/be/src/service/backend_service.cpp:236 #7 0x0000000000fa0372 in doris::BackendServiceProcessor::process_submit_routine_load_task (this=0x5790d80, seqid=1, iprot=<optimized out>, oprot=0x6ee4380, callContext=<optimized out>) at /var/local/incubator-doris/gensrc/build/gen_cpp/BackendService.cpp:6307 #8 0x0000000000f7b324 in doris::BackendServiceProcessor::dispatchCall (this=0x5790d80, iprot=0x6d6e580, oprot=0x6ee4380, fname=..., seqid=1, callContext=0x6d63b80) at /var/local/incubator-doris/gensrc/build/gen_cpp/BackendService.cpp:5200 #9 0x0000000000fa0edc in apache::thrift::TDispatchProcessor::process (this=0x5790d80, in=..., out=..., connectionContext=0x6d63b80) at /var/local/thirdparty/installed/include/thrift/TDispatchProcessor.h:121 #10 0x0000000001fca6c1 in apache::thrift::server::TConnectedClient::run (this=0x6f34d20) at src/thrift/server/TConnectedClient.cpp:62 #11 0x0000000001fc5209 in apache::thrift::concurrency::PthreadThread::threadMain (arg=0x579ad00) at src/thrift/concurrency/PosixThreadFactory.cpp:206 #12 0x00007f5686e35dd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f5687147ead in clone () from /lib64/libc.so.6 ``` This is others request which is waiting for the `_lock` ``` #0 0x00007f5686e3c4ed in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007f5686e37dcb in _L_lock_883 () from /lib64/libpthread.so.0 #2 0x00007f5686e37c98 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00000000021674f2 in pthread_mutex_lock_impl (mutex=0x56b4668) at /var/local/incubator-doris/thirdparty/src/brpc-0.9.0/src/bthread/mutex.cpp:545 #4 pthread_mutex_lock (__mutex=0x56b4668) at /var/local/incubator-doris/thirdparty/src/brpc-0.9.0/src/bthread/mutex.cpp:803 #5 0x000000000140c9d9 in __gthread_mutex_lock (__mutex=0x56b4668) at /usr/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:748 #6 lock (this=0x56b4668) at /usr/include/c++/7.3.0/bits/std_mutex.h:103 #7 lock (this=0x7f55ad85f2d0) at /usr/include/c++/7.3.0/bits/std_mutex.h:267 #8 unique_lock (__m=..., this=0x7f55ad85f2d0) at /usr/include/c++/7.3.0/bits/std_mutex.h:197 #9 doris::RoutineLoadTaskExecutor::submit_task (this=0x56b4380, task=...) at /var/local/incubator-doris/be/src/runtime/routine_load/routine_load_task_executor.cpp:37 #10 0x0000000000ec8dd7 in doris::BackendService::submit_routine_load_task (this=0x5793ce0, t_status=..., tasks=...) at /var/local/incubator-doris/be/src/service/backend_service.cpp:236 #11 0x0000000000fa0372 in doris::BackendServiceProcessor::process_submit_routine_load_task (this=0x5790d80, seqid=1, iprot=<optimized out>, oprot=0x12527800, callContext=<optimized out>) at /var/local/incubator-doris/gensrc/build/gen_cpp/BackendService.cpp:6307 #12 0x0000000000f7b324 in doris::BackendServiceProcessor::dispatchCall (this=0x5790d80, iprot=0x6ee43c0, oprot=0x12527800, fname=..., seqid=1, callContext=0x12b7e1c0) at /var/local/incubator-doris/gensrc/build/gen_cpp/BackendService.cpp:5200 #13 0x0000000000fa0edc in apache::thrift::TDispatchProcessor::process (this=0x5790d80, in=..., out=..., connectionContext=0x12b7e1c0) at /var/local/thirdparty/installed/include/thrift/TDispatchProcessor.h:121 #14 0x0000000001fca6c1 in apache::thrift::server::TConnectedClient::run (this=0x11f35dc0) at src/thrift/server/TConnectedClient.cpp:62 #15 0x0000000001fc5209 in apache::thrift::concurrency::PthreadThread::threadMain (arg=0x12b8e000) at src/thrift/concurrency/PosixThreadFactory.cpp:206 #16 0x00007f5686e35dd5 in start_thread () from /lib64/libpthread.so.0 #17 0x00007f5687147ead in clone () from /lib64/libc.so.6 ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
