[ https://issues.apache.org/jira/browse/IMPALA-12757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17811419#comment-17811419 ]
ASF subversion and git services commented on IMPALA-12757: ---------------------------------------------------------- Commit f3ac2ddbfef0d7cd359b7c9ae47d424791327c6d in impala's branch refs/heads/master from Michael Smith [ https://gitbox.apache.org/repos/asf?p=impala.git;h=f3ac2ddbf ] IMPALA-12747: Atomic update of execution state QueryDriver owns instances of ClientRequestState and TExecRequest. The ClientRequestState is used to track execution state of the client-facing side of a query. TExecRequest encapsulates context about the query produced by the planner. When a QueryDriver is created, it creates an instance of ClientRequestState, but has not yet executed planning. It would create an empty TExecRequest and pass a pointer to it to ClientRequestState, then update the content of TExecRequest when RunFrontendPlanner is called from ImpalaServer::ExecuteInternal. Updating TExecRequest was not atomic, so it was possible other operations - like producing a QueryStateRecord for /queries in the web UI - would try to read the content of TExecRequest while updating. This caused TSAN errors and occasional crashes in internal-server-test, which runs concurrent requests and examines them through calls to /queries. Changes ClientRequestState to - Provide a static placeholder for TExecRequest during creation that represents an empty context for an UNKNOWN statement type (default initialized in Thrift). - Make all references to TExecRequest const so its content cannot be updated in a non-thread-safe manner. - ClientRequestState uses an AtomicPtr which is updated atomically when the filled TExecRequest is available. QueryDriver does not publicly expose access to TExecRequest, so we can ensure its use is thread-safe without atomics. ClientRequestState::exec_request() will return either a reference to the static placeholder or the value provided after - which is never changed - so this reference will always be valid for the lifetime of the ClientRequestState. Updates user_has_profile_access to be AtomicBool for the same reason. Reverts tsan-suppressions for IMPALA-12660 so we get TSAN coverage. Adds suppression for a lock-order-inversion bug (IMPALA-12757) that was uncovered after fixing this data race. Testing: - InternalServerTest.SimultaneousMultipleQueriesOneSession would fail after ~10 test runs. Ran 90 times without failure. - Passed TSAN run of backend tests. Change-Id: I9a967c5c84b6a401f8f5764373f6cd7ee807545f Reviewed-on: http://gerrit.cloudera.org:8080/20956 Reviewed-by: Jason Fehr <jf...@cloudera.com> Reviewed-by: Riza Suminto <riza.sumi...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> > TSAN flags lock-order-inversion during internal-server-test > ----------------------------------------------------------- > > Key: IMPALA-12757 > URL: https://issues.apache.org/jira/browse/IMPALA-12757 > Project: IMPALA > Issue Type: Bug > Affects Versions: Impala 4.4.0 > Reporter: Michael Smith > Priority: Major > > internal-server-test has a tight loop starting queries and fetching /queries. > That's led to identifying several latent threading issues. Once IMPALA-12747 > is addressed, TSAN identifies a new error: > {code} > $ run-jvm-binary.sh ./be/build/debug/service/internal-server-test > I20240125 10:44:44.971467 152733 openssl_util.cc:110] FIPS mode is disabled. > Picked up JAVA_TOOL_OPTIONS: -Dsun.java.command=internal-server-test > 24/01/25 10:44:45 WARN fs.FileSystem: Cannot load filesystem: > java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: > Provider org.apache.hadoop.hive.ql.io.NullScanFileSystem not found > 24/01/25 10:44:45 WARN fs.FileSystem: Cannot load filesystem: > java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: > Provider org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem not found > 24/01/25 10:44:45 INFO util.JvmPauseMonitor: Starting JVM pause monitor > [==========] Running 10 tests from 1 test case. > [----------] Global test environment set-up. > [----------] 10 tests from InternalServerTest > [ RUN ] InternalServerTest.QueryTimeout > [ OK ] InternalServerTest.QueryTimeout (10236 ms) > [ RUN ] InternalServerTest.InvalidQueryOption > [ OK ] InternalServerTest.InvalidQueryOption (76 ms) > [ RUN ] InternalServerTest.MultipleQueriesMultipleSessions > /home/michael/Impala/be/src/service/internal-server-test.cc:289: Failure > Value of: status_.ok() > Actual: false > Expected: true > Error: Failed due to unreachable impalad(s): michaelsmith-22742:27000 > [ FAILED ] InternalServerTest.MultipleQueriesMultipleSessions (17225 ms) > [ RUN ] InternalServerTest.RetryFailedQuery > [ OK ] InternalServerTest.RetryFailedQuery (1206 ms) > [ RUN ] InternalServerTest.MultipleQueriesOneSession > ^C================== > WARNING: ThreadSanitizer: thread leak (pid=152733) > Thread T1086 (tid=154103, finished) created by main thread at: > #0 pthread_create <null> (internal-server-test+0x203c383) > #1 boost::thread::start_thread_noexcept() <null> > (internal-server-test+0x3bcd3fd) > #2 boost::thread::thread<void (*)(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> >, boost::function<void ()>, impala::ThreadDebugInfo*, > impala::Promise<long, (impala::PromiseMode)0>*>(void > (*)(std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&, std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, boost::function<void > ()>, impala::ThreadDebugInfo const*, impala::Promise<long, > (impala::PromiseMode)0>*), std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> >, > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::function<void ()>, impala::ThreadDebugInfo*, impala::Promise<long, > (impala::PromiseMode)0>*) > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/thread/detail/thread.hpp:424:13 > (internal-server-test+0x2d84514) > #3 impala::Thread::StartThread(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, boost::function<void ()> const&, std::unique_ptr<impala::Thread, > std::default_delete<impala::Thread> >*, bool) > /home/michael/Impala/be/src/util/thread.cc:317:13 > (internal-server-test+0x2d8091c) > #4 impala::Status impala::Thread::Create<void > (impala::ClientRequestState::*)(), > impala::ClientRequestState*>(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, void (impala::ClientRequestState::* const&)(), > impala::ClientRequestState* const&, std::unique_ptr<impala::Thread, > std::default_delete<impala::Thread> >*, bool) > /home/michael/Impala/be/src/util/thread.h:81:12 > (internal-server-test+0x2b382f7) > #5 impala::ClientRequestState::WaitAsync() > /home/michael/Impala/be/src/service/client-request-state.cc:1126:10 > (internal-server-test+0x2b2f9f4) > #6 impala::ImpalaServer::WaitForResults(impala::TUniqueId&) > /home/michael/Impala/be/src/service/internal-server.cc:156:3 > (internal-server-test+0x2ad148e) > #7 non-virtual thunk to > impala::ImpalaServer::WaitForResults(impala::TUniqueId&) > /home/michael/Impala/be/src/service/internal-server.cc > (internal-server-test+0x2ad15fd) > #8 > impala::internalservertest::InternalServerTest_MultipleQueriesOneSession_Test::TestBody() > /home/michael/Impala/be/src/service/internal-server-test.cc:380:3 > (internal-server-test+0x209fb9c) > #9 void > testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, > void>(testing::Test*, void (testing::Test::*)(), char const*) <null> > (internal-server-test+0x5212c1c) > #10 <null> <null> (libc.so.6+0x29d8f) > SUMMARY: ThreadSanitizer: thread leak > (/home/michael/Impala/be/build/debug/service/internal-server-test+0x203c383) > in __interceptor_pthread_create > ================== > michael@michaelsmith-22742 ~/Impala ❯❯❯ TSAN_OPTIONS=second_deadlock_stack=1 > run-jvm-binary.sh ./be/build/debug/service/internal-server-test > michael@michaelsmith-22742 ~/Impala ❯❯❯ export > TSAN_OPTIONS=second_deadlock_stack=1 ✘ 130 > michael@michaelsmith-22742 ~/Impala ❯❯❯ run-jvm-binary.sh > ./be/build/debug/service/internal-server-test > I20240125 10:46:49.913501 161734 openssl_util.cc:110] FIPS mode is disabled. > Picked up JAVA_TOOL_OPTIONS: -Dsun.java.command=internal-server-test > 24/01/25 10:46:50 WARN fs.FileSystem: Cannot load filesystem: > java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: > Provider org.apache.hadoop.hive.ql.io.NullScanFileSystem not found > 24/01/25 10:46:50 WARN fs.FileSystem: Cannot load filesystem: > java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: > Provider org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem not found > 24/01/25 10:46:50 INFO util.JvmPauseMonitor: Starting JVM pause monitor > [==========] Running 10 tests from 1 test case. > [----------] Global test environment set-up. > [----------] 10 tests from InternalServerTest > [ RUN ] InternalServerTest.QueryTimeout > [ OK ] InternalServerTest.QueryTimeout (10149 ms) > [ RUN ] InternalServerTest.InvalidQueryOption > [ OK ] InternalServerTest.InvalidQueryOption (75 ms) > [ RUN ] InternalServerTest.MultipleQueriesMultipleSessions > /home/michael/Impala/be/src/service/internal-server-test.cc:289: Failure > Value of: status_.ok() > Actual: false > Expected: true > Error: Failed due to unreachable impalad(s): michaelsmith-22742:27000 > [ FAILED ] InternalServerTest.MultipleQueriesMultipleSessions (17058 ms) > [ RUN ] InternalServerTest.RetryFailedQuery > [ OK ] InternalServerTest.RetryFailedQuery (1205 ms) > [ RUN ] InternalServerTest.MultipleQueriesOneSession > /home/michael/Impala/be/src/service/internal-server-test.cc:381: Failure > Value of: status_.ok() > Actual: false > Expected: true > Error: Failed due to unreachable impalad(s): michaelsmith-22742:27001 > [ FAILED ] InternalServerTest.MultipleQueriesOneSession (15934 ms) > [ RUN ] InternalServerTest.MissingClosingQuote > [ OK ] InternalServerTest.MissingClosingQuote (8436 ms) > [ RUN ] InternalServerTest.SyntaxError > [ OK ] InternalServerTest.SyntaxError (7929 ms) > [ RUN ] InternalServerTest.UnclosedComment > [ OK ] InternalServerTest.UnclosedComment (96 ms) > [ RUN ] InternalServerTest.TableNotExist > [ OK ] InternalServerTest.TableNotExist (7994 ms) > [ RUN ] InternalServerTest.SimultaneousMultipleQueriesOneSession > ================== > WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) > (pid=161734) > Cycle in lock order graph: M49465 (0x7b50005302b4) => M49694 > (0x7b100017f5d0) => M49465 > Mutex M49694 acquired here while holding mutex M49465 in thread T1310: > #0 pthread_mutex_lock <null> (internal-server-test+0x202dbb0) > #1 __gthread_mutex_lock(pthread_mutex_t*) > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/x86_64-pc-linux-gnu/bits/gthr-default.h:749:12 > (internal-server-test+0x20a41b6) > #2 std::mutex::lock() > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/std_mutex.h:100:17 > (internal-server-test+0x20ac089) > #3 std::unique_lock<std::mutex>::lock() > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/unique_lock.h:138:17 > (internal-server-test+0x25e1c9f) > #4 std::unique_lock<std::mutex>::unique_lock(std::mutex&) > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/unique_lock.h:68:2 > (internal-server-test+0x25e1013) > #5 std::cv_status > std::_V2::condition_variable_any::wait_until<std::unique_lock<impala::SpinLock>, > std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, > 1000000000l> > >(std::unique_lock<impala::SpinLock>&, > std::chrono::time_point<std::chrono::_V2::steady_clock, > std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/condition_variable:342:21 > (internal-server-test+0x2731a45) > #6 std::cv_status > std::_V2::condition_variable_any::wait_for<std::unique_lock<impala::SpinLock>, > long, std::ratio<1l, 1000l> >(std::unique_lock<impala::SpinLock>&, > std::chrono::duration<long, std::ratio<1l, 1000l> > const&) > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/condition_variable:366:16 > (internal-server-test+0x272b816) > #7 > impala::KrpcDataStreamSender::Channel::WaitForRpcLocked(std::unique_lock<impala::SpinLock>*) > /home/michael/Impala/be/src/runtime/krpc-data-stream-sender.cc:437:18 > (internal-server-test+0x2721637) > #8 impala::KrpcDataStreamSender::Channel::WaitForRpc() > /home/michael/Impala/be/src/runtime/krpc-data-stream-sender.cc:425:10 > (internal-server-test+0x27214e8) > #9 impala::KrpcDataStreamSender::FlushFinal(impala::RuntimeState*) > /home/michael/Impala/be/src/runtime/krpc-data-stream-sender.cc:1188:5 > (internal-server-test+0x272a1ca) > #10 impala::FragmentInstanceState::ExecInternal() > /home/michael/Impala/be/src/runtime/fragment-instance-state.cc:466:3 > (internal-server-test+0x2711440) > #11 impala::FragmentInstanceState::Exec() > /home/michael/Impala/be/src/runtime/fragment-instance-state.cc:104:14 > (internal-server-test+0x270f29c) > #12 impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) > /home/michael/Impala/be/src/runtime/query-state.cc:991:24 > (internal-server-test+0x2646f48) > #13 impala::QueryState::StartFInstances()::$_15::operator()() const > /home/michael/Impala/be/src/runtime/query-state.cc:896:37 > (internal-server-test+0x264bae2) > #14 > boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::$_15, > void>::invoke(boost::detail::function::function_buffer&) > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:158:11 > (internal-server-test+0x264b899) > #15 boost::function0<void>::operator()() const > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:763:14 > (internal-server-test+0x2592ff1) > #16 impala::Thread::SuperviseThread(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*) > /home/michael/Impala/be/src/util/thread.cc:360:3 > (internal-server-test+0x2d80f98) > #17 void > boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > >, > boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()<void (*)(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list0>(boost::_bi::type<void>, void > (*&)(std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&, std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, boost::function<void > ()>, impala::ThreadDebugInfo const*, impala::Promise<long, > (impala::PromiseMode)0>*), boost::_bi::list0&, int) > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:531:9 > (internal-server-test+0x2d89d1c) > #18 boost::_bi::bind_t<void, void (*)(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > >, > boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::operator()() > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:1294:16 > (internal-server-test+0x2d89c33) > #19 boost::detail::thread_data<boost::_bi::bind_t<void, void > (*)(std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&, std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, boost::function<void > ()>, impala::ThreadDebugInfo const*, impala::Promise<long, > (impala::PromiseMode)0>*), > boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > >, > boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > > >::run() > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/thread/detail/thread.hpp:120:17 > (internal-server-test+0x2d89920) > #20 thread_proxy <null> (internal-server-test+0x3bce0f6) > Mutex M49465 previously acquired by the same thread here: > #0 AnnotateRWLockAcquired <null> (internal-server-test+0x205502c) > #1 base::SpinLock::Lock() > /home/michael/Impala/be/src/gutil/spinlock.h:77:5 > (internal-server-test+0x215d515) > #2 impala::SpinLock::lock() > /home/michael/Impala/be/src/util/spinlock.h:34:8 > (internal-server-test+0x215d4a9) > #3 std::unique_lock<impala::SpinLock>::lock() > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/unique_lock.h:138:17 > (internal-server-test+0x2610cff) > #4 std::unique_lock<impala::SpinLock>::unique_lock(impala::SpinLock&) > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/unique_lock.h:68:2 > (internal-server-test+0x2610663) > #5 impala::KrpcDataStreamSender::Channel::WaitForRpc() > /home/michael/Impala/be/src/runtime/krpc-data-stream-sender.cc:424:30 > (internal-server-test+0x27214d9) > #6 impala::KrpcDataStreamSender::FlushFinal(impala::RuntimeState*) > /home/michael/Impala/be/src/runtime/krpc-data-stream-sender.cc:1188:5 > (internal-server-test+0x272a1ca) > #7 impala::FragmentInstanceState::ExecInternal() > /home/michael/Impala/be/src/runtime/fragment-instance-state.cc:466:3 > (internal-server-test+0x2711440) > #8 impala::FragmentInstanceState::Exec() > /home/michael/Impala/be/src/runtime/fragment-instance-state.cc:104:14 > (internal-server-test+0x270f29c) > #9 impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) > /home/michael/Impala/be/src/runtime/query-state.cc:991:24 > (internal-server-test+0x2646f48) > #10 impala::QueryState::StartFInstances()::$_15::operator()() const > /home/michael/Impala/be/src/runtime/query-state.cc:896:37 > (internal-server-test+0x264bae2) > #11 > boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::$_15, > void>::invoke(boost::detail::function::function_buffer&) > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:158:11 > (internal-server-test+0x264b899) > #12 boost::function0<void>::operator()() const > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:763:14 > (internal-server-test+0x2592ff1) > #13 impala::Thread::SuperviseThread(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*) > /home/michael/Impala/be/src/util/thread.cc:360:3 > (internal-server-test+0x2d80f98) > #14 void > boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > >, > boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()<void (*)(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list0>(boost::_bi::type<void>, void > (*&)(std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&, std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, boost::function<void > ()>, impala::ThreadDebugInfo const*, impala::Promise<long, > (impala::PromiseMode)0>*), boost::_bi::list0&, int) > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:531:9 > (internal-server-test+0x2d89d1c) > #15 boost::_bi::bind_t<void, void (*)(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > >, > boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::operator()() > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:1294:16 > (internal-server-test+0x2d89c33) > #16 boost::detail::thread_data<boost::_bi::bind_t<void, void > (*)(std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&, std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, boost::function<void > ()>, impala::ThreadDebugInfo const*, impala::Promise<long, > (impala::PromiseMode)0>*), > boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > >, > boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > > >::run() > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/thread/detail/thread.hpp:120:17 > (internal-server-test+0x2d89920) > #17 thread_proxy <null> (internal-server-test+0x3bce0f6) > Mutex M49465 acquired here while holding mutex M49694 in thread T1310: > #0 AnnotateRWLockAcquired <null> (internal-server-test+0x205502c) > #1 base::SpinLock::Lock() > /home/michael/Impala/be/src/gutil/spinlock.h:77:5 > (internal-server-test+0x215d515) > #2 impala::SpinLock::lock() > /home/michael/Impala/be/src/util/spinlock.h:34:8 > (internal-server-test+0x215d4a9) > #3 std::unique_lock<impala::SpinLock>::lock() > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/unique_lock.h:138:17 > (internal-server-test+0x2610cff) > #4 > std::_V2::condition_variable_any::_Unlock<std::unique_lock<impala::SpinLock> > >::~_Unlock() > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/condition_variable:283:14 > (internal-server-test+0x2613c16) > #5 std::cv_status > std::_V2::condition_variable_any::wait_until<std::unique_lock<impala::SpinLock>, > std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, > 1000000000l> > >(std::unique_lock<impala::SpinLock>&, > std::chrono::time_point<std::chrono::_V2::steady_clock, > std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/condition_variable:348:7 > (internal-server-test+0x2731a8b) > #6 std::cv_status > std::_V2::condition_variable_any::wait_for<std::unique_lock<impala::SpinLock>, > long, std::ratio<1l, 1000l> >(std::unique_lock<impala::SpinLock>&, > std::chrono::duration<long, std::ratio<1l, 1000l> > const&) > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/condition_variable:366:16 > (internal-server-test+0x272b816) > #7 > impala::KrpcDataStreamSender::Channel::WaitForRpcLocked(std::unique_lock<impala::SpinLock>*) > /home/michael/Impala/be/src/runtime/krpc-data-stream-sender.cc:437:18 > (internal-server-test+0x2721637) > #8 impala::KrpcDataStreamSender::Channel::WaitForRpc() > /home/michael/Impala/be/src/runtime/krpc-data-stream-sender.cc:425:10 > (internal-server-test+0x27214e8) > #9 impala::KrpcDataStreamSender::FlushFinal(impala::RuntimeState*) > /home/michael/Impala/be/src/runtime/krpc-data-stream-sender.cc:1188:5 > (internal-server-test+0x272a1ca) > #10 impala::FragmentInstanceState::ExecInternal() > /home/michael/Impala/be/src/runtime/fragment-instance-state.cc:466:3 > (internal-server-test+0x2711440) > #11 impala::FragmentInstanceState::Exec() > /home/michael/Impala/be/src/runtime/fragment-instance-state.cc:104:14 > (internal-server-test+0x270f29c) > #12 impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) > /home/michael/Impala/be/src/runtime/query-state.cc:991:24 > (internal-server-test+0x2646f48) > #13 impala::QueryState::StartFInstances()::$_15::operator()() const > /home/michael/Impala/be/src/runtime/query-state.cc:896:37 > (internal-server-test+0x264bae2) > #14 > boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::$_15, > void>::invoke(boost::detail::function::function_buffer&) > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:158:11 > (internal-server-test+0x264b899) > #15 boost::function0<void>::operator()() const > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:763:14 > (internal-server-test+0x2592ff1) > #16 impala::Thread::SuperviseThread(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*) > /home/michael/Impala/be/src/util/thread.cc:360:3 > (internal-server-test+0x2d80f98) > #17 void > boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > >, > boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()<void (*)(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list0>(boost::_bi::type<void>, void > (*&)(std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&, std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, boost::function<void > ()>, impala::ThreadDebugInfo const*, impala::Promise<long, > (impala::PromiseMode)0>*), boost::_bi::list0&, int) > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:531:9 > (internal-server-test+0x2d89d1c) > #18 boost::_bi::bind_t<void, void (*)(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > >, > boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::operator()() > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:1294:16 > (internal-server-test+0x2d89c33) > #19 boost::detail::thread_data<boost::_bi::bind_t<void, void > (*)(std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&, std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, boost::function<void > ()>, impala::ThreadDebugInfo const*, impala::Promise<long, > (impala::PromiseMode)0>*), > boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > >, > boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > > >::run() > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/thread/detail/thread.hpp:120:17 > (internal-server-test+0x2d89920) > #20 thread_proxy <null> (internal-server-test+0x3bce0f6) > Mutex M49694 previously acquired by the same thread here: > #0 pthread_mutex_lock <null> (internal-server-test+0x202dbb0) > #1 __gthread_mutex_lock(pthread_mutex_t*) > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/x86_64-pc-linux-gnu/bits/gthr-default.h:749:12 > (internal-server-test+0x20a41b6) > #2 std::mutex::lock() > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/std_mutex.h:100:17 > (internal-server-test+0x20ac089) > #3 std::unique_lock<std::mutex>::lock() > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/unique_lock.h:138:17 > (internal-server-test+0x25e1c9f) > #4 std::unique_lock<std::mutex>::unique_lock(std::mutex&) > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/unique_lock.h:68:2 > (internal-server-test+0x25e1013) > #5 std::cv_status > std::_V2::condition_variable_any::wait_until<std::unique_lock<impala::SpinLock>, > std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, > 1000000000l> > >(std::unique_lock<impala::SpinLock>&, > std::chrono::time_point<std::chrono::_V2::steady_clock, > std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/condition_variable:342:21 > (internal-server-test+0x2731a45) > #6 std::cv_status > std::_V2::condition_variable_any::wait_for<std::unique_lock<impala::SpinLock>, > long, std::ratio<1l, 1000l> >(std::unique_lock<impala::SpinLock>&, > std::chrono::duration<long, std::ratio<1l, 1000l> > const&) > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/condition_variable:366:16 > (internal-server-test+0x272b816) > #7 > impala::KrpcDataStreamSender::Channel::WaitForRpcLocked(std::unique_lock<impala::SpinLock>*) > /home/michael/Impala/be/src/runtime/krpc-data-stream-sender.cc:437:18 > (internal-server-test+0x2721637) > #8 impala::KrpcDataStreamSender::Channel::WaitForRpc() > /home/michael/Impala/be/src/runtime/krpc-data-stream-sender.cc:425:10 > (internal-server-test+0x27214e8) > #9 impala::KrpcDataStreamSender::FlushFinal(impala::RuntimeState*) > /home/michael/Impala/be/src/runtime/krpc-data-stream-sender.cc:1188:5 > (internal-server-test+0x272a1ca) > #10 impala::FragmentInstanceState::ExecInternal() > /home/michael/Impala/be/src/runtime/fragment-instance-state.cc:466:3 > (internal-server-test+0x2711440) > #11 impala::FragmentInstanceState::Exec() > /home/michael/Impala/be/src/runtime/fragment-instance-state.cc:104:14 > (internal-server-test+0x270f29c) > #12 impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) > /home/michael/Impala/be/src/runtime/query-state.cc:991:24 > (internal-server-test+0x2646f48) > #13 impala::QueryState::StartFInstances()::$_15::operator()() const > /home/michael/Impala/be/src/runtime/query-state.cc:896:37 > (internal-server-test+0x264bae2) > #14 > boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::$_15, > void>::invoke(boost::detail::function::function_buffer&) > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:158:11 > (internal-server-test+0x264b899) > #15 boost::function0<void>::operator()() const > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:763:14 > (internal-server-test+0x2592ff1) > #16 impala::Thread::SuperviseThread(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*) > /home/michael/Impala/be/src/util/thread.cc:360:3 > (internal-server-test+0x2d80f98) > #17 void > boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > >, > boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()<void (*)(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list0>(boost::_bi::type<void>, void > (*&)(std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&, std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, boost::function<void > ()>, impala::ThreadDebugInfo const*, impala::Promise<long, > (impala::PromiseMode)0>*), boost::_bi::list0&, int) > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:531:9 > (internal-server-test+0x2d89d1c) > #18 boost::_bi::bind_t<void, void (*)(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > >, > boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::operator()() > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:1294:16 > (internal-server-test+0x2d89c33) > #19 boost::detail::thread_data<boost::_bi::bind_t<void, void > (*)(std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&, std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, boost::function<void > ()>, impala::ThreadDebugInfo const*, impala::Promise<long, > (impala::PromiseMode)0>*), > boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > >, > boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > > >::run() > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/thread/detail/thread.hpp:120:17 > (internal-server-test+0x2d89920) > #20 thread_proxy <null> (internal-server-test+0x3bce0f6) > Thread T1137 (tid=163645, running) created by thread T1134 at: > #0 pthread_create <null> (internal-server-test+0x203c383) > #1 boost::thread::start_thread_noexcept() <null> > (internal-server-test+0x3bcd3fd) > #2 boost::thread::thread<void (*)(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> >, boost::function<void ()>, impala::ThreadDebugInfo*, > impala::Promise<long, (impala::PromiseMode)0>*>(void > (*)(std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&, std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, boost::function<void > ()>, impala::ThreadDebugInfo const*, impala::Promise<long, > (impala::PromiseMode)0>*), std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> >, > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::function<void ()>, impala::ThreadDebugInfo*, impala::Promise<long, > (impala::PromiseMode)0>*) > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/thread/detail/thread.hpp:424:13 > (internal-server-test+0x2d84514) > #3 impala::Thread::StartThread(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, boost::function<void ()> const&, std::unique_ptr<impala::Thread, > std::default_delete<impala::Thread> >*, bool) > /home/michael/Impala/be/src/util/thread.cc:317:13 > (internal-server-test+0x2d8091c) > #4 impala::Status > impala::Thread::Create<impala::QueryState::StartFInstances()::$_15>(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, impala::QueryState::StartFInstances()::$_15 const&, > std::unique_ptr<impala::Thread, std::default_delete<impala::Thread> >*, bool) > /home/michael/Impala/be/src/util/thread.h:74:12 > (internal-server-test+0x26468aa) > #5 impala::QueryState::StartFInstances() > /home/michael/Impala/be/src/runtime/query-state.cc:895:11 > (internal-server-test+0x264623e) > #6 impala::QueryExecMgr::ExecuteQueryHelper(impala::QueryState*) > /home/michael/Impala/be/src/runtime/query-exec-mgr.cc:162:7 > (internal-server-test+0x26347d9) > #7 boost::_mfi::mf1<void, impala::QueryExecMgr, > impala::QueryState*>::operator()(impala::QueryExecMgr*, impala::QueryState*) > const > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/mem_fn_template.hpp:165:29 > (internal-server-test+0x263c0bd) > #8 void boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, > boost::_bi::value<impala::QueryState*> >::operator()<boost::_mfi::mf1<void, > impala::QueryExecMgr, impala::QueryState*>, > boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void, > impala::QueryExecMgr, impala::QueryState*>&, boost::_bi::list0&, int) > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:319:9 > (internal-server-test+0x263c00a) > #9 boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::QueryExecMgr, > impala::QueryState*>, > boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, > boost::_bi::value<impala::QueryState*> > >::operator()() > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:1294:16 > (internal-server-test+0x263bf73) > #10 > boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, > boost::_mfi::mf1<void, impala::QueryExecMgr, impala::QueryState*>, > boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, > boost::_bi::value<impala::QueryState*> > >, > void>::invoke(boost::detail::function::function_buffer&) > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:158:11 > (internal-server-test+0x263bcb1) > #11 boost::function0<void>::operator()() const > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:763:14 > (internal-server-test+0x2592ff1) > #12 impala::Thread::SuperviseThread(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*) > /home/michael/Impala/be/src/util/thread.cc:360:3 > (internal-server-test+0x2d80f98) > #13 void > boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > >, > boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()<void (*)(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list0>(boost::_bi::type<void>, void > (*&)(std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&, std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, boost::function<void > ()>, impala::ThreadDebugInfo const*, impala::Promise<long, > (impala::PromiseMode)0>*), boost::_bi::list0&, int) > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:531:9 > (internal-server-test+0x2d89d1c) > #14 boost::_bi::bind_t<void, void (*)(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > >, > boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::operator()() > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:1294:16 > (internal-server-test+0x2d89c33) > #15 boost::detail::thread_data<boost::_bi::bind_t<void, void > (*)(std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&, std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, boost::function<void > ()>, impala::ThreadDebugInfo const*, impala::Promise<long, > (impala::PromiseMode)0>*), > boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > >, > boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > > >::run() > /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/thread/detail/thread.hpp:120:17 > (internal-server-test+0x2d89920) > #16 thread_proxy <null> (internal-server-test+0x3bce0f6) > SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) > (/home/michael/Impala/be/build/debug/service/internal-server-test+0x202dbb0) > in __interceptor_pthread_mutex_lock > ================== > {code} > That looks similar to > https://stackoverflow.com/questions/71188546/threadsanitizer-double-lock-of-a-mutex-with-stdjthread, > and I'm not sure I understand it yet. I want to address it as a separate > effort. > I encountered this on Ubuntu 22.04 (x86_64). Didn't see it reproduce on > CentOS 7.9. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org