[
https://issues.apache.org/jira/browse/IMPALA-12411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17764077#comment-17764077
]
ASF subversion and git services commented on IMPALA-12411:
----------------------------------------------------------
Commit 900f9f057c6814cda57f71049fc4de9348ff7163 in impala's branch
refs/heads/master from Michael Smith
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=900f9f057 ]
IMPALA-12411: Fix data race in expr-test teardown
TSAN ThreadSanitizer detected a data race during expr-test teardown when
a query finishes after JvmMetricCache's destructor is called. expr-test
can finish with a query still running because ImpaladQueryExecutor
closes the previous query when you start the next query and relies on
the destructor to close the last query.
expr-test creates ImpaladQueryExecutor as a global as part of setting up
an in-memory cluster (with Statestore and InProcessImpalaServer). So the
last query isn't guaranteed to be closed until global destruction, which
leads to a race with other global destruction (in this case with
JvmMetricCache while updating QueryState on query completion).
ImpaladQueryExecutor is only used in expr-test.
Change-Id: I8e289b292cb11154f1245e2b987cde0995b6243c
Reviewed-on: http://gerrit.cloudera.org:8080/20474
Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
> TSAN ThreadSanitizer: data race during expr-test teardown
> ---------------------------------------------------------
>
> Key: IMPALA-12411
> URL: https://issues.apache.org/jira/browse/IMPALA-12411
> Project: IMPALA
> Issue Type: Bug
> Components: be
> Affects Versions: Impala 4.3.0
> Reporter: Andrew Sherman
> Assignee: Michael Smith
> Priority: Critical
> Attachments: expr-test-tsan-failure.log
>
>
> The racing threads are
> {code:java}
> 20:14:05 Read of size 8 at 0x00000a8d3348 by main thread:
> 20:14:05 #0 std::vector<impala::TJvmMemoryPool,
> std::allocator<impala::TJvmMemoryPool> >::~vector()
> /data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/stl_vector.h:680:54
> (unifiedbetests+0x3fcd9b9)
> 20:14:05 #1
> impala::TGetJvmMemoryMetricsResponse::~TGetJvmMemoryMetricsResponse()
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/generated-sources/gen-cpp/Frontend_types.cpp:4158:1
> (unifiedbetests+0x3fc1397)
> 20:14:05 #2 impala::JvmMetricCache::~JvmMetricCache()
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/memory-metrics.h:170:7
> (unifiedbetests+0x4b2989d)
> 20:14:05 #3 at_exit_wrapper(void*)
> /mnt/source/llvm/llvm-5.0.1.src-p7/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:361:31
> (unifiedbetests+0x21b3554)
> {code}
> and
> {code:java}
> 20:14:05 Previous write of size 8 at 0x00000a8d3348 by thread T586:
> 20:14:05 #0 std::vector<impala::TJvmMemoryPool,
> std::allocator<impala::TJvmMemoryPool>
> >::_M_erase_at_end(impala::TJvmMemoryPool*)
> /data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/stl_vector.h:1798:30
> (unifiedbetests+0x4afabcc)
> 20:14:05 #1 std::vector<impala::TJvmMemoryPool,
> std::allocator<impala::TJvmMemoryPool> >::clear()
> /data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/stl_vector.h:1499:9
> (unifiedbetests+0x4afa4b4)
> 20:14:05 #2 unsigned int
> impala::TGetJvmMemoryMetricsResponse::read<apache::thrift::protocol::TProtocol>(apache::thrift::protocol::TProtocol*)
>
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/generated-sources/gen-cpp/Frontend_types.tcc:5673:32
> (unifiedbetests+0x4afa21d)
> 20:14:05 #3 impala::Status
> impala::DeserializeThriftMsg<impala::TGetJvmMemoryMetricsResponse>(unsigned
> char const*, unsigned int*, bool, impala::TGetJvmMemoryMetricsResponse*)
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/rpc/thrift-util.h:136:23
> (unifiedbetests+0x4af9da6)
> 20:14:05 #4 impala::Status
> impala::DeserializeThriftMsg<impala::TGetJvmMemoryMetricsResponse>(JNIEnv_*,
> _jbyteArray*, impala::TGetJvmMemoryMetricsResponse*)
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/rpc/jni-thrift-util.h:61:3
> (unifiedbetests+0x4af9c62)
> 20:14:05 #5 impala::Status
> impala::JniCall::ObjectToResult<impala::TGetJvmMemoryMetricsResponse>(_jobject*,
> impala::TGetJvmMemoryMetricsResponse*)
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/jni-util.h:493:3
> (unifiedbetests+0x4af9b24)
> 20:14:05 #6 impala::Status
> impala::JniCall::Call<impala::TGetJvmMemoryMetricsResponse>(impala::TGetJvmMemoryMetricsResponse*)
>
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/jni-util.h:486:3
> (unifiedbetests+0x4af92a6)
> 20:14:05 #7
> impala::JniUtil::GetJvmMemoryMetrics(impala::TGetJvmMemoryMetricsResponse*)
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/jni-util.cc:299:72
> (unifiedbetests+0x4af89a3)
> 20:14:05 #8 impala::JvmMetricCache::GrabMetricsIfNecessary()
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/memory-metrics.cc:294:19
> (unifiedbetests+0x4b2780d)
> 20:14:05 #9 impala::JvmMetricCache::GetCounterMetric(long
> (*)(impala::TGetJvmMemoryMetricsResponse const&))
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/memory-metrics.cc:305:3
> (unifiedbetests+0x4b27711)
> 20:14:05 #10 impala::JvmMemoryCounterMetric::GetValue()
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/memory-metrics.cc:270:41
> (unifiedbetests+0x4b276bf)
> 20:14:05 #11
> impala::QueryState::Init(impala::ExecQueryFInstancesRequestPB const*,
> impala::TExecPlanFragmentInfo const&)::$_3::operator()() const
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/runtime/query-state.cc:185:50
> (unifiedbetests+0x455bc15)
> 20:14:05 #12
> boost::detail::function::function_obj_invoker0<impala::QueryState::Init(impala::ExecQueryFInstancesRequestPB
> const*, impala::TExecPlanFragmentInfo const&)::$_3,
> long>::invoke(boost::detail::function::function_buffer&)
> /data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:137:18
> (unifiedbetests+0x455b9d9)
> 20:14:05 #13 boost::function0<long>::operator()() const
> /data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:763:14
> (unifiedbetests+0x4b57bf1)
> 20:14:05 #14 impala::RuntimeProfile::DerivedCounter::value() const
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/runtime-profile-counters.h:384:12
> (unifiedbetests+0x4b8b38d)
> 20:14:05 #15
> impala::RuntimeProfile::ToThriftSubclass(std::vector<std::pair<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> impala::RuntimeProfileBase::Counter const*>,
> std::allocator<std::pair<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> impala::RuntimeProfileBase::Counter const*> > >&,
> impala::TRuntimeProfileNode*) const
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/runtime-profile.cc:1783:35
> (unifiedbetests+0x4b7737b)
> 20:14:05 #16
> impala::RuntimeProfileBase::ToThriftHelper(impala::TRuntimeProfileNode*)
> const
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/runtime-profile.cc:1772:3
> (unifiedbetests+0x4b77252)
> 20:14:05 #17
> impala::RuntimeProfileBase::ToThriftHelper(std::vector<impala::TRuntimeProfileNode,
> std::allocator<impala::TRuntimeProfileNode> >*) const
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/runtime-profile.cc:1732:25
> (unifiedbetests+0x4b76e60)
> 20:14:05 #18
> impala::RuntimeProfileBase::ToThrift(impala::TRuntimeProfileTree*) const
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/runtime-profile.cc:1711:3
> (unifiedbetests+0x4b76d4f)
> 20:14:05 #19
> impala::RuntimeProfile::ToThrift(impala::TRuntimeProfileTree*) const
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/runtime-profile.cc:1716:23
> (unifiedbetests+0x4b76f20)
> 20:14:05 #20 impala::QueryState::ConstructReport(bool,
> impala::ReportExecStatusRequestPB*, impala::TRuntimeProfileForest*)
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/runtime/query-state.cc:529:18
> (unifiedbetests+0x455804b)
> 20:14:05 #21 impala::QueryState::ReportExecStatus()
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/runtime/query-state.cc:639:3
> (unifiedbetests+0x455698a)
> 20:14:05 #22 impala::QueryState::UpdateBackendExecState()
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/runtime/query-state.cc:465:40
> (unifiedbetests+0x4556673)
> 20:14:05 #23 impala::QueryState::MonitorFInstances()
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/runtime/query-state.cc:917:3
> (unifiedbetests+0x455a08f)
> 20:14:05 #24
> impala::QueryExecMgr::ExecuteQueryHelper(impala::QueryState*)
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/runtime/query-exec-mgr.cc:162:42
> (unifiedbetests+0x4546de5)
> 20:14:05 #25 boost::_mfi::mf1<void, impala::QueryExecMgr,
> impala::QueryState*>::operator()(impala::QueryExecMgr*, impala::QueryState*)
> const
> /data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/mem_fn_template.hpp:165:29
> (unifiedbetests+0x454e66d)
> 20:14:05 #26 void
> boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>,
> boost::_bi::value<impala::QueryState*> >::operator()<boost::_mfi::mf1<void,
> impala::QueryExecMgr, impala::QueryState*>,
> boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void,
> impala::QueryExecMgr, impala::QueryState*>&, boost::_bi::list0&, int)
> /data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:319:9
> (unifiedbetests+0x454e5ba)
> 20:14:05 #27 boost::_bi::bind_t<void, boost::_mfi::mf1<void,
> impala::QueryExecMgr, impala::QueryState*>,
> boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>,
> boost::_bi::value<impala::QueryState*> > >::operator()()
> /data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:1294:16
> (unifiedbetests+0x454e523)
> 20:14:05 #28
> boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void,
> boost::_mfi::mf1<void, impala::QueryExecMgr, impala::QueryState*>,
> boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>,
> boost::_bi::value<impala::QueryState*> > >,
> void>::invoke(boost::detail::function::function_buffer&)
> /data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:158:11
> (unifiedbetests+0x454e261)
> 20:14:05 #29 boost::function0<void>::operator()() const
> /data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:763:14
> (unifiedbetests+0x44b7371)
> 20:14:05 #30
> impala::Thread::SuperviseThread(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*)
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/thread.cc:360:3
> (unifiedbetests+0x4bbaa88)
> 20:14:05 #31 void
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*>
> >::operator()<void (*)(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*),
> boost::_bi::list0>(boost::_bi::type<void>, void
> (*&)(std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&, boost::function<void
> ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
> (impala::PromiseMode)0>*), boost::_bi::list0&, int)
> /data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:531:9
> (unifiedbetests+0x4bc345c)
> 20:14:05 #32 boost::_bi::bind_t<void, void
> (*)(std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&, boost::function<void
> ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
> (impala::PromiseMode)0>*),
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >
> >::operator()()
> /data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:1294:16
> (unifiedbetests+0x4bc3373)
> 20:14:05 #33 boost::detail::thread_data<boost::_bi::bind_t<void, void
> (*)(std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&, boost::function<void
> ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
> (impala::PromiseMode)0>*),
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >
> >::run()
> /data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/thread/detail/thread.hpp:120:17
> (unifiedbetests+0x4bc3060)
> 20:14:05 #34 thread_proxy <null> (unifiedbetests+0x4e0b6b6)
> {code}
> All the tests seemed to run OK, which, plus the presence of at_exit_wrapper
> in one thread suggests to me that this is happening during the exit of
> expr-test (unifiedbetests).
> See [^expr-test-tsan-failure.log] for slightly more details
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
