[
https://issues.apache.org/jira/browse/KUDU-2059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16586333#comment-16586333
]
Alexey Serbin commented on KUDU-2059:
-------------------------------------
Hit another instance of the race:
[^raft_consensus_election-itest-threadpool-race.txt.xz]
> Data race in DnsResolver
> ------------------------
>
> Key: KUDU-2059
> URL: https://issues.apache.org/jira/browse/KUDU-2059
> Project: Kudu
> Issue Type: Bug
> Components: server
> Affects Versions: 1.4.0
> Reporter: Mike Percy
> Priority: Major
> Labels: tsan
> Attachments: raft_consensus-itest.txt, raft_consensus-itest.txt.gz,
> raft_consensus_election-itest-threadpool-race.txt.xz
>
>
> I got a TSAN failure in a Jenkins run of
> RaftConsensusITest.MultiThreadedInsertWithFailovers:
> http://dist-test.cloudera.org/job?job_id=jenkins-slave.1498799877.11199
> {code}
> WARNING: ThreadSanitizer: data race (pid=14861)
> Write of size 8 at 0x7b5000006af0 by main thread:
> #0 pthread_cond_destroy
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1102
> (raft_consensus-itest+0x4a198c)
> #1 kudu::ConditionVariable::~ConditionVariable()
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/condition_variable.cc:57:12
> (libkudu_util.so+0xf107e)
> #2 kudu::ThreadPool::~ThreadPool()
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:339:1
> (libkudu_util.so+0x1c1b93)
> #3 kudu::DefaultDeleter<kudu::ThreadPool>::operator()(kudu::ThreadPool*)
> const
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/gscoped_ptr.h:145:5
> (libmaster.so+0xc0bbe)
> #4 kudu::internal::gscoped_ptr_impl<kudu::ThreadPool,
> kudu::DefaultDeleter<kudu::ThreadPool> >::~gscoped_ptr_impl()
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/gscoped_ptr.h:228:7
> (libmaster.so+0xc0b89)
> #5 gscoped_ptr<kudu::ThreadPool, kudu::DefaultDeleter<kudu::ThreadPool>
> >::~gscoped_ptr()
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/gscoped_ptr.h:318:7
> (libmaster.so+0xb1b79)
> #6 kudu::DnsResolver::~DnsResolver()
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/net/dns_resolver.cc:45:1
> (libkudu_util.so+0x1863fa)
> #7
> kudu::DefaultDeleter<kudu::DnsResolver>::operator()(kudu::DnsResolver*) const
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/gscoped_ptr.h:145:5
> (libkudu_client.so+0xd0dee)
> #8 kudu::internal::gscoped_ptr_impl<kudu::DnsResolver,
> kudu::DefaultDeleter<kudu::DnsResolver> >::reset(kudu::DnsResolver*)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/gscoped_ptr.h:254:7
> (libkudu_client.so+0xd0da4)
> #9 gscoped_ptr<kudu::DnsResolver, kudu::DefaultDeleter<kudu::DnsResolver>
> >::reset(kudu::DnsResolver*)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/gscoped_ptr.h:375:46
> (libkudu_client.so+0xc5fd0)
> #10 kudu::client::KuduClient::Data::~Data()
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/client-internal.cc:342:17
> (libkudu_client.so+0xd6b1d)
> #11 kudu::client::KuduClient::~KuduClient()
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/client.cc:334:3
> (libkudu_client.so+0xbbfec)
> #12
> std::__1::default_delete<kudu::client::KuduClient>::operator()(kudu::client::KuduClient*)
> const
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/c++/v1/memory:2397:13
> (libkudu_client.so+0xd06bb)
> #13 std::__1::__shared_ptr_pointer<kudu::client::KuduClient*,
> std::__1::default_delete<kudu::client::KuduClient>,
> std::__1::allocator<kudu::client::KuduClient> >::__on_zero_shared()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/c++/v1/memory:3795
> (libkudu_client.so+0xd06bb)
> #14 __release_shared
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/libcxx/src/memory.cpp:67:9
> (libc++.so.1+0xc095d)
> #15 std::__1::__shared_weak_count::__release_shared()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/libcxx/src/memory.cpp:92
> (libc++.so.1+0xc095d)
> #16 std::__1::shared_ptr<kudu::client::KuduClient>::~shared_ptr()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/c++/v1/memory:4626:19
> (raft_consensus-itest+0x548818)
> #17
> kudu::tserver::TabletServerIntegrationTestBase::~TabletServerIntegrationTestBase()
>
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/ts_itest-base.h:69:7
> (raft_consensus-itest+0x54890f)
> #18 kudu::tserver::RaftConsensusITest::~RaftConsensusITest()
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/raft_consensus-itest.cc:107:7
> (raft_consensus-itest+0x547887)
> #19
> kudu::tserver::RaftConsensusITest_MultiThreadedInsertWithFailovers_Test::~RaftConsensusITest_MultiThreadedInsertWithFailovers_Test()
>
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/raft_consensus-itest.cc:1040:1
> (raft_consensus-itest+0x547ac9)
> #20 testing::Test::DeleteSelf_()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/include/gtest/gtest.h:453:24
> (libgmock.so+0x53187)
> #21 void
> testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test,
> void>(testing::Test*, void (testing::Test::*)(), char const*)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10
> (libgmock.so+0x52b39)
> #22 void
> testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,
> void>(testing::Test*, void (testing::Test::*)(), char const*)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438
> (libgmock.so+0x52b39)
> #23 testing::TestInfo::Run()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2661:3
> (libgmock.so+0x340a0)
> #24 testing::TestCase::Run()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28
> (libgmock.so+0x34da6)
> #25 testing::internal::UnitTestImpl::RunAllTests()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43
> (libgmock.so+0x40896)
> #26 bool
> testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,
> bool>(testing::internal::UnitTestImpl*, bool
> (testing::internal::UnitTestImpl::*)(), char const*)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10
> (libgmock.so+0x53a19)
> #27 bool
> testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,
> bool>(testing::internal::UnitTestImpl*, bool
> (testing::internal::UnitTestImpl::*)(), char const*)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438
> (libgmock.so+0x53a19)
> #28 testing::UnitTest::Run()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10
> (libgmock.so+0x402e1)
> #29 RUN_ALL_TESTS()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46
> (libkudu_test_main.so+0x33db)
> #30 main
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/test_main.cc:104:13
> (libkudu_test_main.so+0x2b96)
> Previous read of size 8 at 0x7b5000006af0 by thread T33:
> #0 pthread_cond_signal
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1088
> (raft_consensus-itest+0x4a16c0)
> #1 kudu::ConditionVariable::Signal()
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/condition_variable.cc:136:12
> (libkudu_util.so+0xf14fe)
> #2 kudu::ThreadPool::DoSubmit(std::__1::shared_ptr<kudu::Runnable>,
> kudu::ThreadPoolToken*)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:529:14
> (libkudu_util.so+0x1c099b)
> #3 kudu::ThreadPool::Submit(std::__1::shared_ptr<kudu::Runnable>)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:452:10
> (libkudu_util.so+0x1c260f)
> #4 kudu::ThreadPool::SubmitFunc(boost::function<void ()>)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:448:10
> (libkudu_util.so+0x1c26a9)
> #5 kudu::DnsResolver::ResolveAddresses(kudu::HostPort const&,
> std::__1::vector<kudu::Sockaddr, std::__1::allocator<kudu::Sockaddr> >*,
> kudu::Callback<void (kudu::Status const&)> const&)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/net/dns_resolver.cc:57:21
> (libkudu_util.so+0x1864ab)
> #6
> kudu::client::internal::RemoteTabletServer::InitProxy(kudu::client::KuduClient*,
> kudu::Callback<void (kudu::Status const&)> const&)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/meta_cache.cc:133:33
> (libkudu_client.so+0xf8bf3)
> #7
> kudu::client::internal::MetaCacheServerPicker::PickLeader(kudu::Callback<void
> (kudu::Status const&, kudu::client::internal::RemoteTabletServer*)> const&,
> kudu::MonoTime const&)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/meta_cache.cc:441:11
> (libkudu_client.so+0xfaeaf)
> #8 kudu::rpc::RetriableRpc<kudu::client::internal::RemoteTabletServer,
> kudu::tserver::WriteRequestPB, kudu::tserver::WriteResponsePB>::SendRpc()
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/retriable_rpc.h:144:19
> (libkudu_client.so+0xb2ad8)
> #9 kudu::rpc::RpcRetrier::DelayedRetryCb(kudu::rpc::Rpc*, kudu::Status
> const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/rpc.cc:89:10
> (libkrpc.so+0xcc72c)
> #10 boost::_mfi::mf2<void, kudu::rpc::RpcRetrier, kudu::rpc::Rpc*,
> kudu::Status const&>::operator()(kudu::rpc::RpcRetrier*, kudu::rpc::Rpc*,
> kudu::Status const&) const
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/bind/mem_fn_template.hpp:280:29
> (libkrpc.so+0xcd117)
> #11 void boost::_bi::list3<boost::_bi::value<kudu::rpc::RpcRetrier*>,
> boost::_bi::value<kudu::rpc::Rpc*>, boost::arg<1>
> >::operator()<boost::_mfi::mf2<void, kudu::rpc::RpcRetrier, kudu::rpc::Rpc*,
> kudu::Status const&>, boost::_bi::rrlist1<kudu::Status const&>
> >(boost::_bi::type<void>, boost::_mfi::mf2<void, kudu::rpc::RpcRetrier,
> kudu::rpc::Rpc*, kudu::Status const&>&, boost::_bi::rrlist1<kudu::Status
> const&>&, int)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/bind/bind.hpp:398:9
> (libkrpc.so+0xcd052)
> #12 void boost::_bi::bind_t<void, boost::_mfi::mf2<void,
> kudu::rpc::RpcRetrier, kudu::rpc::Rpc*, kudu::Status const&>,
> boost::_bi::list3<boost::_bi::value<kudu::rpc::RpcRetrier*>,
> boost::_bi::value<kudu::rpc::Rpc*>, boost::arg<1> >
> >::operator()<kudu::Status const&>(kudu::Status const&)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/bind/bind.hpp:1234:16
> (libkrpc.so+0xccf78)
> #13
> boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
> boost::_mfi::mf2<void, kudu::rpc::RpcRetrier, kudu::rpc::Rpc*, kudu::Status
> const&>, boost::_bi::list3<boost::_bi::value<kudu::rpc::RpcRetrier*>,
> boost::_bi::value<kudu::rpc::Rpc*>, boost::arg<1> > >, void, kudu::Status
> const&>::invoke(boost::detail::function::function_buffer&, kudu::Status
> const&)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/function/function_template.hpp:159:11
> (libkrpc.so+0xccd13)
> #14 boost::function1<void, kudu::Status const&>::operator()(kudu::Status
> const&) const
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/function/function_template.hpp:770:14
> (libkrpc.so+0xbb064)
> #15 kudu::rpc::DelayedTask::TimerHandler(ev::timer&, int)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/reactor.cc:579:5
> (libkrpc.so+0xb90b4)
> #16 void ev::base<ev_timer,
> ev::timer>::method_thunk<kudu::rpc::DelayedTask,
> &kudu::rpc::DelayedTask::TimerHandler>(ev_loop*, ev_timer*, int)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/ev++.h:479:7
> (libkrpc.so+0xbf95a)
> #17 ev_invoke_pending
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/libev-4.20/ev.c:3155:11
> (libev.so.4+0x97ec)
> #18 ev_run
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/libev-4.20/ev.c:3555:7
> (libev.so.4+0xa8fd)
> #19 ev::loop_ref::run(int)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/ev++.h:211:7
> (libkrpc.so+0xba938)
> #20 kudu::rpc::ReactorThread::RunThread()
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/reactor.cc:356:9
> (libkrpc.so+0xb566a)
> #21 boost::_mfi::mf0<void,
> kudu::rpc::ReactorThread>::operator()(kudu::rpc::ReactorThread*) const
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/bind/mem_fn_template.hpp:49:29
> (libkrpc.so+0xbdbc6)
> #22 void boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*>
> >::operator()<boost::_mfi::mf0<void, kudu::rpc::ReactorThread>,
> boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void,
> kudu::rpc::ReactorThread>&, boost::_bi::list0&, int)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/bind/bind.hpp:259:9
> (libkrpc.so+0xbdb1a)
> #23 boost::_bi::bind_t<void, boost::_mfi::mf0<void,
> kudu::rpc::ReactorThread>,
> boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> >
> >::operator()()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/bind/bind.hpp:1222:16
> (libkrpc.so+0xbdaa3)
> #24
> boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void,
> boost::_mfi::mf0<void, kudu::rpc::ReactorThread>,
> boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> > >,
> void>::invoke(boost::detail::function::function_buffer&)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/function/function_template.hpp:159:11
> (libkrpc.so+0xbd8a9)
> #25 boost::function0<void>::operator()() const
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/function/function_template.hpp:770:14
> (libkrpc.so+0xb1391)
> #26 kudu::Thread::SuperviseThread(void*)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.cc:591:3
> (libkudu_util.so+0x1ba91e)
> As if synchronized via sleep:
> #0 nanosleep
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:357
> (raft_consensus-itest+0x49e017)
> #1 boost::detail::yield(unsigned int)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/smart_ptr/detail/yield_k.hpp:151:9
> (libmaster.so+0xd1ff6)
> #2 kudu::Thread::StartThread(std::__1::basic_string<char,
> std::__1::char_traits<char>, std::__1::allocator<char> > const&,
> std::__1::basic_string<char, std::__1::char_traits<char>,
> std::__1::allocator<char> > const&, boost::function<void ()> const&, unsigned
> long, scoped_refptr<kudu::Thread>*)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.cc:545:7
> (libkudu_util.so+0x1ba301)
> #3 kudu::Status kudu::Thread::Create<void (kudu::ThreadPool::*)(bool),
> kudu::ThreadPool*, bool>(std::__1::basic_string<char,
> std::__1::char_traits<char>, std::__1::allocator<char> > const&,
> std::__1::basic_string<char, std::__1::char_traits<char>,
> std::__1::allocator<char> > const&, void (kudu::ThreadPool::* const&)(bool),
> kudu::ThreadPool* const&, bool const&, scoped_refptr<kudu::Thread>*)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.h:164:12
> (libkudu_util.so+0x1c4916)
> #4 kudu::ThreadPool::CreateThreadUnlocked()
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:694:14
> (libkudu_util.so+0x1c22f1)
> #5 kudu::ThreadPool::DoSubmit(std::__1::shared_ptr<kudu::Runnable>,
> kudu::ThreadPoolToken*)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:491:21
> (libkudu_util.so+0x1c07b1)
> #6 kudu::ThreadPool::Submit(std::__1::shared_ptr<kudu::Runnable>)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:452:10
> (libkudu_util.so+0x1c260f)
> #7 kudu::ThreadPool::SubmitFunc(boost::function<void ()>)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:448:10
> (libkudu_util.so+0x1c26a9)
> #8 kudu::DnsResolver::ResolveAddresses(kudu::HostPort const&,
> std::__1::vector<kudu::Sockaddr, std::__1::allocator<kudu::Sockaddr> >*,
> kudu::Callback<void (kudu::Status const&)> const&)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/net/dns_resolver.cc:57:21
> (libkudu_util.so+0x1864ab)
> #9
> kudu::client::internal::RemoteTabletServer::InitProxy(kudu::client::KuduClient*,
> kudu::Callback<void (kudu::Status const&)> const&)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/meta_cache.cc:133:33
> (libkudu_client.so+0xf8bf3)
> #10
> kudu::client::KuduClient::Data::GetTabletServer(kudu::client::KuduClient*,
> scoped_refptr<kudu::client::internal::RemoteTablet> const&,
> kudu::client::KuduClient::ReplicaSelection,
> std::__1::set<std::__1::basic_string<char, std::__1::char_traits<char>,
> std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char,
> std::__1::char_traits<char>, std::__1::allocator<char> > >,
> std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>,
> std::__1::allocator<char> > > > const&,
> std::__1::vector<kudu::client::internal::RemoteTabletServer*,
> std::__1::allocator<kudu::client::internal::RemoteTabletServer*> >*,
> kudu::client::internal::RemoteTabletServer**)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/client-internal.cc:423:8
> (libkudu_client.so+0xd71f9)
> #11
> kudu::client::KuduScanner::Data::OpenTablet(std::__1::basic_string<char,
> std::__1::char_traits<char>, std::__1::allocator<char> > const&,
> kudu::MonoTime const&, std::__1::set<std::__1::basic_string<char,
> std::__1::char_traits<char>, std::__1::allocator<char> >,
> std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>,
> std::__1::allocator<char> > >,
> std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>,
> std::__1::allocator<char> > > >*)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/scanner-internal.cc:389:53
> (libkudu_client.so+0x11ca33)
> #12 kudu::client::KuduScanner::Data::OpenNextTablet(kudu::MonoTime
> const&, std::__1::set<std::__1::basic_string<char,
> std::__1::char_traits<char>, std::__1::allocator<char> >,
> std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>,
> std::__1::allocator<char> > >,
> std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>,
> std::__1::allocator<char> > > >*)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/scanner-internal.cc:244:10
> (libkudu_client.so+0x11c1ef)
> #13 kudu::client::KuduScanner::Open()
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/client.cc:1331:3
> (libkudu_client.so+0xc287d)
> #14 kudu::ClusterVerifier::DoCheckRowCount(std::__1::basic_string<char,
> std::__1::char_traits<char>, std::__1::allocator<char> > const&,
> kudu::ClusterVerifier::ComparisonMode, int)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/cluster_verifier.cc:138:3
> (libintegration-tests.so+0x574b4)
> #15 kudu::ClusterVerifier::CheckRowCount(std::__1::basic_string<char,
> std::__1::char_traits<char>, std::__1::allocator<char> > const&,
> kudu::ClusterVerifier::ComparisonMode, int)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/cluster_verifier.cc:116:3
> (libintegration-tests.so+0x57138)
> #16
> kudu::tserver::TabletServerIntegrationTestBase::AssertAllReplicasAgree(int)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/ts_itest-base.h:523:5
> (raft_consensus-itest+0x53ca2f)
> #17
> kudu::tserver::RaftConsensusITest_MultiThreadedInsertWithFailovers_Test::TestBody()
>
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/raft_consensus-itest.cc:1096:3
> (raft_consensus-itest+0x5100cb)
> #18 void
> testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test,
> void>(testing::Test*, void (testing::Test::*)(), char const*)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10
> (libgmock.so+0x52b39)
> #19 void
> testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,
> void>(testing::Test*, void (testing::Test::*)(), char const*)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438
> (libgmock.so+0x52b39)
> #20 testing::Test::Run()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5
> (libgmock.so+0x32b97)
> #21 testing::TestInfo::Run()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11
> (libgmock.so+0x34036)
> #22 testing::TestCase::Run()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28
> (libgmock.so+0x34da6)
> #23 testing::internal::UnitTestImpl::RunAllTests()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43
> (libgmock.so+0x40896)
> #24 bool
> testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,
> bool>(testing::internal::UnitTestImpl*, bool
> (testing::internal::UnitTestImpl::*)(), char const*)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10
> (libgmock.so+0x53a19)
> #25 bool
> testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,
> bool>(testing::internal::UnitTestImpl*, bool
> (testing::internal::UnitTestImpl::*)(), char const*)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438
> (libgmock.so+0x53a19)
> #26 testing::UnitTest::Run()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10
> (libgmock.so+0x402e1)
> #27 RUN_ALL_TESTS()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46
> (libkudu_test_main.so+0x33db)
> #28 main
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/test_main.cc:104:13
> (libkudu_test_main.so+0x2b96)
> Location is heap block of size 504 at 0x7b5000006a00 allocated by main
> thread:
> #0 operator new(unsigned long)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:41
> (raft_consensus-itest+0x5028f3)
> #1 kudu::ThreadPoolBuilder::Build(gscoped_ptr<kudu::ThreadPool,
> kudu::DefaultDeleter<kudu::ThreadPool> >*) const
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:124:15
> (libkudu_util.so+0x1bfaba)
> #2 kudu::DnsResolver::DnsResolver()
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/net/dns_resolver.cc:38:3
> (libkudu_util.so+0x186270)
> #3
> kudu::client::KuduClientBuilder::Build(std::__1::shared_ptr<kudu::client::KuduClient>*)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/client.cc:315:37
> (libkudu_client.so+0xbb81f)
> #4
> kudu::tserver::TabletServerIntegrationTestBase::CreateClient(std::__1::shared_ptr<kudu::client::KuduClient>*)
>
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/ts_itest-base.h:488:5
> (raft_consensus-itest+0x545f82)
> #5
> kudu::tserver::TabletServerIntegrationTestBase::BuildAndStart(std::__1::vector<std::__1::basic_string<char,
> std::__1::char_traits<char>, std::__1::allocator<char> >,
> std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>,
> std::__1::allocator<char> > > > const&,
> std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>,
> std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char,
> std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/ts_itest-base.h:513:5
> (raft_consensus-itest+0x53b576)
> #6
> kudu::tserver::RaftConsensusITest_MultiThreadedInsertWithFailovers_Test::TestBody()
>
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/raft_consensus-itest.cc:1055:3
> (raft_consensus-itest+0x50f8fd)
> #7 void
> testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test,
> void>(testing::Test*, void (testing::Test::*)(), char const*)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10
> (libgmock.so+0x52b39)
> #8 void
> testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,
> void>(testing::Test*, void (testing::Test::*)(), char const*)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438
> (libgmock.so+0x52b39)
> #9 testing::Test::Run()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5
> (libgmock.so+0x32b97)
> #10 testing::TestInfo::Run()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11
> (libgmock.so+0x34036)
> #11 testing::TestCase::Run()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28
> (libgmock.so+0x34da6)
> #12 testing::internal::UnitTestImpl::RunAllTests()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43
> (libgmock.so+0x40896)
> #13 bool
> testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,
> bool>(testing::internal::UnitTestImpl*, bool
> (testing::internal::UnitTestImpl::*)(), char const*)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10
> (libgmock.so+0x53a19)
> #14 bool
> testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,
> bool>(testing::internal::UnitTestImpl*, bool
> (testing::internal::UnitTestImpl::*)(), char const*)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438
> (libgmock.so+0x53a19)
> #15 testing::UnitTest::Run()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10
> (libgmock.so+0x402e1)
> #16 RUN_ALL_TESTS()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46
> (libkudu_test_main.so+0x33db)
> #17 main
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/test_main.cc:104:13
> (libkudu_test_main.so+0x2b96)
> Thread T33 'rpc reactor-160' (tid=16071, running) created by main thread at:
> #0 pthread_create
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:897
> (raft_consensus-itest+0x4a2703)
> #1 kudu::Thread::StartThread(std::__1::basic_string<char,
> std::__1::char_traits<char>, std::__1::allocator<char> > const&,
> std::__1::basic_string<char, std::__1::char_traits<char>,
> std::__1::allocator<char> > const&, boost::function<void ()> const&, unsigned
> long, scoped_refptr<kudu::Thread>*)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.cc:514:15
> (libkudu_util.so+0x1ba117)
> #2 kudu::Status kudu::Thread::Create<void
> (kudu::rpc::ReactorThread::*)(),
> kudu::rpc::ReactorThread*>(std::__1::basic_string<char,
> std::__1::char_traits<char>, std::__1::allocator<char> > const&,
> std::__1::basic_string<char, std::__1::char_traits<char>,
> std::__1::allocator<char> > const&, void (kudu::rpc::ReactorThread::*
> const&)(), kudu::rpc::ReactorThread* const&, scoped_refptr<kudu::Thread>*)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.h:158:12
> (libkrpc.so+0xba1d5)
> #3 kudu::rpc::ReactorThread::Init()
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/reactor.cc:129:10
> (libkrpc.so+0xb54c5)
> #4 kudu::rpc::Reactor::Init()
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/reactor.cc:594:18
> (libkrpc.so+0xb9461)
> #5 kudu::rpc::Messenger::Init()
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/messenger.cc:515:5
> (libkrpc.so+0xa0f32)
> #6
> kudu::rpc::MessengerBuilder::Build(std::__1::shared_ptr<kudu::rpc::Messenger>*)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/messenger.cc:286:3
> (libkrpc.so+0xa0525)
> #7
> kudu::client::KuduClientBuilder::Build(std::__1::shared_ptr<kudu::client::KuduClient>*)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/client.cc:295:3
> (libkudu_client.so+0xbb55f)
> #8
> kudu::tserver::TabletServerIntegrationTestBase::CreateClient(std::__1::shared_ptr<kudu::client::KuduClient>*)
>
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/ts_itest-base.h:488:5
> (raft_consensus-itest+0x545f82)
> #9
> kudu::tserver::TabletServerIntegrationTestBase::BuildAndStart(std::__1::vector<std::__1::basic_string<char,
> std::__1::char_traits<char>, std::__1::allocator<char> >,
> std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>,
> std::__1::allocator<char> > > > const&,
> std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>,
> std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char,
> std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/ts_itest-base.h:513:5
> (raft_consensus-itest+0x53b576)
> #10
> kudu::tserver::RaftConsensusITest_MultiThreadedInsertWithFailovers_Test::TestBody()
>
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/raft_consensus-itest.cc:1055:3
> (raft_consensus-itest+0x50f8fd)
> #11 void
> testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test,
> void>(testing::Test*, void (testing::Test::*)(), char const*)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10
> (libgmock.so+0x52b39)
> #12 void
> testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,
> void>(testing::Test*, void (testing::Test::*)(), char const*)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438
> (libgmock.so+0x52b39)
> #13 testing::Test::Run()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5
> (libgmock.so+0x32b97)
> #14 testing::TestInfo::Run()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11
> (libgmock.so+0x34036)
> #15 testing::TestCase::Run()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28
> (libgmock.so+0x34da6)
> #16 testing::internal::UnitTestImpl::RunAllTests()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43
> (libgmock.so+0x40896)
> #17 bool
> testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,
> bool>(testing::internal::UnitTestImpl*, bool
> (testing::internal::UnitTestImpl::*)(), char const*)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10
> (libgmock.so+0x53a19)
> #18 bool
> testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,
> bool>(testing::internal::UnitTestImpl*, bool
> (testing::internal::UnitTestImpl::*)(), char const*)
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438
> (libgmock.so+0x53a19)
> #19 testing::UnitTest::Run()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10
> (libgmock.so+0x402e1)
> #20 RUN_ALL_TESTS()
> /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46
> (libkudu_test_main.so+0x33db)
> #21 main
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/test_main.cc:104:13
> (libkudu_test_main.so+0x2b96)
> SUMMARY: ThreadSanitizer: data race
> /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/condition_variable.cc:57:12
> in kudu::ConditionVariable::~ConditionVariable()
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
