[
https://issues.apache.org/jira/browse/IMPALA-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711772#comment-16711772
]
ASF subversion and git services commented on IMPALA-6955:
---------------------------------------------------------
Commit 0164fa769ee1efcdbea4d5ba0847fe3fa979c303 in impala's branch
refs/heads/master from [~lv]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=0164fa7 ]
IMPALA-6955: fix test_query_concurrency and server startup sequence
custom_cluster/test_query_concurrency created two threads to poll the
query_plan debug webpage without joining them. Then they could outlive
the test itself and continued to poll the webpage while the minicluster
of the next test started up.
During startup of the coordinator, the ImpalaServer would register its
HTTP handlers with the webserver before registering itself with the
ExecEnv. When the incoming request from the polling threads called
GetClientRequestState(), that call would dereference a nullptr returned
by the ExecEnv, which would cause the process to crash.
To fix this we join the threads in test_query_concurrency before
returning from the test method.
To fix the underlying race that made the crash possible we change the
initialization order to register the HTTP handlers after the
ImpalaServer has been registered with the ExecEnv. We also add some
DCHECKs to make sure that we access the ImpalaServer only through a
singleton instance and that it is properly registered with the ExecEnv.
Change-Id: If22f71ab6edaf9a6b46afc0985c73dc4625b5103
Reviewed-on: http://gerrit.cloudera.org:8080/12019
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Debug webpage request for unknown query ID crashes Impala in
> GetClientRequestState
> ----------------------------------------------------------------------------------
>
> Key: IMPALA-6955
> URL: https://issues.apache.org/jira/browse/IMPALA-6955
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 3.1.0
> Reporter: Vuk Ercegovac
> Assignee: Lars Volker
> Priority: Critical
> Labels: broken-build, flaky
>
> Ran into the following crash on a rhel test recently:
> {noformat}
> Error starting cluster: num_known_live_backends did not reach expected value
> in time{noformat}
> Backtrace:
> {noformat}
> #0 0x00007f92365185c9 in raise () from /lib64/libc.so.6
> #1 0x00007f9236519cd8 in abort () from /lib64/libc.so.6
> #2 0x00007f92393841a5 in os::abort(bool) () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #3 0x00007f9239514843 in VMError::report_and_die() () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #4 0x00007f9239389562 in JVM_handle_linux_signal () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #5 0x00007f92393804f3 in signalHandler(int, siginfo*, void*) () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #6 <signal handler called>
> #7 0x00000000016fded0 in base::subtle::NoBarrier_CompareAndSwap (ptr=0x238,
> old_value=0, new_value=1) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-rhel7/repos/Impala/be/src/gutil/atomicops-internals-x86.h:85
> #8 0x00000000016fdf50 in base::subtle::Acquire_CompareAndSwap (ptr=0x238,
> old_value=0, new_value=1) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-rhel7/repos/Impala/be/src/gutil/atomicops-internals-x86.h:138
> #9 0x00000000016fe26c in base::SpinLock::Lock (this=0x238) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-rhel7/repos/Impala/be/src/gutil/spinlock.h:74
> #10 0x00000000016fe2f6 in impala::SpinLock::lock (this=0x238) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-rhel7/repos/Impala/be/src/util/spinlock.h:34
> #11 0x0000000001aa8c96 in
> impala::ScopedShardedMapRef<std::shared_ptr<impala::ClientRequestState>
> >::ScopedShardedMapRef (this=0x7f91aa81eb90, query_id=..., sharded_map=0x1c0)
> at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-rhel7/repos/Impala/be/src/util/sharded-query-map-util.h:99
> #12 0x0000000001a999e2 in impala::ImpalaServer::GetClientRequestState
> (this=0xa569000, query_id=...) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-rhel7/repos/Impala/be/src/service/impala-server.cc:2123
> #13 0x0000000001b3ace6 in impala::ImpalaHttpHandler::QuerySummaryHandler
> (this=0x6f057a0, include_json_plan=true, include_summary=true, args=...,
> document=0x7f91aa81f230) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-rhel7/repos/Impala/be/src/service/impala-http-handler.cc:755
> #14 0x0000000001b3cc11 in impala::ImpalaHttpHandler::<lambda(const auto:5&,
> auto:6*)>::operator()<std::map<std::basic_string<char>,
> std::basic_string<char> >, rapidjson::GenericDocument<rapidjson::UTF8<> >
> >(const std::map<std::basic_string<char, std::char_traits<char>,
> std::allocator<char> >, std::basic_string<char, std::char_traits<char>,
> std::allocator<char> >, std::less<std::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> std::allocator<std::pair<std::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const, std::basic_string<char, std::char_traits<char>,
> std::allocator<char> > > > > &,
> rapidjson::GenericDocument<rapidjson::UTF8<char>,
> rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > *) const
> (__closure=0xd9884b8, args=..., doc=0x7f91aa81f230) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-rhel7/repos/Impala/be/src/service/impala-http-handler.cc:132
> #15 0x0000000001b3cc46 in
> boost::detail::function::void_function_obj_invoker2<impala::ImpalaHttpHandler::RegisterHandlers(impala::Webserver*)::<lambda(const
> auto:5&, auto:6*)>, void, const std::map<std::basic_string<char,
> std::char_traits<char>, std::allocator<char> >, std::basic_string<char,
> std::char_traits<char>, std::allocator<char> >,
> std::less<std::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, std::allocator<std::pair<const
> std::basic_string<char, std::char_traits<char>, std::allocator<char> >,
> std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >
> >&, rapidjson::GenericDocument<rapidjson::UTF8<char>,
> rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>
> >*>::invoke(boost::detail::function::function_buffer &, const
> std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char>
> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >,
> std::le\
> ss<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >,
> std::allocator<std::pair<std::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const, std::basic_string<char, std::char_traits<char>,
> std::allocator<char> > > > > &,
> rapidjson::GenericDocument<rapidjson::UTF8<char>,
> rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > *)
> (function_obj_ptr=..., a0=..., a1=0x7f91aa81f230) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-rhel7/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
> #16 0x0000000001c4f528 in boost::function2<void, std::map<std::string,
> std::string, std::less<std::string>, std::allocator<std::pair<std::string
> const, std::string> > > const&,
> rapidjson::GenericDocument<rapidjson::UTF8<char>,
> rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >*>::operator()
> (this=0xd9884b0, a0=..., a1=0x7f91aa81f230) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-rhel7/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
> #17 0x0000000001c4d877 in impala::Webserver::RenderUrlWithTemplate
> (this=0xc590b40, arguments=..., url_handler=..., output=0x7f91aa81f820,
> content_type=0x7f91aa81f99c) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-rhel7/repos/Impala/be/src/util/webserver.cc:440
> #18 0x0000000001c4d384 in impala::Webserver::BeginRequestCallback
> (this=0xc590b40, connection=0xc46c000, request_info=0xc46c000) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-rhel7/repos/Impala/be/src/util/webserver.cc:411
> #19 0x0000000001c4ce25 in impala::Webserver::BeginRequestCallbackStatic
> (connection=0xc46c000) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-rhel7/repos/Impala/be/src/util/webserver.cc:370{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
