[ 
https://issues.apache.org/jira/browse/IMPALA-7033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16480167#comment-16480167
 ] 

ASF subversion and git services commented on IMPALA-7033:
---------------------------------------------------------

Commit c1c122a10177920903009420d2faac673d867c4b in impala's branch 
refs/heads/master from [~dhecht]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=c1c122a ]

IMPALA-5384, part 2: Simplify Coordinator locking and clarify state

The is the final change to clarify and break up the Coordinator's lock.
The state machine for the coordinator is made explicit, distinguishing
between executing state and multiple terminal states. Logic to
transition into a terminal state is centralized in one location and
executes exactly once for each coordinator object.

Derived from a patch for IMPALA_5384 by Marcel Kornacker.

Testing:
- exhaustive functional tests
- stress test on minicluster with memory overcommitment. Verified from
  the logs that this exercises all these paths:
  - successful queries
  - client requested cancellation
  - error from exec FInstances RPC
  - error reported asynchronously via report status RPC
  - eos before backend execution completed
- loop query_test & failure for 12 hours with no dchecks or crashes
  (This had previously reproduced IMPALA-7030 and IMPALA-7033 with
  the previous version of this change).

Change-Id: I6dc08da1295f1df3c9dce6d35d65d887b2c00a1c
Reviewed-on: http://gerrit.cloudera.org:8080/10440
Reviewed-by: Dan Hecht <dhe...@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>


> Impala crashes on exhaustive release tests
> ------------------------------------------
>
>                 Key: IMPALA-7033
>                 URL: https://issues.apache.org/jira/browse/IMPALA-7033
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Backend
>    Affects Versions: Impala 3.1.0
>            Reporter: Joe McDonnell
>            Assignee: Dan Hecht
>            Priority: Blocker
>              Labels: broken-build, flaky
>             Fix For: Impala 3.1.0
>
>
> Exhaustive release builds have seen crashes related to memory allocation/free:
> {noformat}
> CORE: ./core.1526387352.3540.impalad
> BINARY: ./be/build/latest/service/impalad
> Core was generated by 
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/bu'.
> Program terminated with signal 6, Aborted.
> #0 0x0000003f10e328e5 in raise () from /lib64/libc.so.6
> To enable execution of this file add
> add-auto-load-safe-path 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/gcc-4.9.2/lib64/libstdc++.so.6.0.20-gdb.py
> line to your configuration file "/var/lib/jenkins/.gdbinit".
> To completely disable this security protection add
> set auto-load safe-path /
> line to your configuration file "/var/lib/jenkins/.gdbinit".
> For more information about this security protection see the
> "Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
> info "(gdb)Auto-loading safe path"
> #0 0x0000003f10e328e5 in raise () from /lib64/libc.so.6
> #1 0x0000003f10e340c5 in abort () from /lib64/libc.so.6
> #2 0x00007f7d7ff261a5 in os::abort(bool) () from 
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #3 0x00007f7d800b6843 in VMError::report_and_die() () from 
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #4 0x00007f7d7ff2b562 in JVM_handle_linux_signal () from 
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #5 0x00007f7d7ff224f3 in signalHandler(int, siginfo*, void*) () from 
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #6 <signal handler called>
> #7 0x00000000026be93f in tc_newarray ()
> #8 0x0000000000c9d508 in allocate (this=0x11f1cae8, __n=1) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/gcc-4.9.2/include/c++/4.9.2/ext/new_allocator.h:104
> #9 allocate (this=0x11f1cae8, __n=1) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/gcc-4.9.2/include/c++/4.9.2/bits/alloc_traits.h:357
> #10 _M_allocate (this=0x11f1cae8, __n=1) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/gcc-4.9.2/include/c++/4.9.2/bits/stl_vector.h:170
> #11 std::vector<int, std::allocator<int> >::_M_default_append 
> (this=0x11f1cae8, __n=1) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/gcc-4.9.2/include/c++/4.9.2/bits/vector.tcc:557
> #12 0x0000000000cb035f in _M_default_append (this=) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/gcc-4.9.2/include/c++/4.9.2/bits/stl_map.h:506
> #13 resize (this=) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/gcc-4.9.2/include/c++/4.9.2/bits/stl_vector.h:676
> #14 impala::THdfsScanNode::read (this=) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/generated-sources/gen-cpp/PlanNodes_types.cpp:1544
> #15 0x0000000000cb3148 in impala::TPlanNode::read (this=) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/generated-sources/gen-cpp/PlanNodes_types.cpp:5302
> #16 0x0000000000cbc206 in impala::TPlan::read (this=0x235e1808, 
> iprot=0x901c000) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/generated-sources/gen-cpp/PlanNodes_types.cpp:5871
> #17 0x0000000000ccfc0d in impala::TPlanFragment::read (this=0x235e17f0, 
> iprot=0x901c000) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/generated-sources/gen-cpp/Planner_types.cpp:110
> #18 0x0000000000b5069f in impala::TPlanFragmentCtx::read (this=0x235e17e8, 
> iprot=) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/generated-sources/gen-cpp/ImpalaInternalService_types.cpp:2849
> #19 0x0000000000b52461 in impala::TExecQueryFInstancesParams::read 
> (this=0x7f7cd4b572e8, iprot=) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/generated-sources/gen-cpp/ImpalaInternalService_types.cpp:3534
> #20 0x00000000012f63ed in 
> impala::ImpalaInternalService_ExecQueryFInstances_args::read 
> (this=0x7f7cd4b572e0, iprot=0x901c000) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/generated-sources/gen-cpp/ImpalaInternalService.cpp:39
> #21 0x00000000012f6e77 in 
> impala::ImpalaInternalServiceProcessor::process_ExecQueryFInstances 
> (this=0xbc19e60, seqid=0, iprot=0x901c000, oprot=0x901c040, callContext=) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/generated-sources/gen-cpp/ImpalaInternalService.cpp:1513
> #22 0x00000000012f601d in 
> impala::ImpalaInternalServiceProcessor::dispatchCall (this=0xbc19e60, 
> iprot=0x901c000, oprot=0x901c040, fname=..., seqid=0, callContext=0x901c180) 
> at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/generated-sources/gen-cpp/ImpalaInternalService.cpp:1496
> #23 0x0000000000a3793a in apache::thrift::TDispatchProcessor::process 
> (this=0xbc19e60, in=..., out=..., connectionContext=0x901c180) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/thrift-0.9.3-p4/include/thrift/TDispatchProcessor.h:121
> #24 0x0000000000d762f9 in 
> apache::thrift::server::TAcceptQueueServer::Task::run (this=0xbc67380) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/src/rpc/TAcceptQueueServer.cpp:68
> #25 0x0000000000d6e4f9 in impala::ThriftThread::RunRunnable (this=) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/src/rpc/thrift-thread.cc:74
> #26 0x0000000000d6f372 in operator() (function_obj_ptr=) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/mem_fn_template.hpp:280
> #27 operator()<boost::_mfi::mf2<void, impala::ThriftThread, 
> boost::shared_ptr<apache::thrift::concurrency::Runnable>, 
> impala::Promise<long unsigned int>*>, boost::_bi::list0> (function_obj_ptr=) 
> at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:392
> #28 operator() (function_obj_ptr=) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
> #29 
> boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, 
> boost::_mfi::mf2<void, impala::ThriftThread, 
> boost::shared_ptr<apache::thrift::concurrency::Runnable>, 
> impala::Promise<unsigned long>*>, 
> boost::_bi::list3<boost::_bi::value<impala::ThriftThread*>, 
> boost::_bi::value<boost::shared_ptr<apache::thrift::concurrency::Runnable> >, 
> boost::_bi::value<impala::Promise<unsigned long>*> > >, void>::invoke 
> (function_obj_ptr=) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
> #30 0x0000000000f9c28f in operator() (name=) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
> #31 impala::Thread::SuperviseThread (name=) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/src/util/thread.cc:356
> #32 0x0000000000f9cd8a in operator()<void (*)(const std::basic_string<char>&, 
> const std::basic_string<char>&, boost::function<void()>, const 
> impala::ThreadDebugInfo*, impala::Promise<long int>*), boost::_bi::list0> 
> (this=0xa803400) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525
> #33 operator() (this=0xa803400) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
> #34 boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(const 
> std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, 
> const std::basic_string<char, std::char_traits<char>, std::allocator<char> 
> >&, boost::function<void()>, const impala::ThreadDebugInfo*, 
> impala::Promise<long int>*), 
> boost::_bi::list5<boost::_bi::value<std::basic_string<char, 
> std::char_traits<char>, std::allocator<char> > >, 
> boost::_bi::value<std::basic_string<char, std::char_traits<char>, 
> std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, 
> boost::_bi::value<impala::ThreadDebugInfo*>, 
> boost::_bi::value<impala::Promise<long int>*> > > >::run(void) 
> (this=0xa803400) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:116
> #35 0x000000000153f8fa in thread_proxy ()
> #36 0x0000003f11207851 in start_thread () from /lib64/libpthread.so.0
> #37 0x0000003f10ee894d in clone () from /lib64/libc.so.6{noformat}
> Additionally:
> {noformat}
> CORE: ./core.1526214223.23025.impalad
> BINARY: ./be/build/latest/service/impalad
> Core was generated by 
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/bu'.
> Program terminated with signal 6, Aborted.
> #0 0x00000034a38328e5 in raise () from /lib64/libc.so.6
> To enable execution of this file add
> add-auto-load-safe-path 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/gcc-4.9.2/lib64/libstdc++.so.6.0.20-gdb.py
> line to your configuration file "/var/lib/jenkins/.gdbinit".
> To completely disable this security protection add
> set auto-load safe-path /
> line to your configuration file "/var/lib/jenkins/.gdbinit".
> For more information about this security protection see the
> "Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
> info "(gdb)Auto-loading safe path"
> #0 0x00000034a38328e5 in raise () from /lib64/libc.so.6
> #1 0x00000034a38340c5 in abort () from /lib64/libc.so.6
> #2 0x000000000260c42d in tcmalloc::Log(tcmalloc::LogMode, char const*, int, 
> tcmalloc::LogItem, tcmalloc::LogItem, tcmalloc::LogItem, tcmalloc::LogItem) ()
> #3 0x000000000260615d in (anonymous namespace)::do_free_helper ()
> #4 0x00000000026bcce8 in tc_free ()
> #5 0x00007f85e5690279 in os::free(void*, unsigned short) () from 
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #6 0x00007f85e5800f4b in Unsafe_FreeMemory () from 
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #7 0x00007f85cba9ab68 in ?? ()
> #8 0x000000039c676880 in ?? ()
> #9 0x00000001e9003bc0 in ?? ()
> #10 0x00007f852148eca0 in ?? ()
> #11 0x000000000260a40c in tcmalloc::CentralFreeList::InsertRange(void*, 
> void*, int) ()
> #12 0x00007f85cb21b360 in ?? ()
> #13 0x0000000003d4d240 in ?? ()
> #14 0x00007f85cb21b360 in ?? ()
> #15 0x006c006200750070 in ?? ()
> #16 0x00007f85e57af13d in AbstractInterpreter::can_be_compiled(methodHandle) 
> () from /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #17 0x00007f85cb2144e7 in ?? ()
> #18 0x000000065fe86af8 in ?? ()
> #19 0x0000000000001fa1 in ?? () at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/boost-1.57.0-p3/include/boost/uuid/sha1.hpp:139
> #20 0x0000000009740590 in ?? ()
> #21 0x0000000012027000 in ?? ()
> #22 0x00007f852148ee10 in ?? ()
> #23 0x000000065fe86af8 in ?? ()
> #24 0x00007f852148edd0 in ?? ()
> #25 0x00007f852148edd0 in ?? ()
> #26 0x00007f852148f108 in ?? ()
> #27 0x000000060000000a in ?? ()
> #28 0x00007f859f00bb08 in ?? ()
> #29 0x00007f85cb2292e0 in ?? ()
> #30 0x00007f852148ef48 in ?? ()
> #31 0x00007f852148ef00 in ?? ()
> #32 0x00007f85e5420356 in JavaCalls::call_helper(JavaValue*, methodHandle*, 
> JavaCallArguments*, Thread*) () from 
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> Backtrace stopped: frame did not save the PC{noformat}
> Not seen on 2.x yet.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

Reply via email to