Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/10415 )
Change subject: IMPALA-6034: Add Cpu and scanned bytes limits per query
......................................................................
Patch Set 6:
(2 comments)
I ran a local stress test with avro/snappy and it crashed almost immediately:
(gdb) down
#9 0x00000000030e0378 in impala::Coordinator::AggregateBackendsResourceUsage
(this=0x0,
query_resource_utilization=0x7f2f1ddc1270) at
be/src/runtime/coordinator.cc:823
823 for (BackendState* backend_state: backend_states_) {
(gdb) p this
$4 = (impala::Coordinator * const) 0x0
(gdb) bt
#0 0x00007f2f8befb428 in __GI_raise (sig=sig@entry=6) at
../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007f2f8befd02a in __GI_abort () at abort.c:89
#2 0x00007f2f8ee372c9 in ?? () from
/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#3 0x00007f2f8efe7c77 in ?? () from
/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#4 0x00007f2f8ee405af in JVM_handle_linux_signal ()
from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#5 0x00007f2f8ee34408 in ?? () from
/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#6 <signal handler called>
#7 0x00000000030e5912 in
__gnu_cxx::__normal_iterator<impala::Coordinator::BackendState**,
std::vector<impala::Coordinator::BackendState*,
std::allocator<impala::Coordinator::BackendState*> > >::__normal_iterator
(this=0x7f2f1ddc11c0,
__i=@0x10: <error reading variable>)
at
/home/tarmstrong/Impala/incubator-impala/toolchain/gcc-4.9.2/include/c++/4.9.2/bits/stl_iterator.h:729
#8 0x00000000030e3e59 in std::vector<impala::Coordinator::BackendState*,
std::allocator<impala::Coordinator::BackendState*> >::begin (this=0x10)
at
/home/tarmstrong/Impala/incubator-impala/toolchain/gcc-4.9.2/include/c++/4.9.2/bits/stl_vector.h:548
#9 0x00000000030e0378 in impala::Coordinator::AggregateBackendsResourceUsage
(this=0x0,
query_resource_utilization=0x7f2f1ddc1270) at
be/src/runtime/coordinator.cc:823
#10 0x0000000001e0210f in impala::ImpalaServer::ExpireQueries (this=0xc611800)
at be/src/service/impala-server.cc:2001
#11 0x0000000001e322c5 in boost::_mfi::mf0<void,
impala::ImpalaServer>::operator() (this=0x7f2f1ddc1ce8, p=0xc611800)
at
/opt/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/mem_fn_template.hpp:49
#12 0x0000000001e3095c in
boost::_bi::list1<boost::_bi::value<impala::ImpalaServer*>
>::operator()<boost::_mfi::mf0<void, impala::ImpalaServer>, boost::_bi::list0>
(this=0x7f2f1ddc1cf8, f=..., a=...)
at /opt/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:253
#13 0x0000000001e2de65 in boost::_bi::bind_t<void, boost::_mfi::mf0<void,
impala::ImpalaServer>,
boost::_bi::list1<boost::_bi::value<impala::ImpalaServer*> > >::operator()
(this=0x7f2f1ddc1ce8)
at
/opt/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
#14 0x0000000001e2a6e7 in
boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void,
boost::_mfi::mf0<void, impala::ImpalaServer>,
boost::_bi::list1<boost::_bi::value<impala::ImpalaServer*> > >, void>::invoke (
function_obj_ptr=...) at
/opt/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
#15 0x0000000001b92aca in boost::function0<void>::operator()
(this=0x7f2f1ddc1ce0)
at
/opt/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
#16 0x0000000001f865e9 in impala::Thread::SuperviseThread(std::string const&,
std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
impala::Promise<long, (impala::PromiseMode)0>*) (name="query-expirer",
category="impala-server", functor=..., parent_thread_info=0x0,
thread_started=0x7fff1e696170)
at be/src/util/thread.cc:356
#17 0x0000000001f8e6c1 in boost::_bi::list5<boost::_bi::value<std::string>,
boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >,
boost::_bi::value<impala::ThreadDebugInfo*>,
boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*>
>::operator()<void (*)(std::string const&, std::string const&,
boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
(impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void
(*&)(std::string const&, std::string const&, boost::function<void ()>,
impala::ThreadDebugInfo const*, impala::Promise<long,
(impala::PromiseMode)0>*), boost::_bi::list0&, int) (this=0xe6561c0,
f=@0xe6561b8: 0x1f86282 <impala::Thread::SuperviseThread(std::string
const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo
const*, impala::Promise<long, (impala::PromiseMode)0>*)>, a=...)
at /opt/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525
#18 0x0000000001f8e5e5 in boost::_bi::bind_t<void, void (*)(std::string const&,
std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
impala::Promise<long, (impala::PromiseMode)0>*),
boost::_bi::list5<boost::_bi::value<std::string>,
boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >,
boost::_bi::value<impala::ThreadDebugInfo*>,
boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >
>::operator()() (
this=0xe6561b8) at
/opt/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
#19 0x0000000001f8e5a8 in boost::detail::thread_data<boost::_bi::bind_t<void,
void (*)(std::string const&, std::string const&, boost::function<void ()>,
impala::ThreadDebugInfo const*, impala::Promise<long,
(impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>,
boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >,
boost::_bi::value<impala::ThreadDebugInfo*>,
boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::run()
(this=0xe656000) at
/opt/Impala-Toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:116
#20 0x00000000031db81a in thread_proxy ()
#21 0x00007f2f8c2976ba in start_thread (arg=0x7f2f1ddc2700) at
pthread_create.c:333
#22 0x00007f2f8bfcd41d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
http://gerrit.cloudera.org:8080/#/c/10415/6//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10415/6//COMMIT_MSG@18
PS6, Line 18: Added tests for various permutations for MAX_CPU_TIME_S and
MAX_SCAN_BYTES
Something I noticed after playing around: How about we rename this to have a
_limit suffix for consistency with other options.
E.g.
s/MAX_CPU_TIME_S/CPU_TIME_LIMIT_S/
s/MAX_SCAN_BYTES/SCAN_BYTES_LIMIT/
http://gerrit.cloudera.org:8080/#/c/10415/6/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/10415/6/be/src/service/impala-server.cc@1970
PS6, Line 1970: expired
This comment applies to my previous work here, but "expired" is actually kind
of a weird word choice. Something like "terminated" seems more natural to me.
--
To view, visit http://gerrit.cloudera.org:8080/10415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4c6015e21da684bb9f33e236d71309dd4c178a20
Gerrit-Change-Number: 10415
Gerrit-PatchSet: 6
Gerrit-Owner: Mostafa Mokhtar <[email protected]>
Gerrit-Reviewer: Bikramjeet Vig <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Mostafa Mokhtar <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
Gerrit-Comment-Date: Sat, 16 Jun 2018 00:28:16 +0000
Gerrit-HasComments: Yes
