[
https://issues.apache.org/jira/browse/KUDU-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16543677#comment-16543677
]
Dan Burkert edited comment on KUDU-2312 at 7/13/18 8:40 PM:
------------------------------------------------------------
Saw an instance of this in the wild today where EL 6 tservers running 1.6.x
crashed when evaluating a query with 22 predicates. Turns out that the STL on
that platform uses a simple (probably insertion) sort for buffers of elements
<= length 16, but when it switches to the other algorithm (quicksort?) it
doesn't work properly with invalid comparators. Here's the backtrace from a
minidump:
{code:java}
#0 SelectivityRank (predicate=...) at
/usr/src/debug/kudu-1.6.0-cdh5.14.2/src/kudu/common/column_predicate.cc:726
#1 kudu::SelectivityComparator (left=..., right=...) at
/usr/src/debug/kudu-1.6.0-cdh5.14.2/src/kudu/common/column_predicate.cc:731
#2 0x0000000001bb284e in operator()<kudu::ColumnPredicate,
__gnu_cxx::__normal_iterator<kudu::ColumnPredicate*,
std::vector<kudu::ColumnPredicate> > > (__it=..., __val=..., this=<synthetic
pointer>)
at /opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/bits/predefined_ops.h:166
#3
std::__unguarded_linear_insert<__gnu_cxx::__normal_iterator<kudu::ColumnPredicate*,
std::vector<kudu::ColumnPredicate, std::allocator<kudu::ColumnPredicate> > >,
__gnu_cxx::__ops::_Val_comp_iter<int (*)(kudu::ColumnPredicate const&,
kudu::ColumnPredicate const&)> > (__last=..., __last@entry=..., __comp=...) at
/opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/bits/stl_algo.h:1827
#4 0x0000000001baf1a1 in
__unguarded_insertion_sort<__gnu_cxx::__normal_iterator<kudu::ColumnPredicate*,
std::vector<kudu::ColumnPredicate> >, __gnu_cxx::__ops::_Iter_comp_iter<int
(*)(kudu::ColumnPredicate const&, kudu::ColumnPredicate const&)> > (__comp=...,
__last=..., __first=...) at
/opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/bits/stl_algo.h:1866
#5 __final_insertion_sort<__gnu_cxx::__normal_iterator<kudu::ColumnPredicate*,
std::vector<kudu::ColumnPredicate> >, __gnu_cxx::__ops::_Iter_comp_iter<int
(*)(kudu::ColumnPredicate const&, kudu::ColumnPredicate const&)> > (
__comp=..., __last=..., __first=...) at
/opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/bits/stl_algo.h:1885
#6 __sort<__gnu_cxx::__normal_iterator<kudu::ColumnPredicate*,
std::vector<kudu::ColumnPredicate> >, __gnu_cxx::__ops::_Iter_comp_iter<int
(*)(kudu::ColumnPredicate const&, kudu::ColumnPredicate const&)> > (__comp=...,
__last=..., __first=...) at
/opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/bits/stl_algo.h:1970
#7 sort<__gnu_cxx::__normal_iterator<kudu::ColumnPredicate*,
std::vector<kudu::ColumnPredicate> >, int (*)(kudu::ColumnPredicate const&,
kudu::ColumnPredicate const&)> (__comp=<optimized out>, __last=..., __first=...)
at /opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/bits/stl_algo.h:4717
#8 kudu::PredicateEvaluatingIterator::Init (this=0xd74cf5f0, spec=0x79b4400)
at /usr/src/debug/kudu-1.6.0-cdh5.14.2/src/kudu/common/generic_iterators.cc:607
#9 0x0000000001ba9518 in kudu::PredicateEvaluatingIterator::InitAndMaybeWrap
(base_iter=base_iter@entry=0x88c8c00, spec=0x79b4400) at
/usr/src/debug/kudu-1.6.0-cdh5.14.2/src/kudu/common/generic_iterators.cc:587
#10 0x0000000001baa6fd in kudu::UnionIterator::InitSubIterators
(this=this@entry=0x5a9d2c0, spec=0x7e04e40) at
/usr/src/debug/kudu-1.6.0-cdh5.14.2/src/kudu/common/generic_iterators.cc:370
#11 0x0000000001baa8fb in kudu::UnionIterator::Init (this=0x5a9d2c0,
spec=<optimized out>) at
/usr/src/debug/kudu-1.6.0-cdh5.14.2/src/kudu/common/generic_iterators.cc:346
#12 0x00000000009e3422 in kudu::tablet::Tablet::Iterator::Init
(this=0x5875e5a0, spec=0x7e04e40) at
/usr/src/debug/kudu-1.6.0-cdh5.14.2/src/kudu/tablet/tablet.cc:2295
#13 0x00000000008bb7bb in
kudu::tserver::TabletServiceImpl::HandleNewScanRequest
(this=this@entry=0x3d33590, replica=0x13298820, req=req@entry=0x1941bc00,
rpc_context=rpc_context@entry=0x777f2ea0,
result_collector=result_collector@entry=0x7f991fe65a70,
scanner_id=0x7f991fe65a10, snap_timestamp=0x7f991fe659e0,
has_more_results=0x7f991fe65906, error_code=0x7f991fe65908)
at
/usr/src/debug/kudu-1.6.0-cdh5.14.2/src/kudu/tserver/tablet_service.cc:1775
#14 0x00000000008bcdb6 in kudu::tserver::TabletServiceImpl::Scan
(this=0x3d33590, req=0x1941bc00, resp=0x55cb440, context=0x777f2ea0) at
/usr/src/debug/kudu-1.6.0-cdh5.14.2/src/kudu/tserver/tablet_service.cc:1301
#15 0x0000000001b428fd in operator() (__args#2=0x777f2ea0, __args#1=0x55cb440,
__args#0=<optimized out>, this=0x17349938) at
/opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/functional:2439
#16 kudu::rpc::GeneratedServiceIf::Handle (this=<optimized out>,
call=<optimized out>) at
/usr/src/debug/kudu-1.6.0-cdh5.14.2/src/kudu/rpc/service_if.cc:145
#17 0x0000000001b43613 in kudu::rpc::ServicePool::RunThread (this=0xa3f1590) at
/usr/src/debug/kudu-1.6.0-cdh5.14.2/src/kudu/rpc/service_pool.cc:222
#18 0x0000000001c9ef71 in operator() (this=0x1f996b88) at
/usr/src/debug/kudu-1.6.0-cdh5.14.2/thirdparty/installed/uninstrumented/include/boost/function/function_template.hpp:771
#19 kudu::Thread::SuperviseThread (arg=0x1f996b60) at
/usr/src/debug/kudu-1.6.0-cdh5.14.2/src/kudu/util/thread.cc:624
#20 0x00000031ba207aa1 in ?? ()
#21 0x00007f991fe66700 in ?? ()
#22 0x0000000000000000 in ?? (){code}
And the crash message: \{{column_predicate.cc:724] unknown predicate type}},
but note that that's a bit misleading, since it's really complaining about
accessing an invalid ColumnPredicate which is outside the vector buf.
was (Author: danburkert):
Saw an instance of this in the wild today where EL 6 tservers running 1.6.x
crashed when evaluating a query with 22 predicates. Turns out that the STL on
that platform uses a simple (probably insertion) sort for buffers of elements
<= length 16, but when it switches to the other algorithm (quicksort?) it
doesn't work properly with invalid comparators. Here's the backtrace from a
minidump:
{code:java}
#0 SelectivityRank (predicate=...) at
/usr/src/debug/kudu-1.6.0-cdh5.14.2/src/kudu/common/column_predicate.cc:726
#1 kudu::SelectivityComparator (left=..., right=...) at
/usr/src/debug/kudu-1.6.0-cdh5.14.2/src/kudu/common/column_predicate.cc:731
#2 0x0000000001bb284e in operator()<kudu::ColumnPredicate,
__gnu_cxx::__normal_iterator<kudu::ColumnPredicate*,
std::vector<kudu::ColumnPredicate> > > (__it=..., __val=..., this=<synthetic
pointer>)
at /opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/bits/predefined_ops.h:166
#3
std::__unguarded_linear_insert<__gnu_cxx::__normal_iterator<kudu::ColumnPredicate*,
std::vector<kudu::ColumnPredicate, std::allocator<kudu::ColumnPredicate> > >,
__gnu_cxx::__ops::_Val_comp_iter<int (*)(kudu::ColumnPredicate const&,
kudu::ColumnPredicate const&)> > (__last=..., __last@entry=..., __comp=...) at
/opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/bits/stl_algo.h:1827
#4 0x0000000001baf1a1 in
__unguarded_insertion_sort<__gnu_cxx::__normal_iterator<kudu::ColumnPredicate*,
std::vector<kudu::ColumnPredicate> >, __gnu_cxx::__ops::_Iter_comp_iter<int
(*)(kudu::ColumnPredicate const&, kudu::ColumnPredicate const&)> > (__comp=...,
__last=..., __first=...) at
/opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/bits/stl_algo.h:1866
#5 __final_insertion_sort<__gnu_cxx::__normal_iterator<kudu::ColumnPredicate*,
std::vector<kudu::ColumnPredicate> >, __gnu_cxx::__ops::_Iter_comp_iter<int
(*)(kudu::ColumnPredicate const&, kudu::ColumnPredicate const&)> > (
__comp=..., __last=..., __first=...) at
/opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/bits/stl_algo.h:1885
#6 __sort<__gnu_cxx::__normal_iterator<kudu::ColumnPredicate*,
std::vector<kudu::ColumnPredicate> >, __gnu_cxx::__ops::_Iter_comp_iter<int
(*)(kudu::ColumnPredicate const&, kudu::ColumnPredicate const&)> > (__comp=...,
__last=..., __first=...) at
/opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/bits/stl_algo.h:1970
#7 sort<__gnu_cxx::__normal_iterator<kudu::ColumnPredicate*,
std::vector<kudu::ColumnPredicate> >, int (*)(kudu::ColumnPredicate const&,
kudu::ColumnPredicate const&)> (__comp=<optimized out>, __last=..., __first=...)
at /opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/bits/stl_algo.h:4717
#8 kudu::PredicateEvaluatingIterator::Init (this=0xd74cf5f0, spec=0x79b4400)
at /usr/src/debug/kudu-1.6.0-cdh5.14.2/src/kudu/common/generic_iterators.cc:607
#9 0x0000000001ba9518 in kudu::PredicateEvaluatingIterator::InitAndMaybeWrap
(base_iter=base_iter@entry=0x88c8c00, spec=0x79b4400) at
/usr/src/debug/kudu-1.6.0-cdh5.14.2/src/kudu/common/generic_iterators.cc:587
#10 0x0000000001baa6fd in kudu::UnionIterator::InitSubIterators
(this=this@entry=0x5a9d2c0, spec=0x7e04e40) at
/usr/src/debug/kudu-1.6.0-cdh5.14.2/src/kudu/common/generic_iterators.cc:370
#11 0x0000000001baa8fb in kudu::UnionIterator::Init (this=0x5a9d2c0,
spec=<optimized out>) at
/usr/src/debug/kudu-1.6.0-cdh5.14.2/src/kudu/common/generic_iterators.cc:346
#12 0x00000000009e3422 in kudu::tablet::Tablet::Iterator::Init
(this=0x5875e5a0, spec=0x7e04e40) at
/usr/src/debug/kudu-1.6.0-cdh5.14.2/src/kudu/tablet/tablet.cc:2295
#13 0x00000000008bb7bb in
kudu::tserver::TabletServiceImpl::HandleNewScanRequest
(this=this@entry=0x3d33590, replica=0x13298820, req=req@entry=0x1941bc00,
rpc_context=rpc_context@entry=0x777f2ea0,
result_collector=result_collector@entry=0x7f991fe65a70,
scanner_id=0x7f991fe65a10, snap_timestamp=0x7f991fe659e0,
has_more_results=0x7f991fe65906, error_code=0x7f991fe65908)
at
/usr/src/debug/kudu-1.6.0-cdh5.14.2/src/kudu/tserver/tablet_service.cc:1775
#14 0x00000000008bcdb6 in kudu::tserver::TabletServiceImpl::Scan
(this=0x3d33590, req=0x1941bc00, resp=0x55cb440, context=0x777f2ea0) at
/usr/src/debug/kudu-1.6.0-cdh5.14.2/src/kudu/tserver/tablet_service.cc:1301
#15 0x0000000001b428fd in operator() (__args#2=0x777f2ea0, __args#1=0x55cb440,
__args#0=<optimized out>, this=0x17349938) at
/opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/functional:2439
#16 kudu::rpc::GeneratedServiceIf::Handle (this=<optimized out>,
call=<optimized out>) at
/usr/src/debug/kudu-1.6.0-cdh5.14.2/src/kudu/rpc/service_if.cc:145
#17 0x0000000001b43613 in kudu::rpc::ServicePool::RunThread (this=0xa3f1590) at
/usr/src/debug/kudu-1.6.0-cdh5.14.2/src/kudu/rpc/service_pool.cc:222
#18 0x0000000001c9ef71 in operator() (this=0x1f996b88) at
/usr/src/debug/kudu-1.6.0-cdh5.14.2/thirdparty/installed/uninstrumented/include/boost/function/function_template.hpp:771
#19 kudu::Thread::SuperviseThread (arg=0x1f996b60) at
/usr/src/debug/kudu-1.6.0-cdh5.14.2/src/kudu/util/thread.cc:624
#20 0x00000031ba207aa1 in ?? ()
#21 0x00007f991fe66700 in ?? ()
#22 0x0000000000000000 in ?? (){code}
And the crash message: \{column_predicate.cc:724] unknown predicate type}, but
note that that's a bit misleading, since it's really complaining about
accessing an invalid ColumnPredicate which is outside the vector buf.
> Scan predicate application ordering is possibly buggy
> -----------------------------------------------------
>
> Key: KUDU-2312
> URL: https://issues.apache.org/jira/browse/KUDU-2312
> Project: Kudu
> Issue Type: Bug
> Components: tserver
> Affects Versions: 1.6.0
> Reporter: Dan Burkert
> Assignee: Dan Burkert
> Priority: Major
> Fix For: 1.7.0
>
>
> It appears that the {{SelectivityComparator}} that is supposed to ensure that
> predicates are applied in selectivity order does not follow the interface for
> STL comparators. Additionally, we aren't properly using tie-breakers to
> ensure a stable evaluation order.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
