[
https://issues.apache.org/jira/browse/IMPALA-9957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang resolved IMPALA-9957.
------------------------------------
Fix Version/s: Impala 4.0
Resolution: Fixed
> Impalad crashes when serializing large rows in aggregation spilling
> -------------------------------------------------------------------
>
> Key: IMPALA-9957
> URL: https://issues.apache.org/jira/browse/IMPALA-9957
> Project: IMPALA
> Issue Type: Bug
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
> Fix For: Impala 4.0
>
>
> Queries to reproduce the crash using the testdata:
> {code:sql}
> create table bigstrs stored as parquet as
> select *, repeat(uuid(), cast(random() * 100000 as int)) as bigstr
> from functional.alltypes;
> set MAX_ROW_SIZE=3.5MB;
> set MEM_LIMIT=4GB;
> create table my_str_group stored as parquet as
> select group_concat(string_col) as ss, bigstr
> from bigstrs group by bigstr;
> {code}
> The last query 1) has large rows, 2) needs spilling in aggregation 3) has
> aggregation on functions needs serialize (e.g. group_concat, appx_median,
> min(string), etc). With these 3 conditions, it will trigger this bug.
> The crash stacktraces are different in different build modes. Crash
> stacktrace in RELEASE build with codegen enabled:
> {code:java}
> Thread 316 (crashed)
> 0 impalad!impala::HashTable::Close() [hash-table.cc : 512 + 0x0]
> 1 impalad!impala::GroupingAggregator::Partition::Spill(bool)
> [grouping-aggregator-partition.cc : 180 + 0x9]
> 2 impalad!impala::GroupingAggregator::SpillPartition(bool)
> [grouping-aggregator.cc : 904 + 0x10]
> 3 0x7f5fba83db3c
> 4 impalad!impala::GroupingAggregator::AddBatch(impala::RuntimeState*,
> impala::RowBatch*) [grouping-aggregator.cc : 437 + 0x2]
> 5 impalad!impala::AggregationNode::Open(impala::RuntimeState*)
> [aggregation-node.cc : 70 + 0x6]
> 6 libstdc++.so.6.0.24 + 0x120b28
> 7
> impalad!apache::hive::service::cli::thrift::TColumnValue::printTo(std::ostream&)
> const [converter_lexical_streams.hpp : 161 + 0x8]
> 8 impalad!impala::FragmentInstanceState::Open() [fragment-instance-state.cc
> : 396 + 0x11]
> 9 impalad!tc_newarray + 0x171
> {code}
> Crash stacktrace in RELEASE build with codegen disabled (set
> DISABLE_CODEGEN=true):
> {code:java}
> Thread 320 (crashed)
> 0 impalad!impala::HashTable::Close() [hash-table.cc : 512 + 0x0]
> 1 impalad!impala::GroupingAggregator::Partition::Spill(bool)
> [grouping-aggregator-partition.cc : 180 + 0x9]
> 2 impalad!impala::GroupingAggregator::SpillPartition(bool)
> [grouping-aggregator.cc : 904 + 0x10]
> 3 impalad!impala::Status
> impala::GroupingAggregator::AddBatchImpl<false>(impala::RowBatch*,
> impala::TPrefetchMode::type, impala::HashTableCtx*)
> [grouping-aggregator-ir.cc : 148 + 0x11]
> 4 impalad!impala::GroupingAggregator::AddBatch(impala::RuntimeState*,
> impala::RowBatch*) [grouping-aggregator.cc : 439 + 0x5]
> 5 impalad!impala::AggregationNode::Open(impala::RuntimeState*)
> [aggregation-node.cc : 70 + 0x6]
> 6 impalad!impala::FragmentInstanceState::Open() [fragment-instance-state.cc
> : 396 + 0x11]
> 7 impalad!impala::FragmentInstanceState::Exec() [fragment-instance-state.cc
> : 97 + 0x12]
> 8 impalad!impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
> [query-state.cc : 815 + 0x19]
> 9 impalad!impala::Thread::SuperviseThread(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*) [function_template.hpp : 770
> + 0x7]
> 10 impalad!boost::detail::thread_data<boost::_bi::bind_t<void, void
> (*)(std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&, boost::function<void
> ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
> (impala::PromiseMode)0>*),
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >
> >::run() [bind.hpp : 531 + 0xc]
> 11 impalad!thread_proxy + 0x72
> 12 libpthread-2.23.so + 0x76ba
> 13 libc-2.23.so + 0x1074dd
> {code}
> Crash stacktrace in DEBUG build with codegen disabled is a bit ealier -
> crashed at a DCHECK:
> {code:java}
> F0715 20:29:24.389505 16868 grouping-aggregator-partition.cc:125]
> 1d4b40df02e6ad76:433ed57400000003] Check failed: !status.ok() Stream was
> unpinned - AddRow() only fails on error
> *** Check failure stack trace: ***
> @ 0x513f31c google::LogMessage::Fail()
> @ 0x5140c0c google::LogMessage::SendToLog()
> @ 0x513ec7a google::LogMessage::Flush()
> @ 0x5142878 google::LogMessageFatal::~LogMessageFatal()
> @ 0x28b2ca7
> impala::GroupingAggregator::Partition::SerializeStreamForSpilling()
> @ 0x28b360f impala::GroupingAggregator::Partition::Spill()
> @ 0x28a4122 impala::GroupingAggregator::SpillPartition()
> @ 0x28b169c impala::GroupingAggregator::AddIntermediateTuple<>()
> @ 0x28b09d9 impala::GroupingAggregator::ProcessRow<>()
> @ 0x28af535 impala::GroupingAggregator::AddBatchImpl<>()
> @ 0x289f6ad impala::GroupingAggregator::AddBatch()
> @ 0x28db463 impala::AggregationNode::Open()
> @ 0x22598bd impala::FragmentInstanceState::Open()
> @ 0x22562f4 impala::FragmentInstanceState::Exec()
> @ 0x22801ed impala::QueryState::ExecFInstance()
> @ 0x227e5ef
> _ZZN6impala10QueryState15StartFInstancesEvENKUlvE_clEv
> @ 0x2281d8e
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala10QueryState15StartFInstancesEvEUlvE_vE6invokeERNS1_15function_bufferE
> @ 0x204d7d1 boost::function0<>::operator()()
> @ 0x26702d5 impala::Thread::SuperviseThread()
> @ 0x2678272 boost::_bi::list5<>::operator()<>()
> @ 0x2678196 boost::_bi::bind_t<>::operator()()
> @ 0x2678157 boost::detail::thread_data<>::run()
> @ 0x3e45d71 thread_proxy
> @ 0x7ff1bfdc46b9 start_thread
> @ 0x7ff1bc98f4dc clone
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
