[
https://issues.apache.org/jira/browse/IMPALA-9957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17158132#comment-17158132
]
Quanlong Huang commented on IMPALA-9957:
----------------------------------------
The crash in DEBUG build reveals the error.
[https://github.com/apache/impala/blob/f602c3f80f5f61ccaebdf1493ff7c89230b77410/be/src/exec/grouping-aggregator-partition.cc#L125]
{code:java}
100 Status GroupingAggregator::Partition::SerializeStreamForSpilling() {
101 DCHECK(!parent->is_streaming_preagg_);
102 if (parent->needs_serialize_) {
...
115
116 // Serialize and copy the spilled partition's stream into the new
stream.
117 Status status;
118 BufferedTupleStream* new_stream = parent->serialize_stream_.get();
119 HashTable::Iterator it = hash_tbl->Begin(parent->ht_ctx_.get());
120 while (!it.AtEnd()) {
121 Tuple* tuple = it.GetTuple();
122 it.Next();
123 AggFnEvaluator::Serialize(agg_fn_evals, tuple);
124 if (UNLIKELY(!new_stream->AddRow(reinterpret_cast<TupleRow*>(&tuple),
&status))) {
125 DCHECK(!status.ok()) << "Stream was unpinned - AddRow() only fails
on error"; <<---- Failed here
126 // Even if we can't add to new_stream, finish up processing this
agg stream to
127 // make clean up easier (someone has to finalize this stream and we
don't want to
128 // remember where we are).
129 parent->CleanupHashTbl(agg_fn_evals, it);
130 hash_tbl->Close();
131 hash_tbl.reset();
132 aggregated_row_stream->Close(nullptr,
RowBatch::FlushMode::NO_FLUSH_RESOURCES);
133 return status;
134 }
135 }
{code}
new_stream->AddRow returns false with an OK status. So we close the hash_tbl in
this method and return to the caller at
https://github.com/apache/impala/blob/f602c3f80f5f61ccaebdf1493ff7c89230b77410/be/src/exec/grouping-aggregator-partition.cc#L180
{code}
163 Status GroupingAggregator::Partition::Spill(bool more_aggregate_rows) {
...
169 RETURN_IF_ERROR(SerializeStreamForSpilling());
170
171 // Free the in-memory result data.
172 AggFnEvaluator::Close(agg_fn_evals, parent->state_);
173 agg_fn_evals.clear();
174
175 if (agg_fn_perm_pool.get() != nullptr) {
176 agg_fn_perm_pool->FreeAll();
177 agg_fn_perm_pool.reset();
178 }
179
180 hash_tbl->Close(); <<---- RELEASE build crash in this
181 hash_tbl.reset();
182
{code}
At line180, we close the hash_tbl again, which causes the crash in RELEASE
build.
> Impalad crashes when serializing large rows in aggregation spilling
> -------------------------------------------------------------------
>
> Key: IMPALA-9957
> URL: https://issues.apache.org/jira/browse/IMPALA-9957
> Project: IMPALA
> Issue Type: Bug
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
>
> Queries to reproduce the crash using the testdata:
> {code:sql}
> create table bigstrs stored as parquet as
> select *, repeat(uuid(), cast(random() * 100000 as int)) as bigstr
> from functional.alltypes;
> set MAX_ROW_SIZE=3.5MB;
> set MEM_LIMIT=4GB;
> create table my_str_group stored as parquet as
> select group_concat(string_col) as ss, bigstr
> from bigstrs group by bigstr;
> {code}
> The crash stacktraces are different in different build modes. Crash
> stacktrace in RELEASE build with codegen enabled:
> {code}
> Thread 316 (crashed)
> 0 impalad!impala::HashTable::Close() [hash-table.cc : 512 + 0x0]
> 1 impalad!impala::GroupingAggregator::Partition::Spill(bool)
> [grouping-aggregator-partition.cc : 180 + 0x9]
> 2 impalad!impala::GroupingAggregator::SpillPartition(bool)
> [grouping-aggregator.cc : 904 + 0x10]
> 3 0x7f5fba83db3c
> 4 impalad!impala::GroupingAggregator::AddBatch(impala::RuntimeState*,
> impala::RowBatch*) [grouping-aggregator.cc : 437 + 0x2]
> 5 impalad!impala::AggregationNode::Open(impala::RuntimeState*)
> [aggregation-node.cc : 70 + 0x6]
> 6 libstdc++.so.6.0.24 + 0x120b28
> 7
> impalad!apache::hive::service::cli::thrift::TColumnValue::printTo(std::ostream&)
> const [converter_lexical_streams.hpp : 161 + 0x8]
> 8 impalad!impala::FragmentInstanceState::Open() [fragment-instance-state.cc
> : 396 + 0x11]
> 9 impalad!tc_newarray + 0x171
> {code}
> Crash stacktrace in RELEASE build with codegen disabled (set
> DISABLE_CODEGEN=true):
> {code:java}
> Thread 320 (crashed)
> 0 impalad!impala::HashTable::Close() [hash-table.cc : 512 + 0x0]
> 1 impalad!impala::GroupingAggregator::Partition::Spill(bool)
> [grouping-aggregator-partition.cc : 180 + 0x9]
> 2 impalad!impala::GroupingAggregator::SpillPartition(bool)
> [grouping-aggregator.cc : 904 + 0x10]
> 3 impalad!impala::Status
> impala::GroupingAggregator::AddBatchImpl<false>(impala::RowBatch*,
> impala::TPrefetchMode::type, impala::HashTableCtx*)
> [grouping-aggregator-ir.cc : 148 + 0x11]
> 4 impalad!impala::GroupingAggregator::AddBatch(impala::RuntimeState*,
> impala::RowBatch*) [grouping-aggregator.cc : 439 + 0x5]
> 5 impalad!impala::AggregationNode::Open(impala::RuntimeState*)
> [aggregation-node.cc : 70 + 0x6]
> 6 impalad!impala::FragmentInstanceState::Open() [fragment-instance-state.cc
> : 396 + 0x11]
> 7 impalad!impala::FragmentInstanceState::Exec() [fragment-instance-state.cc
> : 97 + 0x12]
> 8 impalad!impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
> [query-state.cc : 815 + 0x19]
> 9 impalad!impala::Thread::SuperviseThread(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*) [function_template.hpp : 770
> + 0x7]
> 10 impalad!boost::detail::thread_data<boost::_bi::bind_t<void, void
> (*)(std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&, boost::function<void
> ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
> (impala::PromiseMode)0>*),
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >
> >::run() [bind.hpp : 531 + 0xc]
> 11 impalad!thread_proxy + 0x72
> 12 libpthread-2.23.so + 0x76ba
> 13 libc-2.23.so + 0x1074dd
> {code}
> Crash stacktrace in DEBUG build with codegen disabled is a bit ealier -
> crashed at a DCHECK:
> {code}
> F0715 20:29:24.389505 16868 grouping-aggregator-partition.cc:125]
> 1d4b40df02e6ad76:433ed57400000003] Check failed: !status.ok() Stream was
> unpinned - AddRow() only fails on error
> *** Check failure stack trace: ***
> @ 0x513f31c google::LogMessage::Fail()
> @ 0x5140c0c google::LogMessage::SendToLog()
> @ 0x513ec7a google::LogMessage::Flush()
> @ 0x5142878 google::LogMessageFatal::~LogMessageFatal()
> @ 0x28b2ca7
> impala::GroupingAggregator::Partition::SerializeStreamForSpilling()
> @ 0x28b360f impala::GroupingAggregator::Partition::Spill()
> @ 0x28a4122 impala::GroupingAggregator::SpillPartition()
> @ 0x28b169c impala::GroupingAggregator::AddIntermediateTuple<>()
> @ 0x28b09d9 impala::GroupingAggregator::ProcessRow<>()
> @ 0x28af535 impala::GroupingAggregator::AddBatchImpl<>()
> @ 0x289f6ad impala::GroupingAggregator::AddBatch()
> @ 0x28db463 impala::AggregationNode::Open()
> @ 0x22598bd impala::FragmentInstanceState::Open()
> @ 0x22562f4 impala::FragmentInstanceState::Exec()
> @ 0x22801ed impala::QueryState::ExecFInstance()
> @ 0x227e5ef
> _ZZN6impala10QueryState15StartFInstancesEvENKUlvE_clEv
> @ 0x2281d8e
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala10QueryState15StartFInstancesEvEUlvE_vE6invokeERNS1_15function_bufferE
> @ 0x204d7d1 boost::function0<>::operator()()
> @ 0x26702d5 impala::Thread::SuperviseThread()
> @ 0x2678272 boost::_bi::list5<>::operator()<>()
> @ 0x2678196 boost::_bi::bind_t<>::operator()()
> @ 0x2678157 boost::detail::thread_data<>::run()
> @ 0x3e45d71 thread_proxy
> @ 0x7ff1bfdc46b9 start_thread
> @ 0x7ff1bc98f4dc clone
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
