[jira] [Commented] (IMPALA-6764) Codegend UnionNode::MaterializeBatch() causes memory corruption crash of Impalad
[
https://issues.apache.org/jira/browse/IMPALA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604907#comment-16604907
]
Zoram Thanga commented on IMPALA-6764:
--
IMPALA-6059 changed the implementation of StringFunctions::Trim() enough that
this is not reproducible anymore.
Marking this as fixed by the above-mentioned JIRA.
> Codegend UnionNode::MaterializeBatch() causes memory corruption crash of
> Impalad
>
>
> Key: IMPALA-6764
> URL: https://issues.apache.org/jira/browse/IMPALA-6764
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.11.0
>Reporter: Zoram Thanga
>Assignee: Zoram Thanga
>Priority: Critical
> Attachments: bad-materializebatch-disasm.txt,
> good-materializebatch-disasm.txt
>
>
> A CTAS statement involving UNION ALL with LEFT JOIN children is reliably
> crashing with a stack trace similar to the following:
> {noformat}
> (gdb) bt
> #0 0x7fb85fdf11f7 in raise () from ./debug-stuff/lib64/libc.so.6
> #1 0x7fb85fdf28e8 in abort () from ./debug-stuff/lib64/libc.so.6
> #2 0x7fb862106f35 in os::abort(bool) () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #3 0x7fb8622aaf33 in VMError::report_and_die() () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #4 0x7fb86210d22f in JVM_handle_linux_signal () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #5 0x7fb862103253 in signalHandler(int, siginfo*, void*) () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #6
> #7 0x7fb85ff08706 in __memcpy_ssse3_back () from
> ./debug-stuff/lib64/libc.so.6
> #8 0x7fb840700d73 in
> impala::UnionNode::MaterializeBatch(impala::RowBatch*, unsigned char**)
> [clone .588] ()
> #9 0x01001806 in impala::UnionNode::GetNextMaterialized
> (this=this@entry=0x828, state=state@entry=0x848ed00,
> row_batch=row_batch@entry=0xcef9950)
> at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/union-node.cc:228
> #10 0x01001b5c in impala::UnionNode::GetNext (this=0x828,
> state=0x848ed00, row_batch=0xcef9950, eos=0x7fb7fe9a987e)
> at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/union-node.cc:294
> #11 0x00b724d2 in impala::FragmentInstanceState::ExecInternal
> (this=this@entry=0x4c030c0)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/fragment-instance-state.cc:270
> #12 0x00b74e42 in impala::FragmentInstanceState::Exec
> (this=this@entry=0x4c030c0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/fragment-instance-state.cc:89
> #13 0x00b64488 in impala::QueryState::ExecFInstance (this=0x8559200,
> fis=0x4c030c0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/query-state.cc:382
> #14 0x00d13613 in boost::function0::operator()
> (this=0x7fb7fe9a9c60)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
> #15 impala::Thread::SuperviseThread(std::string const&, std::string const&,
> boost::function, impala::Promise*) (name=..., category=...,
> functor=...,
> thread_started=0x7fb7f999f0f0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/util/thread.cc:352
> #16 0x00d13d54 in
> boost::_bi::list4 std::char_traits, std::allocator > >,
> boost::_bi::value,
> std::allocator > >, boost::_bi::value >,
> boost::_bi::value*> >::operator() std::basic_string&, const std::basic_string&,
> boost::function, impala::Promise*), boost::_bi::list0> (
> f=@0x808bfb8: 0xd13460 const&, std::string const&, boost::function,
> impala::Promise*)>, a=,
> this=0x808bfc0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:457
> #17 boost::_bi::bind_t boost::function, impala::Promise*),
> boost::_bi::list4,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value*> > >::operator()() (this=0x808bfb8)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
> #18 boost::detail::thread_data const&, std::string const&, boost::function,
> impala::Promise*), boost::_bi::list4,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value*> > > >::run() (this=0x808be00)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:116
> #19 0x0128e8ea in thread_proxy ()
> #20 0x7fb860186e25 in start_thread () from
> ./debug-stuff/lib64/libpthread.so.0
> #21 0x7fb85feb434d in clone () from ./debug-stuff/lib64/libc.so.6
> {noformat}
> The exact location or reason of the crash varies, i.e., sometimes we crash
[jira] [Commented] (IMPALA-6764) Codegend UnionNode::MaterializeBatch() causes memory corruption crash of Impalad
[
https://issues.apache.org/jira/browse/IMPALA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470966#comment-16470966
]
Zoram Thanga commented on IMPALA-6764:
--
[~dhecht]:
>> Do we have an example of the bad IR to x86 generation?
Attaching the assembly code of UnionNode::MaterializeBatch() for both cases.
The only change from 'bad' to 'good' is this change to StringFunctions::Trim():
{code:java}
$ git diff
diff --git a/be/src/exprs/string-functions-ir.cc
b/be/src/exprs/string-functions-ir.cc
index 49bc8c1..88af5d8 100644
--- a/be/src/exprs/string-functions-ir.cc
+++ b/be/src/exprs/string-functions-ir.cc
@@ -409,7 +409,7 @@ StringVal StringFunctions::Trim(FunctionContext* context,
const StringVal& str)
}
// Find new ending position.
int32_t end = str.len - 1;
- while (end > begin && str.ptr[end] == ' ') {
+ while (end >= begin && str.ptr[end] == ' ') {
--end;
}
return StringVal(str.ptr + begin, end - begin + 1);
{code}
>> Is there a later LLVM 3.9.x release that has the fix?
I am not sure when exactly the fix came in. FYI, there's no 3.9.x where x > 1.
LLVM changed their release taxonomy after 3.9.1. See
[http://releases.llvm.org/|http://releases.llvm.org/]. My 'wild guess' is that
the issue may have been fixed in 5.0.0.
> Codegend UnionNode::MaterializeBatch() causes memory corruption crash of
> Impalad
>
>
> Key: IMPALA-6764
> URL: https://issues.apache.org/jira/browse/IMPALA-6764
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.11.0
>Reporter: Zoram Thanga
>Assignee: Zoram Thanga
>Priority: Critical
>
> A CTAS statement involving UNION ALL with LEFT JOIN children is reliably
> crashing with a stack trace similar to the following:
> {noformat}
> (gdb) bt
> #0 0x7fb85fdf11f7 in raise () from ./debug-stuff/lib64/libc.so.6
> #1 0x7fb85fdf28e8 in abort () from ./debug-stuff/lib64/libc.so.6
> #2 0x7fb862106f35 in os::abort(bool) () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #3 0x7fb8622aaf33 in VMError::report_and_die() () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #4 0x7fb86210d22f in JVM_handle_linux_signal () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #5 0x7fb862103253 in signalHandler(int, siginfo*, void*) () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #6
> #7 0x7fb85ff08706 in __memcpy_ssse3_back () from
> ./debug-stuff/lib64/libc.so.6
> #8 0x7fb840700d73 in
> impala::UnionNode::MaterializeBatch(impala::RowBatch*, unsigned char**)
> [clone .588] ()
> #9 0x01001806 in impala::UnionNode::GetNextMaterialized
> (this=this@entry=0x828, state=state@entry=0x848ed00,
> row_batch=row_batch@entry=0xcef9950)
> at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/union-node.cc:228
> #10 0x01001b5c in impala::UnionNode::GetNext (this=0x828,
> state=0x848ed00, row_batch=0xcef9950, eos=0x7fb7fe9a987e)
> at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/union-node.cc:294
> #11 0x00b724d2 in impala::FragmentInstanceState::ExecInternal
> (this=this@entry=0x4c030c0)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/fragment-instance-state.cc:270
> #12 0x00b74e42 in impala::FragmentInstanceState::Exec
> (this=this@entry=0x4c030c0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/fragment-instance-state.cc:89
> #13 0x00b64488 in impala::QueryState::ExecFInstance (this=0x8559200,
> fis=0x4c030c0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/query-state.cc:382
> #14 0x00d13613 in boost::function0::operator()
> (this=0x7fb7fe9a9c60)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
> #15 impala::Thread::SuperviseThread(std::string const&, std::string const&,
> boost::function, impala::Promise*) (name=..., category=...,
> functor=...,
> thread_started=0x7fb7f999f0f0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/util/thread.cc:352
> #16 0x00d13d54 in
> boost::_bi::list4 std::char_traits, std::allocator > >,
> boost::_bi::value std::allocator > >, boost::_bi::value >,
> boost::_bi::value >::operator() std::basic_string&, const std::basic_string&,
> boost::function, impala::Promise*), boost::_bi::list0> (
> f=@0x808bfb8: 0xd13460 const&, std::string const&, boost::function,
> impala::Promise*)>, a=,
> this=0x808bfc0)
