Joe McDonnell created IMPALA-7704:
-------------------------------------
Summary: ASAN tests failing in HdfsParquetTableWriter
Key: IMPALA-7704
URL: https://issues.apache.org/jira/browse/IMPALA-7704
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 3.1.0
Reporter: Joe McDonnell
ASAN tests have been failing for the last few runs. Here is the output:
{noformat}
==117268==ERROR: AddressSanitizer: use-after-poison on address 0x7ef9312e0ec0
at pc 0x0000016a0c98 bp 0x7ef91e502540 sp 0x7ef91e501cf0
READ of size 32681 at 0x7ef9312e0ec0 thread T82364
#0 0x16a0c97 in __interceptor_memcpy
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:738
#1 0x7f02e43c2dca in jni_SetByteArrayRegion
(/usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so+0x6d3dca)
#2 0x4ae29b7 in hdfsWrite
/container.redhat6/build/cdh/hadoop/3.0.0-cdh6.x-SNAPSHOT/rpm/BUILD/hadoop-3.0.0-cdh6.x-SNAPSHOT/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c:1626
#3 0x332ca67 in impala::HdfsTableWriter::Write(unsigned char const*, int)
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-writer.cc:46:13
#4 0x25ac037 in impala::HdfsParquetTableWriter::BaseColumnWriter::Flush(long*,
long*, long*)
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:736:5
#5 0x25b0e57 in impala::HdfsParquetTableWriter::FlushCurrentRowGroup()
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:1195:5
#6 0x25b4147 in impala::HdfsParquetTableWriter::Finalize()
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:1161:3
#7 0x251e424 in
impala::HdfsTableSink::FinalizePartitionFile(impala::RuntimeState*,
impala::OutputPartition*)
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-sink.cc:620:5
#8 0x2523d1b in impala::HdfsTableSink::FlushFinal(impala::RuntimeState*)
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-sink.cc:660:5
#9 0x1fc09f0 in impala::FragmentInstanceState::ExecInternal()
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:324:3
#10 0x1fbbd5c in impala::FragmentInstanceState::Exec()
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:95:14
#11 0x1fd5f94 in
impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/query-state.cc:478:24
#12 0x1cdef96 in boost::function0<void>::operator()() const
/data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
#13 0x23ac27e in impala::Thread::SuperviseThread(std::string const&,
std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
impala::Promise<long, (impala::PromiseMode)0>*)
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/thread.cc:359:3
#14 0x23b7708 in void boost::_bi::list5<boost::_bi::value<std::string>,
boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >,
boost::_bi::value<impala::ThreadDebugInfo*>,
boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*>
>::operator()<void (*)(std::string const&, std::string const&,
boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
(impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void
(*&)(std::string const&, std::string const&, boost::function<void ()>,
impala::ThreadDebugInfo const*, impala::Promise<long,
(impala::PromiseMode)0>*), boost::_bi::list0&, int)
/data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525:9
#15 0x23b755b in boost::_bi::bind_t<void, void (*)(std::string const&,
std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
impala::Promise<long, (impala::PromiseMode)0>*),
boost::_bi::list5<boost::_bi::value<std::string>,
boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >,
boost::_bi::value<impala::ThreadDebugInfo*>,
boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >
>::operator()()
/data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20:27
#16 0x39a11e9 in thread_proxy
(/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x39a11e9)
#17 0x7f02e19f4e24 in start_thread (/lib64/libpthread.so.0+0x7e24)
#18 0x7f02e150b34c in __clone (/lib64/libc.so.6+0xf834c)
0x7ef9312e0ec0 is located 161472 bytes inside of 524288-byte region
[0x7ef9312b9800,0x7ef931339800)
allocated by thread T82364 here:
#0 0x16fcd88 in malloc
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:67
#1 0x1f2a5ec in impala::MemPool::FindChunk(long, bool)
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/mem-pool.cc:140:45
#2 0x1f44c37 in unsigned char* impala::MemPool::Allocate<false>(long, int)
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/mem-pool.h:284:9
#3 0x1f44c37 in impala::MemPool::Allocate(long)
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/mem-pool.h:107
#4 0x25ad32c in
impala::HdfsParquetTableWriter::BaseColumnWriter::FinalizeCurrentPage()
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:797:61
#5 0x25bbdea in
impala::HdfsParquetTableWriter::BaseColumnWriter::AppendRow(impala::TupleRow*)
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:590:5
#6 0x25b36cc in impala::HdfsParquetTableWriter::AppendRows(impala::RowBatch*,
std::vector<int, std::allocator<int> > const&, bool*)
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:1124:7
#7 0x251ddec in
impala::HdfsTableSink::WriteRowsToPartition(impala::RuntimeState*,
impala::RowBatch*, std::pair<std::unique_ptr<impala::OutputPartition,
std::default_delete<impala::OutputPartition> >, std::vector<int,
std::allocator<int> > >*)
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-sink.cc:256:5
#8 0x2522f01 in impala::HdfsTableSink::Send(impala::RuntimeState*,
impala::RowBatch*)
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-sink.cc:590:5
#9 0x1fc07ad in impala::FragmentInstanceState::ExecInternal()
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:316:5
#10 0x1fbbd5c in impala::FragmentInstanceState::Exec()
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:95:14
#11 0x1fd5f94 in
impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/query-state.cc:478:24
#12 0x1cdef96 in boost::function0<void>::operator()() const
/data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
#13 0x23ac27e in impala::Thread::SuperviseThread(std::string const&,
std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
impala::Promise<long, (impala::PromiseMode)0>*)
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/thread.cc:359:3
#14 0x23b7708 in void boost::_bi::list5<boost::_bi::value<std::string>,
boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >,
boost::_bi::value<impala::ThreadDebugInfo*>,
boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*>
>::operator()<void (*)(std::string const&, std::string const&,
boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
(impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void
(*&)(std::string const&, std::string const&, boost::function<void ()>,
impala::ThreadDebugInfo const*, impala::Promise<long,
(impala::PromiseMode)0>*), boost::_bi::list0&, int)
/data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525:9
#15 0x23b755b in boost::_bi::bind_t<void, void (*)(std::string const&,
std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
impala::Promise<long, (impala::PromiseMode)0>*),
boost::_bi::list5<boost::_bi::value<std::string>,
boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >,
boost::_bi::value<impala::ThreadDebugInfo*>,
boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >
>::operator()()
/data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20:27
#16 0x39a11e9 in thread_proxy
(/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x39a11e9)
Thread T82364 created by T82363 here:
#0 0x164420d in __interceptor_pthread_create
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
#1 0x39a05c9 in boost::thread::start_thread_noexcept()
(/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x39a05c9)
#2 0x45e0360d (<unknown module>)
Thread T82363 created by T207 here:
#0 0x164420d in __interceptor_pthread_create
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
#1 0x39a05c9 in boost::thread::start_thread_noexcept()
(/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x39a05c9)
#2 0x45e0360d (<unknown module>)
Thread T207 created by T199 here:
#0 0x164420d in __interceptor_pthread_create
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
#1 0x39a05c9 in boost::thread::start_thread_noexcept()
(/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x39a05c9)
#2 0x45e0360d (<unknown module>)
Thread T199 created by T198 here:
#0 0x164420d in __interceptor_pthread_create
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
#1 0x39a05c9 in boost::thread::start_thread_noexcept()
(/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x39a05c9)
#2 0x45e0360d (<unknown module>)
Thread T198 created by T0 here:
#0 0x164420d in __interceptor_pthread_create
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
#1 0x39a05c9 in boost::thread::start_thread_noexcept()
(/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x39a05c9)
#2 0x45e0360d (<unknown module>)
SUMMARY: AddressSanitizer: use-after-poison
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:738
in __interceptor_memcpy
Shadow bytes around the buggy address:
0x0fdfa6254180: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
0x0fdfa6254190: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
0x0fdfa62541a0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
0x0fdfa62541b0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
0x0fdfa62541c0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
=>0x0fdfa62541d0: f7 f7 f7 f7 f7 f7 f7 f7[f7]f7 f7 f7 f7 f7 f7 f7
0x0fdfa62541e0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
0x0fdfa62541f0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
0x0fdfa6254200: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
0x0fdfa6254210: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
0x0fdfa6254220: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==117268==ABORTING{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
