[
https://issues.apache.org/jira/browse/IMPALA-7113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16498890#comment-16498890
]
Pranay Singh edited comment on IMPALA-7113 at 6/2/18 5:22 AM:
--------------------------------------------------------------
Certainly, this problem seems to be related to fuzz test, the above problem can
be explained if the offset is a large value.
The fuzz test can do that , do we have a minidump or a core here, the large
value of offset will tell that.
Though I added the bound check,
// Buffer access out of bounds.
if (offset > size) return -1 >>>> looks like size should have a smaller value
Code
--------------
int8_t firstbyte = (int8_t) buf[0 + offset]; >>>>> offset is large value.
Stack
--------------------
==31616==ERROR: AddressSanitizer: heap-buffer-overflow on address
0x619002c94827 at pc 0x000002293cf2 bp 0x7f653d570eb0 sp 0x7f653d570ea8
READ of size 1 at 0x619002c94827 thread T125815
#0 0x2293cf1 in impala::ReadWriteUtil::GetVLong(unsigned char*, long,
long*, int)
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/read-write-util.h:200:31
Test log
------------
04:44:53
query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_alltypes[exec_option:
{'debug_action': None, 'abort_on_error': False, 'mem_limit': '512m',
'num_nodes': 0} | table_format: rc/snap/block]
-Pranay
was (Author: pranay_singh):
Certainly, this problem seems to be related to fuzz test, the above problem can
be explained if the offset is a large value.
The fuzz test can do that , do we have a minidump or a core here, the large
value of offset will tell that.
Though I added the bound check,
// Buffer access out of bounds.
if (offset > size) return -1 >>>> looks like size would have a smaller value
Code
--------------
int8_t firstbyte = (int8_t) buf[0 + offset]; >>>>> offset is large value.
Stack
--------------------
==31616==ERROR: AddressSanitizer: heap-buffer-overflow on address
0x619002c94827 at pc 0x000002293cf2 bp 0x7f653d570eb0 sp 0x7f653d570ea8
READ of size 1 at 0x619002c94827 thread T125815
#0 0x2293cf1 in impala::ReadWriteUtil::GetVLong(unsigned char*, long,
long*, int)
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/read-write-util.h:200:31
Test log
------------
04:44:53
query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_alltypes[exec_option:
{'debug_action': None, 'abort_on_error': False, 'mem_limit': '512m',
'num_nodes': 0} | table_format: rc/snap/block]
-Pranay
> ASAN heap-buffer-overflow in impala::HdfsRCFileScanner::GetCurrentKeyBuffer()
> -----------------------------------------------------------------------------
>
> Key: IMPALA-7113
> URL: https://issues.apache.org/jira/browse/IMPALA-7113
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.13.0, Impala 3.1.0
> Reporter: Lars Volker
> Assignee: Rahul Shivu Mahadev
> Priority: Blocker
> Labels: asan, broken-build
>
> [~pranay_singh] - I'm assigning this to you since you changed this code last
> in IMPALA-3833.
> {noformat}
> ==31616==ERROR: AddressSanitizer: heap-buffer-overflow on address
> 0x619002c94827 at pc 0x000002293cf2 bp 0x7f653d570eb0 sp 0x7f653d570ea8
> READ of size 1 at 0x619002c94827 thread T125815
> #0 0x2293cf1 in impala::ReadWriteUtil::GetVLong(unsigned char*, long,
> long*, int)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/read-write-util.h:200:31
> #1 0x2292114 in impala::ReadWriteUtil::GetVInt(unsigned char*, int*, int)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/read-write-util.h:184:13
> #2 0x228e5c6 in impala::HdfsRCFileScanner::GetCurrentKeyBuffer(int, bool,
> unsigned char**, int)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-rcfile-scanner.cc:379:20
> #3 0x228ce07 in impala::HdfsRCFileScanner::ReadKeyBuffers()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-rcfile-scanner.cc:354:41
> #4 0x228b8a0 in impala::HdfsRCFileScanner::StartRowGroup()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-rcfile-scanner.cc:259:41
> #5 0x228f006 in
> impala::HdfsRCFileScanner::ProcessRange(impala::RowBatch*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-rcfile-scanner.cc:531:41
> #6 0x3039cef in
> impala::BaseSequenceScanner::GetNextInternal(impala::RowBatch*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/base-sequence-scanner.cc:181:19
> #7 0x225c891 in impala::HdfsScanner::ProcessSplit()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-scanner.cc:134:21
> #8 0x221ad33 in
> impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext,
> std::allocator<impala::FilterContext> > const&, impala::MemPool*,
> impala::io::ScanRange*, long*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-scan-node.cc:453:21
> #9 0x2219e50 in impala::HdfsScanNode::ScannerThread(bool, long)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-scan-node.cc:360:16
> #10 0x1c4ffb6 in boost::function0<void>::operator()() const
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
> #11 0x211216e in impala::Thread::SuperviseThread(std::string const&,
> std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long>*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/thread.cc:356:3
> #12 0x211d3f8 in void boost::_bi::list5<boost::_bi::value<std::string>,
> boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string
> const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo
> const*, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>,
> void (*&)(std::string const&, std::string const&, boost::function<void ()>,
> impala::ThreadDebugInfo const*, impala::Promise<long>*), boost::_bi::list0&,
> int)
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525:9
> #13 0x211d24b in boost::_bi::bind_t<void, void (*)(std::string const&,
> std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long>*), boost::_bi::list5<boost::_bi::value<std::string>,
> boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long>*> > >::operator()()
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20:16
> #14 0x377bf79 in thread_proxy
> (/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x377bf79)
> #15 0x32d4a07850 in start_thread (/lib64/libpthread.so.0+0x32d4a07850)
> #16 0x32d46e894c in clone (/lib64/libc.so.6+0x32d46e894c)
> 0x619002c94827 is located 89 bytes to the left of 991-byte region
> [0x619002c94880,0x619002c94c5f)
> allocated by thread T125815 here:
> #0 0x1654e88 in operator new(unsigned long)
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent64-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src/projects/compiler-rt/lib/asan/asan_new_delete.cc:92
> #1 0x20e2a05 in std::vector<unsigned char, std::allocator<unsigned char>
> >::_M_default_append(unsigned long)
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/vector.tcc:557:34
> #2 0x228c837 in impala::HdfsRCFileScanner::ReadKeyBuffers()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-rcfile-scanner.cc:312:53
> #3 0x228b8a0 in impala::HdfsRCFileScanner::StartRowGroup()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-rcfile-scanner.cc:259:41
> #4 0x228f006 in
> impala::HdfsRCFileScanner::ProcessRange(impala::RowBatch*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-rcfile-scanner.cc:531:41
> #5 0x3039cef in
> impala::BaseSequenceScanner::GetNextInternal(impala::RowBatch*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/base-sequence-scanner.cc:181:19
> #6 0x225c891 in impala::HdfsScanner::ProcessSplit()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-scanner.cc:134:21
> #7 0x221ad33 in
> impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext,
> std::allocator<impala::FilterContext> > const&, impala::MemPool*,
> impala::io::ScanRange*, long*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-scan-node.cc:453:21
> #8 0x2219e50 in impala::HdfsScanNode::ScannerThread(bool, long)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-scan-node.cc:360:16
> #9 0x1c4ffb6 in boost::function0<void>::operator()() const
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
> #10 0x211216e in impala::Thread::SuperviseThread(std::string const&,
> std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long>*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/thread.cc:356:3
> #11 0x211d3f8 in void boost::_bi::list5<boost::_bi::value<std::string>,
> boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string
> const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo
> const*, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>,
> void (*&)(std::string const&, std::string const&, boost::function<void ()>,
> impala::ThreadDebugInfo const*, impala::Promise<long>*), boost::_bi::list0&,
> int)
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525:9
> #12 0x211d24b in boost::_bi::bind_t<void, void (*)(std::string const&,
> std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long>*), boost::_bi::list5<boost::_bi::value<std::string>,
> boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long>*> > >::operator()()
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20:16
> #13 0x377bf79 in thread_proxy
> (/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x377bf79)
> Thread T125815 created by T125808 here:
> #0 0x1565d8d in __interceptor_pthread_create
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent64-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
> #1 0x377b359 in boost::thread::start_thread_noexcept()
> (/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x377b359)
> #2 0x45e0360d (<unknown module>)
> Thread T125808 created by T125805 here:
> #0 0x1565d8d in __interceptor_pthread_create
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent64-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
> #1 0x377b359 in boost::thread::start_thread_noexcept()
> (/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x377b359)
> #2 0x45e0360d (<unknown module>)
> Thread T125805 created by T198 here:
> #0 0x1565d8d in __interceptor_pthread_create
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent64-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
> #1 0x377b359 in boost::thread::start_thread_noexcept()
> (/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x377b359)
> #2 0x45e0360d (<unknown module>)
> Thread T198 created by T186 here:
> #0 0x1565d8d in __interceptor_pthread_create
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent64-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
> #1 0x377b359 in boost::thread::start_thread_noexcept()
> (/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x377b359)
> #2 0x45e0360d (<unknown module>)
> Thread T186 created by T185 here:
> #0 0x1565d8d in __interceptor_pthread_create
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent64-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
> #1 0x377b359 in boost::thread::start_thread_noexcept()
> (/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x377b359)
> #2 0x45e0360d (<unknown module>)
> Thread T185 created by T0 here:
> #0 0x1565d8d in __interceptor_pthread_create
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent64-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
> #1 0x377b359 in boost::thread::start_thread_noexcept()
> (/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x377b359)
> #2 0x45e0360d (<unknown module>)
> SUMMARY: AddressSanitizer: heap-buffer-overflow
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/read-write-util.h:200:31
> in impala::ReadWriteUtil::GetVLong(unsigned char*, long, long*, int)
> Shadow bytes around the buggy address:
> 0x0c328058a8b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c328058a8c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c328058a8d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c328058a8e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c328058a8f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> =>0x0c328058a900: fa fa fa fa[fa]fa fa fa fa fa fa fa fa fa fa fa
> 0x0c328058a910: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0c328058a920: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0c328058a930: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0c328058a940: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0c328058a950: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> Shadow byte legend (one shadow byte represents 8 application bytes):
> Addressable: 00
> Partially addressable: 01 02 03 04 05 06 07
> Heap left redzone: fa
> Freed heap region: fd
> Stack left redzone: f1
> Stack mid redzone: f2
> Stack right redzone: f3
> Stack after return: f5
> Stack use after scope: f8
> Global redzone: f9
> Global init order: f6
> Poisoned by user: f7
> Container overflow: fc
> Array cookie: ac
> Intra object redzone: bb
> ASan internal: fe
> Left alloca redzone: ca
> Right alloca redzone: cb
> ==31616==ABORTING
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
