[
https://issues.apache.org/jira/browse/IMPALA-5798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Tauber-Marshall resolved IMPALA-5798.
--------------------------------------------
Resolution: Fixed
Fix Version/s: Impala 2.10.0
commit cb645b1bcb3e18123bee0916da9fbcf7ff55050d
Author: Thomas Tauber-Marshall <[email protected]>
Date: Mon Aug 21 17:58:09 2017 -0700
IMPALA-5798: ASAN use-after-poison in Parquet decoder
In ParquetLevelDecoder::Init() for RLE encoding, we read the metadata
size and advance the data buffer past it. If the metadata size is
corrupted, it can cause us to incorrectly read past the end of the
buffer.
This patch checks that the metadata size is less than the total size
of the buffer, and returns an error if it isn't.
Testing:
- Ran test_scanners_fuzz.py under ASAN 500 times without hitting the
use-after-poison (previously it would usually hit in < 100 runs).
Change-Id: I3f3d0d998f7581c7c935d98fde886f145efd61a8
Reviewed-on: http://gerrit.cloudera.org:8080/7769
Reviewed-by: Alex Behm <[email protected]>
Reviewed-by: Matthew Jacobs <[email protected]>
Tested-by: Impala Public Jenkins
> ASAN use-after-poison in Parquet decoder
> ----------------------------------------
>
> Key: IMPALA-5798
> URL: https://issues.apache.org/jira/browse/IMPALA-5798
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.10.0
> Reporter: Joe McDonnell
> Assignee: Thomas Tauber-Marshall
> Priority: Blocker
> Labels: broken-build
> Fix For: Impala 2.10.0
>
>
> ASAN build fails with a use-after-poison. Note that the poison logic was
> recently enhanced with IMPALA-5666:
> {code}
> ==649==ERROR: AddressSanitizer: use-after-poison on address 0x62100627150a at
> pc 0x000001022e45 bp 0x7f48623255d0 sp 0x7f4862324d80
> READ of size 8 at 0x62100627150a thread T81673
> #0 0x1022e44 in __asan_memcpy
> /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:393
> #1 0x19301d2 in impala::BitReader::Reset(unsigned char const*, int)
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/bit-stream-utils.h:119:5
> #2 0x192ff2b in impala::RleDecoder::Reset(unsigned char*, int, int)
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/rle-encoding.h:99:5
> #3 0x1924445 in impala::ParquetLevelDecoder::Init(std::string const&,
> parquet::Encoding::type, impala::MemPool*, int, int, int, unsigned char**,
> int*)
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/parquet-column-readers.cc:97:7
> #4 0x192ce68 in impala::BaseScalarColumnReader::ReadDataPage()
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/parquet-column-readers.cc:1061:31
> #5 0x192e1d5 in impala::BaseScalarColumnReader::NextPage()
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/parquet-column-readers.cc:1106:28
> #6 0x198e68a in bool
> impala::ScalarColumnReader<impala::DecimalValue<int>,
> true>::ReadValueBatch<false>(impala::MemPool*, int, int, unsigned char*,
> int*)
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/parquet-column-readers.cc:316:14
> #7 0x189d282 in
> impala::HdfsParquetScanner::AssembleRows(std::vector<impala::ParquetColumnReader*,
> std::allocator<impala::ParquetColumnReader*> > const&, impala::RowBatch*,
> bool*)
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:976:30
> #8 0x1899134 in
> impala::HdfsParquetScanner::GetNextInternal(impala::RowBatch*)
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:514:19
> #9 0x18972f6 in impala::HdfsParquetScanner::ProcessSplit()
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:411:21
> #10 0x17fd0ff in
> impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext,
> std::allocator<impala::FilterContext> > const&,
> impala::DiskIoMgr::ScanRange*)
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-scan-node.cc:528:12
> #11 0x17fc3fe in impala::HdfsScanNode::ScannerThread()
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-scan-node.cc:418:16
> #12 0x1336802 in boost::function0<void>::operator()() const
> /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
> #13 0x1722b9d in impala::Thread::SuperviseThread(std::string const&,
> std::string const&, boost::function<void ()>, impala::Promise<long>*)
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/thread.cc:330:3
> #14 0x172cd6a in void boost::_bi::list4<boost::_bi::value<std::string>,
> boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string
> const&, std::string const&, boost::function<void ()>,
> impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void
> (*&)(std::string const&, std::string const&, boost::function<void ()>,
> impala::Promise<long>*), boost::_bi::list0&, int)
> /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:457:9
> #15 0x172cbf7 in boost::_bi::bind_t<void, void (*)(std::string const&,
> std::string const&, boost::function<void ()>, impala::Promise<long>*),
> boost::_bi::list4<boost::_bi::value<std::string>,
> boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::Promise<long>*> > >::operator()()
> /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20:16
> #16 0x1e1b1c9 in thread_proxy
> (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/service/impalad+0x1e1b1c9)
> #17 0x31f7607850 in start_thread (/lib64/libpthread.so.0+0x31f7607850)
> #18 0x31f72e894c in clone (/lib64/libc.so.6+0x31f72e894c)
> 0x62100627150a is located 10 bytes inside of 4096-byte region
> [0x621006271500,0x621006272500)
> allocated by thread T81673 here:
> #0 0x1038da8 in __interceptor_malloc
> /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:52
> #1 0x1360f1a in impala::MemPool::FindChunk(long, bool)
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/mem-pool.cc:149:45
> #2 0x10b2eda in unsigned char* impala::MemPool::Allocate<true>(long, int)
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/mem-pool.h:266:32
> #3 0x192c775 in impala::BaseScalarColumnReader::ReadDataPage()
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/parquet-column-readers.cc:1024:11
> #4 0x192e1d5 in impala::BaseScalarColumnReader::NextPage()
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/parquet-column-readers.cc:1106:28
> #5 0x198e68a in bool
> impala::ScalarColumnReader<impala::DecimalValue<int>,
> true>::ReadValueBatch<false>(impala::MemPool*, int, int, unsigned char*,
> int*)
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/parquet-column-readers.cc:316:14
> #6 0x189d282 in
> impala::HdfsParquetScanner::AssembleRows(std::vector<impala::ParquetColumnReader*,
> std::allocator<impala::ParquetColumnReader*> > const&, impala::RowBatch*,
> bool*)
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:976:30
> #7 0x1899134 in
> impala::HdfsParquetScanner::GetNextInternal(impala::RowBatch*)
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:514:19
> #8 0x18972f6 in impala::HdfsParquetScanner::ProcessSplit()
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:411:21
> #9 0x17fd0ff in
> impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext,
> std::allocator<impala::FilterContext> > const&,
> impala::DiskIoMgr::ScanRange*)
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-scan-node.cc:528:12
> #10 0x17fc3fe in impala::HdfsScanNode::ScannerThread()
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-scan-node.cc:418:16
> #11 0x1336802 in boost::function0<void>::operator()() const
> /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
> #12 0x1722b9d in impala::Thread::SuperviseThread(std::string const&,
> std::string const&, boost::function<void ()>, impala::Promise<long>*)
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/thread.cc:330:3
> #13 0x172cd6a in void boost::_bi::list4<boost::_bi::value<std::string>,
> boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string
> const&, std::string const&, boost::function<void ()>,
> impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void
> (*&)(std::string const&, std::string const&, boost::function<void ()>,
> impala::Promise<long>*), boost::_bi::list0&, int)
> /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:457:9
> #14 0x172cbf7 in boost::_bi::bind_t<void, void (*)(std::string const&,
> std::string const&, boost::function<void ()>, impala::Promise<long>*),
> boost::_bi::list4<boost::_bi::value<std::string>,
> boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::Promise<long>*> > >::operator()()
> /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20:16
> #15 0x1e1b1c9 in thread_proxy
> (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/service/impalad+0x1e1b1c9)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
