[jira] [Commented] (IMPALA-7113) ASAN heap-buffer-overflow in impala::HdfsRCFileScanner::GetCurrentKeyBuffer()
[
https://issues.apache.org/jira/browse/IMPALA-7113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498915#comment-16498915
]
Lars Volker commented on IMPALA-7113:
-
Unfortunately ASAN builds don't leave cores files or minidumps. :(
> ASAN heap-buffer-overflow in impala::HdfsRCFileScanner::GetCurrentKeyBuffer()
> -
>
> Key: IMPALA-7113
> URL: https://issues.apache.org/jira/browse/IMPALA-7113
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Lars Volker
>Assignee: Rahul Shivu Mahadev
>Priority: Blocker
> Labels: asan, broken-build
>
> [~pranay_singh] - I'm assigning this to you since you changed this code last
> in IMPALA-3833.
> {noformat}
> ==31616==ERROR: AddressSanitizer: heap-buffer-overflow on address
> 0x619002c94827 at pc 0x02293cf2 bp 0x7f653d570eb0 sp 0x7f653d570ea8
> READ of size 1 at 0x619002c94827 thread T125815
> #0 0x2293cf1 in impala::ReadWriteUtil::GetVLong(unsigned char*, long,
> long*, int)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/read-write-util.h:200:31
> #1 0x2292114 in impala::ReadWriteUtil::GetVInt(unsigned char*, int*, int)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/read-write-util.h:184:13
> #2 0x228e5c6 in impala::HdfsRCFileScanner::GetCurrentKeyBuffer(int, bool,
> unsigned char**, int)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-rcfile-scanner.cc:379:20
> #3 0x228ce07 in impala::HdfsRCFileScanner::ReadKeyBuffers()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-rcfile-scanner.cc:354:41
> #4 0x228b8a0 in impala::HdfsRCFileScanner::StartRowGroup()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-rcfile-scanner.cc:259:41
> #5 0x228f006 in
> impala::HdfsRCFileScanner::ProcessRange(impala::RowBatch*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-rcfile-scanner.cc:531:41
> #6 0x3039cef in
> impala::BaseSequenceScanner::GetNextInternal(impala::RowBatch*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/base-sequence-scanner.cc:181:19
> #7 0x225c891 in impala::HdfsScanner::ProcessSplit()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-scanner.cc:134:21
> #8 0x221ad33 in
> impala::HdfsScanNode::ProcessSplit(std::vector std::allocator > const&, impala::MemPool*,
> impala::io::ScanRange*, long*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-scan-node.cc:453:21
> #9 0x2219e50 in impala::HdfsScanNode::ScannerThread(bool, long)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-scan-node.cc:360:16
> #10 0x1c4ffb6 in boost::function0::operator()() const
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
> #11 0x211216e in impala::Thread::SuperviseThread(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/thread.cc:356:3
> #12 0x211d3f8 in void boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*> >::operator() const&, std::string const&, boost::function, impala::ThreadDebugInfo
> const*, impala::Promise*), boost::_bi::list0>(boost::_bi::type,
> void (*&)(std::string const&, std::string const&, boost::function,
> impala::ThreadDebugInfo const*, impala::Promise*), boost::_bi::list0&,
> int)
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525:9
> #13 0x211d24b in boost::_bi::bind_t std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*), boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*> > >::operator()()
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20:16
> #14 0x377bf79 in thread_proxy
> (/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x377bf79)
> #15 0x32d4a07850 in start_thread (/lib64/libpthread.so.0+0x32d4a07850)
> #16 0x32d46e894c in clone (/lib64/libc.so.6+0x32d46e894c)
> 0x619002c94827 is located 89 bytes to the left of 991-byte region
> [0x619002c94880,0x619002c94c5f)
> allocated by thread T125815 here:
> #0 0x1654e88 in operator new(unsigned long)
>
[jira] [Commented] (IMPALA-7113) ASAN heap-buffer-overflow in impala::HdfsRCFileScanner::GetCurrentKeyBuffer()
[
https://issues.apache.org/jira/browse/IMPALA-7113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498673#comment-16498673
]
Tim Armstrong commented on IMPALA-7113:
---
It looks like the fuzz test was running for RC, so this is probably just
IMPALA-7058, unless I'm missing something.
{noformat}
04:44:53
query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_alltypes[exec_option:
{'debug_action': None, 'abort_on_error': False, 'mem_limit': '512m',
'num_nodes': 0} | table_format: rc/snap/block]
04:44:54 [gw2] PASSED
query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_alltypes[exec_option:
{'debug_action': None, 'abort_on_error': False, 'mem_limit': '512m',
'num_nodes': 0} | table_format: rc/snap/block]
04:45:07
query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_alltypes[exec_option:
{'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5',
'abort_on_error': False, 'mem_limit': '512m', 'num_nodes': 0} | table_format:
rc/snap/block]
04:45:07 [gw1] PASSED
query_test/test_scratch_limit.py::TestScratchLimit::test_with_zero_scratch_limit_no_memory_limit[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: text/none]
04:45:37
query_test/test_sort.py::TestQueryFullSort::test_multiple_mem_limits[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
04:45:37 [gw0] FAILED
query_test/test_decimal_queries.py::TestDecimalExprs::test_exprs[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: kudu/none]
04:45:44
query_test/test_decimal_queries.py::TestAvroDecimalQueries::test_avro_queries[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: avro/snap/block]
04:45:44 [gw0] FAILED
query_test/test_decimal_queries.py::TestAvroDecimalQueries::test_avro_queries[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: avro/snap/block]
04:45:45
query_test/test_delimited_text.py::TestDelimitedText::test_delimited_text[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: text/none]
04:45:45 [gw1] FAILED
query_test/test_sort.py::TestQueryFullSort::test_multiple_mem_limits[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
04:45:45
query_test/test_sort.py::TestQueryFullSort::test_multiple_mem_limits_full_output[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
04:45:45 [gw3] FAILED
query_test/test_queries.py::TestQueries::test_subquery[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
04:46:06 query_test/test_queries.py::TestQueries::test_alias[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
04:46:06 [gw2] PASSED
query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_alltypes[exec_option:
{'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5',
'abort_on_error': False, 'mem_limit': '512m', 'num_nodes': 0} | table_format:
rc/snap/block]
04:46:06 [gw3] FAILED
query_test/test_queries.py::TestQueries::test_alias[exec_option: {'batch_size':
0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen':
False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
04:46:06
query_test/test_queries.py::TestQueries::test_subquery_in_constant_lhs[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': False, 'abort_on_error': 1,
[jira] [Commented] (IMPALA-7113) ASAN heap-buffer-overflow in impala::HdfsRCFileScanner::GetCurrentKeyBuffer()
[
https://issues.apache.org/jira/browse/IMPALA-7113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498654#comment-16498654
]
Tim Armstrong commented on IMPALA-7113:
---
See also IMPALA-7058. We should confirm whether the commit that crashed had the
fix for IMPALA-7058 (if not, it's a dupe of that).
> ASAN heap-buffer-overflow in impala::HdfsRCFileScanner::GetCurrentKeyBuffer()
> -
>
> Key: IMPALA-7113
> URL: https://issues.apache.org/jira/browse/IMPALA-7113
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Lars Volker
>Assignee: Rahul Shivu Mahadev
>Priority: Blocker
> Labels: asan, broken-build
>
> [~pranay_singh] - I'm assigning this to you since you changed this code last
> in IMPALA-3833.
> {noformat}
> ==31616==ERROR: AddressSanitizer: heap-buffer-overflow on address
> 0x619002c94827 at pc 0x02293cf2 bp 0x7f653d570eb0 sp 0x7f653d570ea8
> READ of size 1 at 0x619002c94827 thread T125815
> #0 0x2293cf1 in impala::ReadWriteUtil::GetVLong(unsigned char*, long,
> long*, int)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/read-write-util.h:200:31
> #1 0x2292114 in impala::ReadWriteUtil::GetVInt(unsigned char*, int*, int)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/read-write-util.h:184:13
> #2 0x228e5c6 in impala::HdfsRCFileScanner::GetCurrentKeyBuffer(int, bool,
> unsigned char**, int)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-rcfile-scanner.cc:379:20
> #3 0x228ce07 in impala::HdfsRCFileScanner::ReadKeyBuffers()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-rcfile-scanner.cc:354:41
> #4 0x228b8a0 in impala::HdfsRCFileScanner::StartRowGroup()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-rcfile-scanner.cc:259:41
> #5 0x228f006 in
> impala::HdfsRCFileScanner::ProcessRange(impala::RowBatch*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-rcfile-scanner.cc:531:41
> #6 0x3039cef in
> impala::BaseSequenceScanner::GetNextInternal(impala::RowBatch*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/base-sequence-scanner.cc:181:19
> #7 0x225c891 in impala::HdfsScanner::ProcessSplit()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-scanner.cc:134:21
> #8 0x221ad33 in
> impala::HdfsScanNode::ProcessSplit(std::vector std::allocator > const&, impala::MemPool*,
> impala::io::ScanRange*, long*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-scan-node.cc:453:21
> #9 0x2219e50 in impala::HdfsScanNode::ScannerThread(bool, long)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-scan-node.cc:360:16
> #10 0x1c4ffb6 in boost::function0::operator()() const
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
> #11 0x211216e in impala::Thread::SuperviseThread(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/thread.cc:356:3
> #12 0x211d3f8 in void boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*> >::operator() const&, std::string const&, boost::function, impala::ThreadDebugInfo
> const*, impala::Promise*), boost::_bi::list0>(boost::_bi::type,
> void (*&)(std::string const&, std::string const&, boost::function,
> impala::ThreadDebugInfo const*, impala::Promise*), boost::_bi::list0&,
> int)
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525:9
> #13 0x211d24b in boost::_bi::bind_t std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*), boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*> > >::operator()()
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20:16
> #14 0x377bf79 in thread_proxy
> (/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x377bf79)
> #15 0x32d4a07850 in start_thread (/lib64/libpthread.so.0+0x32d4a07850)
> #16 0x32d46e894c in clone (/lib64/libc.so.6+0x32d46e894c)
> 0x619002c94827 is located 89 bytes to the left of 991-byte region
> [0x619002c94880,0x619002c94c5f)
> allocated by thread
[jira] [Commented] (IMPALA-7113) ASAN heap-buffer-overflow in impala::HdfsRCFileScanner::GetCurrentKeyBuffer()
[
https://issues.apache.org/jira/browse/IMPALA-7113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498606#comment-16498606
]
Sailesh Mukil commented on IMPALA-7113:
---
Assigning this to [~rahul.mahadev]. Please reach out to [~pranay_singh] if you
have any concerns specific to IMPALA-3833.
> ASAN heap-buffer-overflow in impala::HdfsRCFileScanner::GetCurrentKeyBuffer()
> -
>
> Key: IMPALA-7113
> URL: https://issues.apache.org/jira/browse/IMPALA-7113
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Lars Volker
>Assignee: Rahul Shivu Mahadev
>Priority: Blocker
> Labels: asan, broken-build
>
> [~pranay_singh] - I'm assigning this to you since you changed this code last
> in IMPALA-3833.
> {noformat}
> ==31616==ERROR: AddressSanitizer: heap-buffer-overflow on address
> 0x619002c94827 at pc 0x02293cf2 bp 0x7f653d570eb0 sp 0x7f653d570ea8
> READ of size 1 at 0x619002c94827 thread T125815
> #0 0x2293cf1 in impala::ReadWriteUtil::GetVLong(unsigned char*, long,
> long*, int)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/read-write-util.h:200:31
> #1 0x2292114 in impala::ReadWriteUtil::GetVInt(unsigned char*, int*, int)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/read-write-util.h:184:13
> #2 0x228e5c6 in impala::HdfsRCFileScanner::GetCurrentKeyBuffer(int, bool,
> unsigned char**, int)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-rcfile-scanner.cc:379:20
> #3 0x228ce07 in impala::HdfsRCFileScanner::ReadKeyBuffers()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-rcfile-scanner.cc:354:41
> #4 0x228b8a0 in impala::HdfsRCFileScanner::StartRowGroup()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-rcfile-scanner.cc:259:41
> #5 0x228f006 in
> impala::HdfsRCFileScanner::ProcessRange(impala::RowBatch*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-rcfile-scanner.cc:531:41
> #6 0x3039cef in
> impala::BaseSequenceScanner::GetNextInternal(impala::RowBatch*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/base-sequence-scanner.cc:181:19
> #7 0x225c891 in impala::HdfsScanner::ProcessSplit()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-scanner.cc:134:21
> #8 0x221ad33 in
> impala::HdfsScanNode::ProcessSplit(std::vector std::allocator > const&, impala::MemPool*,
> impala::io::ScanRange*, long*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-scan-node.cc:453:21
> #9 0x2219e50 in impala::HdfsScanNode::ScannerThread(bool, long)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-scan-node.cc:360:16
> #10 0x1c4ffb6 in boost::function0::operator()() const
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
> #11 0x211216e in impala::Thread::SuperviseThread(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/thread.cc:356:3
> #12 0x211d3f8 in void boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*> >::operator() const&, std::string const&, boost::function, impala::ThreadDebugInfo
> const*, impala::Promise*), boost::_bi::list0>(boost::_bi::type,
> void (*&)(std::string const&, std::string const&, boost::function,
> impala::ThreadDebugInfo const*, impala::Promise*), boost::_bi::list0&,
> int)
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525:9
> #13 0x211d24b in boost::_bi::bind_t std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*), boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*> > >::operator()()
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20:16
> #14 0x377bf79 in thread_proxy
> (/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x377bf79)
> #15 0x32d4a07850 in start_thread (/lib64/libpthread.so.0+0x32d4a07850)
> #16 0x32d46e894c in clone (/lib64/libc.so.6+0x32d46e894c)
> 0x619002c94827 is located 89 bytes to the left of 991-byte region
> [0x619002c94880,0x619002c94c5f)
> allocated by thread T125815
