[
https://issues.apache.org/jira/browse/IMPALA-5819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-5819.
-----------------------------------
Resolution: Fixed
Fix Version/s: Impala 2.10.0
IMPALA-5819: DCHECK in HdfsTextScanner::Close()
The DCHECK is only valid if the scanner successfully processes
the entire scan range. If the scanner is cancelled or hits an
error in-between buffers, boundary_column_ may be non-empty.
The fix is to move the DCHECK to the end of FinishScanRange(),
where it only executes if the scan range is fully processed.
Testing:
Ran TestCancellation in a loop locally for an hour.
Change-Id: I3d63f0b0c1ac82103a60b68b244156b9fbd575e0
Reviewed-on: http://gerrit.cloudera.org:8080/7756
Reviewed-by: Alex Behm <[email protected]>
Tested-by: Impala Public Jenkins
> HdfsTextScanner::Close hitting DCHECK(boundary_column_.IsEmpty())
> -----------------------------------------------------------------
>
> Key: IMPALA-5819
> URL: https://issues.apache.org/jira/browse/IMPALA-5819
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.10.0
> Reporter: Joe McDonnell
> Assignee: Tim Armstrong
> Priority: Blocker
> Labels: broken-build
> Fix For: Impala 2.10.0
>
>
> Test runs are seeing an Impala crash in HdfsTextScanner::Close due to this
> DCHECK(boundary_column_.IsEmpty()):
> https://github.com/apache/incubator-impala/blob/master/be/src/exec/hdfs-text-scanner.cc#L165
> This DCHECK is new in IMPALA-5776.
> The test that fails from this is
> query_test/test_cancellation.py::TestCancellationSerial::test_cancel_insert,
> but other tests are also running concurrently.
> The stack is:
> {code}
> #0 0x000000308ce328e5 in raise () from /lib64/libc.so.6
> #1 0x000000308ce340c5 in abort () from /lib64/libc.so.6
> #2 0x0000000002946484 in google::DumpStackTraceAndExit() ()
> #3 0x000000000293cf2d in google::LogMessage::Fail() ()
> #4 0x000000000293e7d2 in google::LogMessage::SendToLog() ()
> #5 0x000000000293c907 in google::LogMessage::Flush() ()
> #6 0x000000000293fece in google::LogMessageFatal::~LogMessageFatal() ()
> #7 0x000000000178a9ec in impala::HdfsTextScanner::Close (this=0x9c93cc0,
> row_batch=0x15fc1550) at
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-text-scanner.cc:165
> #8 0x0000000001754e18 in impala::HdfsScanner::Close (this=0x9c93cc0) at
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-scanner.cc:136
> #9 0x000000000172c98c in impala::HdfsScanNode::ProcessSplit (this=0xbf4e800,
> filter_ctxs=..., scan_range=0x101b12c0) at
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-scan-node.cc:546
> #10 0x000000000172ba4e in impala::HdfsScanNode::ScannerThread
> (this=0xbf4e800) at
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-scan-node.cc:418
> #11 0x000000000172b1bb in impala::HdfsScanNode::<lambda()>::operator()(void)
> const (__closure=0x7f49fb7adc28) at
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-scan-node.cc:355
> #12 0x000000000172cf15 in
> boost::detail::function::void_function_obj_invoker0<impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourceMgr::ResourcePool*)::<lambda()>,
> void>::invoke(boost::detail::function::function_buffer &)
> (function_obj_ptr=...) at
> /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
> #13 0x00000000013be698 in boost::function0<void>::operator()
> (this=0x7f49fb7adc20) at
> /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
> #14 0x0000000001689aa9 in impala::Thread::SuperviseThread (name=...,
> category=..., functor=..., thread_started=0x7f49f8fa8d70) at
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/thread.cc:329
> #15 0x0000000001692478 in
> boost::_bi::list4<boost::_bi::value<std::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void()> >,
> boost::_bi::value<impala::Promise<long int>*> >::operator()<void (*)(const
> std::basic_string<char>&, const std::basic_string<char>&,
> boost::function<void()>, impala::Promise<long int>*),
> boost::_bi::list0>(boost::_bi::type<void>, void (*&)(const
> std::basic_string<char, std::char_traits<char>, std::allocator<char> > &,
> const std::basic_string<char, std::char_traits<char>, std::allocator<char> >
> &, boost::function<void()>, impala::Promise<long> *), boost::_bi::list0 &,
> int) (this=0xdb1cbc0, f=@0xdb1cbb8, a=...) at
> /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:457
> #16 0x00000000016923bb in boost::_bi::bind_t<void, void (*)(const
> std::basic_string<char, std::char_traits<char>, std::allocator<char> >&,
> const std::basic_string<char, std::char_traits<char>, std::allocator<char>
> >&, boost::function<void()>, impala::Promise<long int>*),
> boost::_bi::list4<boost::_bi::value<std::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void()> >,
> boost::_bi::value<impala::Promise<long int>*> > >::operator()(void)
> (this=0xdb1cbb8) at
> /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
> #17 0x000000000169237e in boost::detail::thread_data<boost::_bi::bind_t<void,
> void (*)(const std::basic_string<char, std::char_traits<char>,
> std::allocator<char> >&, const std::basic_string<char,
> std::char_traits<char>, std::allocator<char> >&, boost::function<void()>,
> impala::Promise<long int>*),
> boost::_bi::list4<boost::_bi::value<std::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void()> >,
> boost::_bi::value<impala::Promise<long int>*> > > >::run(void)
> (this=0xdb1ca00) at
> /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:116
> #18 0x0000000001b6e8da in thread_proxy ()
> #19 0x000000308d207851 in start_thread () from /lib64/libpthread.so.0
> #20 0x000000308cee894d in clone () from /lib64/libc.so.6
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
