Tim Armstrong created IMPALA-9449:
-------------------------------------
Summary: Crash in LZO scan with data cache enabled
Key: IMPALA-9449
URL: https://issues.apache.org/jira/browse/IMPALA-9449
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Tim Armstrong
{noformat}
at org.apache.hadoop.cF0301 08:25:39.334376 21960
hdfs-lzo-text-scanner.cc:582] 744cd34df9463fba:610c3d1000000003] Check failed:
stream_->eof()
*** Check failure stack trace: ***
@ 0x4e7bf5c google::LogMessage::Fail()
@ 0x4e7d801 google::LogMessage::SendToLog()
@ 0x4e7b936 google::LogMessage::Flush()
@ 0x4e7eefd google::LogMessageFatal::~LogMessageFatal()
@ 0x7fcf1248c2bd impala::HdfsLzoTextScanner::ReadAndDecompressData()
@ 0x7fcf1248a0da impala::HdfsLzoTextScanner::ReadData()
@ 0x7fcf1248a635 impala::HdfsLzoTextScanner::FillByteBuffer()
@ 0x269f4ae impala::HdfsTextScanner::FillByteBufferWrapper()
@ 0x269c047 impala::HdfsTextScanner::FinishScanRange()
@ 0x269f272 impala::HdfsTextScanner::GetNextInternal()
@ 0x7fcf12488b0d impala::HdfsLzoTextScanner::GetNextInternal()
@ 0x2655808 impala::HdfsScanner::ProcessSplit()
@ 0x27c1a4b impala::HdfsScanNode::ProcessSplit()
@ 0x27c0c20 impala::HdfsScanNode::ScannerThread()
@ 0x27bff7d
_ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
@ 0x27c2571
_ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
@ 0x1fb39d5 boost::function0<>::operator()()
@ 0x257402a impala::Thread::SuperviseThread()
@ 0x257c2ae boost::_bi::list5<>::operator()<>()
@ 0x257c1d2 boost::_bi::bind_t<>::operator()()
@ 0x257c195 boost::detail::thread_data<>::run()
@ 0x3da22f9 thread_proxy
@ 0x7fcfdab10e24 start_thread
@ 0x7fcfd767c34c __clone
rypto.CryptoInputStream.read(CryptoInputStr
{noformat}
{noformat}
CORE: ./core.1583079954.31711.impalad
BINARY: ./be/build/latest/service/impalad
Core was generated by
`/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f231918b1f7 in raise () from /lib64/libc.so.6
To enable execution of this file add
add-auto-load-safe-path
/data0/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/gcc-4.9.2/lib64/libstdc++.so.6.0.20-gdb.py
line to your configuration file "/var/lib/jenkins/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/var/lib/jenkins/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
info "(gdb)Auto-loading safe path"
#0 0x00007f231918b1f7 in raise () from /lib64/libc.so.6
#1 0x00007f231918c8e8 in abort () from /lib64/libc.so.6
#2 0x0000000004e85504 in google::DumpStackTraceAndExit() ()
#3 0x0000000004e7bf5d in google::LogMessage::Fail() ()
#4 0x0000000004e7d802 in google::LogMessage::SendToLog() ()
#5 0x0000000004e7b937 in google::LogMessage::Flush() ()
#6 0x0000000004e7eefe in google::LogMessageFatal::~LogMessageFatal() ()
#7 0x00007f2254b1d2be in impala::HdfsLzoTextScanner::ReadAndDecompressData
(this=0x182062c0, pool=0xdb6cb68) at
/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala-lzo/hdfs-lzo-text-scanner.cc:582
#8 0x00007f2254b1b0db in impala::HdfsLzoTextScanner::ReadData
(this=0x182062c0, pool=0xdb6cb68) at
/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala-lzo/hdfs-lzo-text-scanner.cc:307
#9 0x00007f2254b1b636 in impala::HdfsLzoTextScanner::FillByteBuffer
(this=0x182062c0, pool=0xdb6cb68, eosr=0x7f224091f49e, num_bytes=65536) at
/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala-lzo/hdfs-lzo-text-scanner.cc:342
#10 0x000000000269f4af in impala::HdfsTextScanner::FillByteBufferWrapper
(this=0x182062c0, pool=0xdb6cb68, eosr=0x7f224091f49e, num_bytes=65536) at
/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-text-scanner.cc:476
#11 0x000000000269c048 in impala::HdfsTextScanner::FinishScanRange
(this=0x182062c0, row_batch=0xdb6cb40) at
/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-text-scanner.cc:282
#12 0x000000000269f273 in impala::HdfsTextScanner::GetNextInternal
(this=0x182062c0, row_batch=0xdb6cb40) at
/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-text-scanner.cc:467
#13 0x00007f2254b19b0e in impala::HdfsLzoTextScanner::GetNextInternal
(this=0x182062c0, row_batch=0xdb6cb40) at
/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala-lzo/hdfs-lzo-text-scanner.cc:144
#14 0x0000000002655809 in impala::HdfsScanner::ProcessSplit (this=0x182062c0)
at
/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-scanner.cc:135
#15 0x00000000027c1a4c in impala::HdfsScanNode::ProcessSplit (this=0xa3d78d00,
filter_ctxs=..., expr_results_pool=0x7f2240920410, scan_range=0x109d1f180,
scanner_thread_reservation=0x7f2240920368) at
/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-scan-node.cc:515
#16 0x00000000027c0c21 in impala::HdfsScanNode::ScannerThread (this=0xa3d78d00,
first_thread=false, scanner_thread_reservation=8192) at
/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-scan-node.cc:417
#17 0x00000000027bff7e in impala::HdfsScanNode::<lambda()>::operator()(void)
const (__closure=0x7f2240920b98) at
/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-scan-node.cc:338
#18 0x00000000027c2572 in
boost::detail::function::void_function_obj_invoker0<impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*)::<lambda()>,
void>::invoke(boost::detail::function::function_buffer &)
(function_obj_ptr=...) at
/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159
#19 0x0000000001fb39d6 in boost::function0<void>::operator()
(this=0x7f2240920b90) at
/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:771
#20 0x000000000257402b in impala::Thread::SuperviseThread(std::string const&,
std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
impala::Promise<long, (impala::PromiseMode)0>*) (name=..., category=...,
functor=..., parent_thread_info=0x7f2242720840, thread_started=0x7f224271f540)
at
/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/util/thread.cc:360
#21 0x000000000257c2af in boost::_bi::list5<boost::_bi::value<std::string>,
boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >,
boost::_bi::value<impala::ThreadDebugInfo*>,
boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*>
>::operator()<void (*)(std::string const&, std::string const&,
boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
(impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void
(*&)(std::string const&, std::string const&, boost::function<void ()>,
impala::ThreadDebugInfo const*, impala::Promise<long,
(impala::PromiseMode)0>*), boost::_bi::list0&, int) (this=0x1989013c0,
f=@0x1989013b8: 0x2573cc4 <impala::Thread::SuperviseThread(std::string const&,
std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
impala::Promise<long, (impala::PromiseMode)0>*)>, a=...) at
/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531
#22 0x000000000257c1d3 in boost::_bi::bind_t<void, void (*)(std::string const&,
std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
impala::Promise<long, (impala::PromiseMode)0>*),
boost::_bi::list5<boost::_bi::value<std::string>,
boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >,
boost::_bi::value<impala::ThreadDebugInfo*>,
boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >
>::operator()() (this=0x1989013b8) at
/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222
#23 0x000000000257c196 in boost::detail::thread_data<boost::_bi::bind_t<void,
void (*)(std::string const&, std::string const&, boost::function<void ()>,
impala::ThreadDebugInfo const*, impala::Promise<long,
(impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>,
boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >,
boost::_bi::value<impala::ThreadDebugInfo*>,
boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::run()
(this=0x198901200) at
/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116
#24 0x0000000003da22fa in thread_proxy ()
#25 0x00007f231c6e2e25 in start_thread () from /lib64/libpthread.so.0
#26 0x00007f231924e34d in clone () from /lib64/libc.so.6
{noformat}
This crashed all 3 impalads at around the same time:
{noformat}
Log file created at: 2020/03/01 08:26:07
Running on machine:
impala-ec2-centos74-m5-4xlarge-ondemand-0a0f.vpc.cloudera.com
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0301 08:26:07.666946 27213 hdfs-lzo-text-scanner.cc:582]
f44a5121256f5b7d:8e0bf92c00000001] Check failed: stream_->eof()
Log file created at: 2020/03/01 08:25:53
Running on machine:
impala-ec2-centos74-m5-4xlarge-ondemand-0a0f.vpc.cloudera.com
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0301 08:25:53.913563 25657 hdfs-lzo-text-scanner.cc:582]
1d4781605704837d:2a54525400000001] Check failed: stream_->eof()
Log file created at: 2020/03/01 08:25:39
Running on machine:
impala-ec2-centos74-m5-4xlarge-ondemand-0a0f.vpc.cloudera.com
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0301 08:25:39.334376 21960 hdfs-lzo-text-scanner.cc:582]
744cd34df9463fba:610c3d1000000003] Check failed: stream_->eof()
{noformat}
There was some noise in the log around the same time, unsure if it's relevant
{noformat}
hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
ReadOnlyBufferException: java.nio.ReadOnlyBufferException
at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
at
org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
at
org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
at
org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
ReadOnlyBufferException: java.nio.ReadOnlyBufferException
at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
at
org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
at
org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
at
org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
ReadOnlyBufferException: java.nio.ReadOnlyBufferException
at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
at
org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
at
org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
at
org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
ReadOnlyBufferException: java.nio.ReadOnlyBufferException
at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
at
org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
at
org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
at
org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
ReadOnlyBufferException: java.nio.ReadOnlyBufferException
at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
at
org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
at
org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
at
org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
ReadOnlyBufferException: java.nio.ReadOnlyBufferException
at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
at
org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
at
org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
at
org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
ReadOnlyBufferException: java.nio.ReadOnlyBufferException
at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
at
org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
at
org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
at
org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
ReadOnlyBufferException: java.nio.ReadOnlyBufferException
at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
at
org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
at org.apache.hadoop.cF0301 08:25:39.334376 21960
hdfs-lzo-text-scanner.cc:582] 744cd34df9463fba:610c3d1000000003] Check failed:
stream_->eof()
*** Check failure stack trace: ***
{noformat}
This was testing with a slightly old impala-lzo commit:
https://github.com/cloudera/impala-lzo/commit/36d19a8a4559ec6703e5879684b0d341c75c0202
I'll attach logs and more info soon.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
