Philip Zeyliger created IMPALA-6932:
---------------------------------------
Summary: Simple LIMIT 1 query can be really slow on many-filed
Avro datasets
Key: IMPALA-6932
URL: https://issues.apache.org/jira/browse/IMPALA-6932
Project: IMPALA
Issue Type: Task
Components: Backend
Reporter: Philip Zeyliger
I recently ran across really slow behavior with the trivial {{SELECT * FROM
table LIMIT 1}} query. The table used Avro as a file format and had about
45,000 files across about 250 partitions. An optimization kicked in to set
NUM_NODES to 1.
The query ran for about an hour, and the profile indicated that it was opening
files:
- TotalRawHdfsOpenFileTime(*): 1.0h (3622833666032)
I took a single minidump while this query was running, and I suspect the query
was here:
{code:java}
1 impalad!impala::ScannerContext::Stream::GetNextBuffer(long)
[scanner-context.cc : 115 + 0x13]
2 impalad!impala::ScannerContext::Stream::GetBytesInternal(long, unsigned
char**, bool, long*) [scanner-context.cc : 241 + 0x5]
3 impalad!impala::HdfsAvroScanner::ReadFileHeader() [scanner-context.inline.h :
54 + 0x1f]
4 impalad!impala::BaseSequenceScanner::GetNextInternal(impala::RowBatch*)
[base-sequence-scanner.cc : 157 + 0x13]
5 impalad!impala::HdfsScanner::ProcessSplit() [hdfs-scanner.cc : 129 + 0xc]
6 impalad!impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext,
std::allocator<impala::FilterContext> > const&, impala::MemPool*,
impala::io::ScanRange*) [hdfs-scan-node.cc : 527 + 0x17]
7 impalad!impala::HdfsScanNode::ScannerThread() [hdfs-scan-node.cc : 437 + 0x1c]
8 impalad!impala::Thread::SuperviseThread(std::string const&, std::string
const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
impala::Promise<long>*) [function_template.hpp : 767 + 0x7]{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
