Zoram Thanga created IMPALA-6996:
------------------------------------
Summary: PartitionedAggregationNode::Close() should not allocate
memory
Key: IMPALA-6996
URL: https://issues.apache.org/jira/browse/IMPALA-6996
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 2.12.0
Reporter: Zoram Thanga
A CTAS query with MEM_LIMIT=6GB hit the limit while in an AggregationNode. Upon
processing the failure, the fragment instance that hit the MEM_LIMIT tries to
allocate more memory, and repeatedly dumps the following error message and
stack trace to Impalad log file, once per row.
{noformat}
I0508 01:32:10.611563 40614 status.cc:55] Memory limit exceeded:
FunctionContextImpl::AllocateLocal's allocations exceeded memory limits.
Exprs could not allocate 3.00 B without exceeding limit.
Error occurred on backend <redacted>.com:22000 by fragment
b429287bdabe184:9e7b5e960000002c
Memory left in process limit: 384.47 GB
Memory left in query limit: -6947009.00 B
Query(b429287bdabe184:9e7b5e9600000000): memory limit exceeded. Limit=6.00 GB
Total=6.01 GB Peak=6.01 GB
Fragment b429287bdabe184:9e7b5e960000002c: Total=2.76 GB Peak=2.76 GB
AGGREGATION_NODE (id=3): Total=2.75 GB Peak=2.75 GB
Exprs: Total=751.94 MB Peak=751.94 MB
EXCHANGE_NODE (id=2): Total=0 Peak=0
DataStreamRecvr: Total=7.42 MB Peak=7.42 MB
CodeGen: Total=5.57 KB Peak=745.50 KB
Block Manager: Limit=4.80 GB Total=4.39 GB Peak=4.40 GB
Fragment b429287bdabe184:9e7b5e960000000c: Total=3.25 GB Peak=3.25 GB
AGGREGATION_NODE (id=1): Total=3.19 GB Peak=3.19 GB
Exprs: Total=816.06 MB Peak=816.06 MB
HDFS_SCAN_NODE (id=0): Total=59.96 MB Peak=200.09 MB
CodeGen: Total=5.96 KB Peak=876.00 KB
@ 0x83c78a impala::Status::Status()
@ 0x83c98e impala::Status::MemLimitExceeded()
@ 0xa24344 impala::MemTracker::MemLimitExceeded()
@ 0xa35ccd impala::RuntimeState::SetMemLimitExceeded()
@ 0xb6bd6d impala::FunctionContextImpl::CheckAllocResult()
@ 0xb6ae78 impala::FunctionContextImpl::AllocateLocal()
@ 0xb6b10f impala_udf::StringVal::StringVal()
@ 0xb6b16a impala_udf::StringVal::CopyFrom()
@ 0x8a2641 impala::AggregateFunctions::StringValGetValue()
@ 0x8a2661
impala::AggregateFunctions::StringValSerializeOrFinalize()
@ 0xd411c5 impala::AggFnEvaluator::SerializeOrFinalize()
@ 0xce6569 impala::PartitionedAggregationNode::CleanupHashTbl()
@ 0xce693b impala::PartitionedAggregationNode::Partition::Close()
@ 0xce8152 impala::PartitionedAggregationNode::ClosePartitions()
@ 0xcf163e impala::PartitionedAggregationNode::Close()
@ 0xa7ae35 impala::FragmentInstanceState::Close()
@ 0xa7ebfb impala::FragmentInstanceState::Exec()
@ 0xa6aaf6 impala::QueryState::ExecFInstance()
@ 0xbef9d9 impala::Thread::SuperviseThread()
@ 0xbf0394 boost::detail::thread_data<>::run()
@ 0xe588aa (unknown)
@ 0x7f8a54c32e25 start_thread
@ 0x7f8a5496034d __clone
{noformat}
Here't the profile summary:
{noformat}
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est.
Peak Mem Detail
-----------------------------------------------------------------------------------------------------------------------
03:AGGREGATE 32 0.000ns 0.000ns 0 1.06B 2.75 GB
165.23 GB FINALIZE
02:EXCHANGE 32 1s246ms 7s429ms 412.85M 1.06B 0
0 HASH(<redacted>)
01:AGGREGATE 32 22s539ms 27s631ms 422.16M 1.06B 3.09 GB
165.23 GB STREAMING
00:SCAN HDFS 32 1s221ms 2s093ms 810.12M 1.06B 222.22 MB
616.00 MB <redacted>
Errors: Memory limit exceeded: Error occurred on backend
<redacted>.com:22000 by fragment b429287bdabe184:9e7b5e960000002c
Memory left in process limit: 386.76 GB
Memory left in query limit: -19080.00 B
{noformat}
And the query string:
{noformat}
create TABLE TT><redacted>
STORED AS PARQUET
TBLPROPERTIES('parquet.compression'='SNAPPY')
T>AS
T>SELECT T> <redacted>,
<redacted>,
<redacted>,
MAX(<redacted>) AS <redacted>,
MAX(<redacted>) AS <redacted>,
MAX(<redacted>) AS <redacted>,
MAX(<redacted>) AS <redacted>
T>FROM TT> <redacted>
GROUP BY <redacted>,
<redacted>,
<redacted>
Coordinator: <redacted>.com:22000
Query Options (non default):
MEM_LIMIT=6442450944,REQUEST_POOL=<redacted>,SYNC_DDL=1,MT_DOP=0
DDL Type: CREATE_TABLE_AS_SELECT
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
