Yida Wu created IMPALA-14316:
--------------------------------
Summary: Large batch insertion can lead to high memory usage and
backpressure
Key: IMPALA-14316
URL: https://issues.apache.org/jira/browse/IMPALA-14316
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 4.5.0
Reporter: Yida Wu
Impalad could experience rapid and significant memory growth when INSERT
statements with a large number of VALUES clauses (e.g., 10000). The memory is
not released after the query completes, and end up with backpressure errors
while causing impalad to become unresponsive to new queries. This behavior is
not observed with smaller batch sizes (e.g., 100).
{code:java}
Exec() rpc failed: Remote error: Service unavailable: ExecQueryFInstances
request on impala.ControlService from 127.0.0.1:58626 dropped due to
backpressure. The service queue contains 0 items out of a maximum of
2147483647; memory consumption is 0.
{code}
pprof shows that the massive allocation is caused by TUnionNode and TExpr
objects, it is possible that the expression vectors in TUnionNode not being
released promptly and needs further investigation into the cause.
{code:java}
10000 batch pprof growth:
Total: 5847.7 MB
4989.0 85.3% 85.3% 5103.0 87.3% std::vector::operator= (inline)
527.1 9.0% 94.3% 527.1 9.0% JVM_FindSignal
114.0 1.9% 96.3% 5103.0 87.3% std::vector::vector (inline)
0.0 0.0% 100.0% 5504.7 94.1%
boost::detail::function::void_function_obj_invoker0::invoke
0.0 0.0% 100.0% 5504.7 94.1%
boost::detail::function::void_function_obj_invoker0::invoke (inline)
0.0 0.0% 100.0% 5512.7 94.3% boost::detail::thread_data::run
0.0 0.0% 100.0% 5512.7 94.3% boost::detail::thread_data::run (inline)
...
0.0 0.0% 100.0% 478.7 8.2% impala::DeserializeThriftMsg
...
0.0 0.0% 100.0% 4849.0 82.9% impala::QueryExecMgr::ExecuteQueryHelper
0.0 0.0% 100.0% 4849.0 82.9% impala::QueryState::StartFInstances
....
0.0 0.0% 100.0% 4989.0 85.3% impala::TExpr::TExpr
0.0 0.0% 100.0% 94.0 1.6% impala::TExpr::read
0.0 0.0% 100.0% 90.0 1.5% impala::TExpr::read (inline)
0.0 0.0% 100.0% 287.0 4.9% impala::TExprNode::TExprNode
0.0 0.0% 100.0% 4.0 0.1% impala::TExprNode::read
0.0 0.0% 100.0% 447.0 7.6% impala::TPlan::operator=
0.0 0.0% 100.0% 95.0 1.6% impala::TPlan::read
0.0 0.0% 100.0% 88.0 1.5% impala::TPlanExecInfo::read
0.0 0.0% 100.0% 447.0 7.6% impala::TPlanFragment::TPlanFragment
0.0 0.0% 100.0% 95.0 1.6% impala::TPlanFragment::read
0.0 0.0% 100.0% 5103.0 87.3% impala::TPlanNode::TPlanNode
0.0 0.0% 100.0% 95.0 1.6% impala::TPlanNode::read
0.0 0.0% 100.0% 88.0 1.5% impala::TQueryExecRequest::read
0.0 0.0% 100.0% 5103.0 87.3% impala::TUnionNode::operator=
0.0 0.0% 100.0% 95.0 1.6% impala::TUnionNode::read
0.0 0.0% 100.0% 1.0 0.0% impala::TUnionNode::read (inline)
...
100 batch pprof growth:
Total: 402.0 MB
227.5 56.6% 56.6% 227.5 56.6% JVM_FindSignal
...
9.0 2.2% 91.5% 10.0 2.5% std::vector::operator= (inline)
2.0 0.5% 97.0% 2.0 0.5% std::vector::_M_default_append (inline)
1.0 0.2% 97.3% 2.0 0.5% Java_java_lang_ClassLoader_defineClass1
...
0.0 0.0% 100.0% 23.0 5.7%
impala::ClientRequestState::FinishExecQueryOrDmlRequest
0.0 0.0% 100.0% 2.0 0.5%
impala::ControlService::ExecQueryFInstances
...
0.0 0.0% 100.0% 23.0 5.7% impala::Coordinator::Exec
0.0 0.0% 100.0% 22.0 5.5% impala::Coordinator::FinishBackendStartup
0.0 0.0% 100.0% 1.0 0.2% impala::Coordinator::StartBackendExec
0.0 0.0% 100.0% 2.0 0.5% impala::DeserializeThriftMsg
...
0.0 0.0% 100.0% 9.0 2.2% impala::TExpr::TExpr
0.0 0.0% 100.0% 2.0 0.5% impala::TExpr::read
0.0 0.0% 100.0% 2.0 0.5% impala::TExpr::read (inline)
0.0 0.0% 100.0% 1.0 0.2% impala::TExprNode::TExprNode
0.0 0.0% 100.0% 2.0 0.5% impala::TPlan::operator=
0.0 0.0% 100.0% 2.0 0.5% impala::TPlan::read
0.0 0.0% 100.0% 2.0 0.5% impala::TPlanFragment::TPlanFragment
0.0 0.0% 100.0% 2.0 0.5% impala::TPlanFragment::read
0.0 0.0% 100.0% 10.0 2.5% impala::TPlanNode::TPlanNode
0.0 0.0% 100.0% 2.0 0.5% impala::TPlanNode::read
0.0 0.0% 100.0% 10.0 2.5% impala::TUnionNode::operator=
0.0 0.0% 100.0% 2.0 0.5% impala::TUnionNode::read
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
