[
https://issues.apache.org/jira/browse/IMPALA-4835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583081#comment-16583081
]
ASF subversion and git services commented on IMPALA-4835:
---------------------------------------------------------
Commit 7ccf7369085aa49a8fc0daf6f91d97b8a3135682 in impala's branch
refs/heads/master from [[email protected]]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=7ccf736 ]
IMPALA-7096: restore scanner thread memory heuristics
This restores some of the heuristics removed in IMPALA-4835 that can
help scans from hitting OOM conditions. The heuristics are implemented
at the query level rather than in each scan node in isolation.
Introduce a ScannerMemLimiter class that belongs to the QueryState that
tracks the amount of memory estimated to be consumed for all scanner
threads running for the query on the current backend.
Also check soft memory limits to see if scanner threads should be
started or the current scanner thread should stop.
The long-term plan is to switch to the MT scan node implementations.
When that happens this code can be removed. In the meantime this
code is imperfect but will help avoid OOM in many scenarios.
Testing:
Added regression tests for HDFS and Kudu where we previously could
run out of memory with a low mem_limit.
Manual testing:
* Ran query tests with --thread_creation_fault_injection=true for a
bit, confirmed no crashes.
* ran single-node stress test for Kudu and Parquet for 10-20 min each.
Change-Id: Ib9907fa8c4d2b0b85f67f4f160899c1c258ad82b
Reviewed-on: http://gerrit.cloudera.org:8080/11103
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> HDFS scans should operate with a constrained number of I/O buffers
> ------------------------------------------------------------------
>
> Key: IMPALA-4835
> URL: https://issues.apache.org/jira/browse/IMPALA-4835
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Affects Versions: Impala 2.9.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Major
> Labels: resource-management
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> The HDFS scan nodes should be able to operate with a fixed number of I/O
> buffers in most cases (excluding very large rows). We should modify them to
> claim a reservation upfront and use this for all disk I/O.
> This probably also requires switching DiskIoMgr to allocate memory from
> BufferPool.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
