[
https://issues.apache.org/jira/browse/IMPALA-11805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795485#comment-17795485
]
ASF subversion and git services commented on IMPALA-11805:
----------------------------------------------------------
Commit 8874ea07b68c09cee3c4e300b70706f3c5ea40af in impala's branch
refs/heads/master from Michael Smith
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=8874ea07b ]
IMPALA-11805: Fix LLVM memory manager bytes allocated
Previously the bytes_allocated function would undercount memory
allocated by the LLVM memory manager, as it allocates memory in full
pages but bytes_allocated only tracked the actual storage requested.
Implements bytes_allocated in the SectionMemoryManager to access the
actual storage size allocated.
Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Reviewed-on: http://gerrit.cloudera.org:8080/20697
Reviewed-by: Yida Wu <[email protected]>
Reviewed-by: Csaba Ringhofer <[email protected]>
Tested-by: Michael Smith <[email protected]>
> Codegen cache size estimation is less than the actual allocation
> ----------------------------------------------------------------
>
> Key: IMPALA-11805
> URL: https://issues.apache.org/jira/browse/IMPALA-11805
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 4.3.0
> Reporter: Yida Wu
> Assignee: Yida Wu
> Priority: Major
>
> In IMPALA-11470, we implement the cache for codegen functions, however, the
> expected size of a cache entry is much less than the actual allocation,
> according to the data in tcmalloc memory tracker. This could lead to a result
> of unexpected query failure when the memory tracker hits the capacity.
> The current way to estimate the memory consumption of a codegen cache entry,
> mainly the memory consumption of a llvm::ExecutionEngine that stored in each
> entry, is to use the customized ImpalaMCJITMemoryManager
> [https://github.com/apache/impala/blob/f705496e34ac474e8e1c999619e3b928c5e39e0f/be/src/codegen/mcjit-mem-mgr.h#L60],
> to accumulated bytes when the execution engine allocates code or data
> section. However in fact, the actual bytes allocated by the execution engine
> could be much larger.
> Tested in tpch and tpcds queries, in normal mode, the final consumption could
> be 3~4 times of the estimation, and it would be worse in the optimal mode,
> because the main difference is between the memory_manager_->bytes_allocated()
> and the actual execution engine allocation, and in normal mode it contains
> the size of the key, which is accurate.
> When the execution engine is only existing a short period in runtime, the
> issue isn't that bad. However, when it becomes a part of the long-living
> cache entry, it could cause more problems by consuming much more memory than
> estimation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
