[
https://issues.apache.org/jira/browse/IMPALA-14900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell resolved IMPALA-14900.
------------------------------------
Fix Version/s: Impala 5.0.0
Resolution: Fixed
> Add support for running Impala without tcmalloc's aggressive decommit mode
> --------------------------------------------------------------------------
>
> Key: IMPALA-14900
> URL: https://issues.apache.org/jira/browse/IMPALA-14900
> Project: IMPALA
> Issue Type: Task
> Components: Backend
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
> Fix For: Impala 5.0.0
>
>
> Impala currently sets tcmalloc's aggressive decommit mode and asserts that it
> is set. This forces tcmalloc to return all free spans to the operating
> system. Impala does this for a couple reasons:
> # Without aggressive decommit, tcmalloc doesn't release memory fast enough
> for Impala's memory access patterns. Impala would need to manually force
> tcmalloc to free memory with a background thread and integration with the
> process memory tracker's garbage collection. A long time ago, there were
> performance issues with the way this was implemented. See IMPALA-2800.
> # When Impala uses huge pages, we don't want tcmalloc to reuse the huge page
> as smaller pieces. This can potentially cause inaccuracy in our memory use
> statistics. See this comment in system-allocator.cc:
> https://github.com/apache/impala/blob/cefeb760a886d28c05f98917cdfeac864728f270/be/src/runtime/bufferpool/system-allocator.cc#L152-L157
>
> However, aggressive decommit increases the memory allocation / deallocation
> rate and can lead to extra interactions with the operating system and
> contention on tcmalloc locks. We have seen issues when Parquet files have
> large data pages. We have also seen issues with queries operating on nested
> data.
> It would be useful to have a way to run Impala with aggressive decommit
> disabled. This will require a background thread monitoring memory use and
> integration with the process mem tracker's garbage collection. One option is
> to allow aggressive decommit to be disabled when huge pages are disabled.
> Another option is to use mmap for huge pages to allow using huge pages with
> aggressive decommit disabled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
