[
https://issues.apache.org/jira/browse/IMPALA-14900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18091052#comment-18091052
]
ASF subversion and git services commented on IMPALA-14900:
----------------------------------------------------------
Commit 048b951f9dcc5cf646773d5f52f2d77c5e497096 in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=048b951f9 ]
IMPALA-14900: Add support for turning off aggressive decommit
Impala has used TCMalloc's aggressive decommit setting
for several years, but it increases the OS allocation /
deallocation rate and can lead to contention on TCMalloc's
central structures. There are many pieces of code that
still rely on malloc for their memory, including
performance sensitive pieces of query execution. Retaining
some malloc memory can accelerate those codepaths by
avoiding OS allocation / deallocation cycles. TCMalloc
holds a lock while allocating and deallocating memory,
and retaining memory can also avoid extreme cases with
high lock contention. For example, in IMPALA-13966, we
saw performance issues using 1MB Parquet data pages,
because the large allocations bypass the thread caches
and come directly from the central structures.
There is a long history behind this setting:
- As of late 2015 / Impala 2.3, Impala would let tcmalloc
retain memory. It had two mechanisms for releasing
memory. The first was a periodic check to see if the
overhead of tcmalloc exceeded the memory used. The
second was a garbage collection function that ran
when hitting the process memory limit. Both mechanisms
would free ALL excess tcmalloc heap memory via a single
call to ReleaseFreeMemory(). TCMalloc is holding a lock
for this call, and this can stall other work until it
completes. It could be freeing dozens of GBs and this
could hold the lock for 15 seconds. This issue was
reported via IMPALA-2800.
- In IMPALA-3162, Impala moved to gperftools 2.5, which
had aggressive decommit enabled by default. This frees
memory immediately, so the mechanisms to free memory
had nothing to do. This solved IMPALA-2800. The obsolete
code for the periodic check and garbage collection
function were removed in IMPALA-5220.
- Gperftools only had aggressive decommit enabled by
default for a short period of time. It was enabled by
default in 2.4 and was disabled by default in 2.6.
- When Impala upgraded gperftools later, we added code
to manually set aggressive decommit.
This adds back an option to turn off aggressive decommit.
The shape is similar to the old mechanisms: there is a
background thread doing a periodic check to manage the
memory overhead and a garbage collection function that
gets called when hitting the process memory limit. This
has been redesigned to avoid the issue from IMPALA-2800
(based on an early approach to IMPALA-2800 by Todd Lipcon):
- Both enforcement locations are freeing a specific amount
of memory rather than all accumulated memory (i.e. it
calls ReleaseToSystem() with a target amount of memory
to free). The background thread is maintaining an overhead
specified by the tcmalloc_max_free_bytes startup option.
This can be an absolute value or a percentage of the
process memory limit. It defaults to 5% of the process
memory limit. The garbage collection function is
freeing enough memory to avoid hitting the process
memory limit, plus a bit extra (512MB) to avoid calling
the GC function too frequently.
- Both enforcement locations free memory in small chunks
to avoid holding the lock for extended periods of time.
The chunk size is specified by the tcmalloc_garbage_collection_chunk_size
startup option and defaults to 10MB.
- The implementation retains significantly less memory and
frees it without holding the lock for extended periods of
time.
- Other things have changed since then: The buffer pool
retains memory and frees it gradually over time. This also
reduces the need for freeing a large amount of memory
immediately.
Turning off aggressive decommit is currently incompatible with
the madvise_huge_pages=true startup option. This modifies the
startup check so that aggressive decommit can be false if
madvise_huge_pages is false. A future change may provide a
way to mmap huge buffers to allow these to work together.
This adds the --tcmalloc_aggressive_decommit option to
bin/start-impala-cluster.py to make it easier to startup
the cluster. The default value is determined by the
IMPALA_TCMALLOC_AGGRESSIVE_DECOMMIT environment variable,
so this makes it possible to run cluster tests with this
option.
Testing:
- Added a custom cluster test to run TPC-DS with tcmalloc
aggressive decommit off
- Ran a core job with IMPALA_TCMALLOC_AGGRESSIVE_DECOMMIT=false
- Ran the scenario from IMPALA-13966 and verified that turning
off aggressive decommit avoids the issues.
Change-Id: If6022f14093f362a5de9a854f4f4496c90b049b8
Reviewed-on: http://gerrit.cloudera.org:8080/24402
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Add support for running Impala without tcmalloc's aggressive decommit mode
> --------------------------------------------------------------------------
>
> Key: IMPALA-14900
> URL: https://issues.apache.org/jira/browse/IMPALA-14900
> Project: IMPALA
> Issue Type: Task
> Components: Backend
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
>
> Impala currently sets tcmalloc's aggressive decommit mode and asserts that it
> is set. This forces tcmalloc to return all free spans to the operating
> system. Impala does this for a couple reasons:
> # Without aggressive decommit, tcmalloc doesn't release memory fast enough
> for Impala's memory access patterns. Impala would need to manually force
> tcmalloc to free memory with a background thread and integration with the
> process memory tracker's garbage collection. A long time ago, there were
> performance issues with the way this was implemented. See IMPALA-2800.
> # When Impala uses huge pages, we don't want tcmalloc to reuse the huge page
> as smaller pieces. This can potentially cause inaccuracy in our memory use
> statistics. See this comment in system-allocator.cc:
> https://github.com/apache/impala/blob/cefeb760a886d28c05f98917cdfeac864728f270/be/src/runtime/bufferpool/system-allocator.cc#L152-L157
>
> However, aggressive decommit increases the memory allocation / deallocation
> rate and can lead to extra interactions with the operating system and
> contention on tcmalloc locks. We have seen issues when Parquet files have
> large data pages. We have also seen issues with queries operating on nested
> data.
> It would be useful to have a way to run Impala with aggressive decommit
> disabled. This will require a background thread monitoring memory use and
> integration with the process mem tracker's garbage collection. One option is
> to allow aggressive decommit to be disabled when huge pages are disabled.
> Another option is to use mmap for huge pages to allow using huge pages with
> aggressive decommit disabled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
