[
https://issues.apache.org/jira/browse/IMPALA-13479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17892640#comment-17892640
]
Joe McDonnell commented on IMPALA-13479:
----------------------------------------
The 1GB limit is very old (~15 years), so I'm guessing it made sense at the
time and gradually stopped making sense.
> Patch gperftools to allow max_total_thread_cache_bytes to exceed 1GB
> --------------------------------------------------------------------
>
> Key: IMPALA-13479
> URL: https://issues.apache.org/jira/browse/IMPALA-13479
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
> Affects Versions: Impala 4.5.0
> Reporter: Joe McDonnell
> Priority: Major
>
> gperftools limits max_total_thread_cache_bytes to 1GB here:
> [https://github.com/gperftools/gperftools/blob/gperftools-2.10/src/thread_cache.cc#L520-L523]
> {noformat}
> void ThreadCache::set_overall_thread_cache_size(size_t new_size) {
> // Clip the value to a reasonable range
> if (new_size < kMinThreadCacheSize) new_size = kMinThreadCacheSize;
> if (new_size > (1<<30)) new_size = (1<<30); // Limit to 1GB{noformat}
> I confirmed that setting --tcmalloc_max_total_thread_cache_bytes=2147483648
> still results in a 1GB limit.
> Sometimes, we would want a higher limit for systems with a large amount of
> memory and CPUs. For example, some systems now have 1TB of memory and 96
> CPUs. With high concurrency, there is high contention on tcmalloc locks on
> central data structures. Increasing the total thread cache size could avoid
> this, and a value higher than 1GB is still a small part of system memory.
> We can patch our toolchain gperftools to allow a higher value (and notify
> gperftools community).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
