Joe McDonnell created IMPALA-13479:
--------------------------------------
Summary: Patch gperftools to allow max_total_thread_cache_bytes to
exceed 1GB
Key: IMPALA-13479
URL: https://issues.apache.org/jira/browse/IMPALA-13479
Project: IMPALA
Issue Type: Improvement
Components: Infrastructure
Affects Versions: Impala 4.5.0
Reporter: Joe McDonnell
gperftools limits max_total_thread_cache_bytes to 1GB here:
[https://github.com/gperftools/gperftools/blob/gperftools-2.10/src/thread_cache.cc#L520-L523]
{noformat}
void ThreadCache::set_overall_thread_cache_size(size_t new_size) {
// Clip the value to a reasonable range
if (new_size < kMinThreadCacheSize) new_size = kMinThreadCacheSize;
if (new_size > (1<<30)) new_size = (1<<30); // Limit to 1GB{noformat}
I confirmed that setting --tcmalloc_max_total_thread_cache_bytes=2147483648
still results in a 1GB limit.
Sometimes, we would want a higher limit for systems with a large amount of
memory and CPUs. For example, some systems now have 1TB of memory and 96 CPUs.
With high concurrency, there is high contention on tcmalloc locks on central
data structures. Increasing the total thread cache size could avoid this, and a
value higher than 1GB is still a small part of system memory.
We can patch our toolchain gperftools to allow a higher value (and notify
gperftools community).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
