Michael Smith created IMPALA-12939:
--------------------------------------
Summary: Improve default for IMPALA_BUILD_THREADS
Key: IMPALA-12939
URL: https://issues.apache.org/jira/browse/IMPALA-12939
Project: IMPALA
Issue Type: Task
Components: Infrastructure
Reporter: Michael Smith
Improve the default selection for IMPALA_BUILD_THREADS and other
parallelization config.
Impala's build process needs 2GB of memory per CPU core with {{-notests}} or
4GB of memory per CPU core when building unit test binaries (the link process
is especially memory-intensive). Exceeding these can lead to systems with many
cores running out of memory during the build. We currently do not consider
memory when selecting a default value for IMPALA_BUILD_THREADS.
We currently default IMPALA_BUILD_THREADS to {{nproc}}, which also does not
reflect CPU slicing that may happen in containers. How to detect this differs
between cgroups v1 and v2 (v2 is becoming more common, present on Ubuntu 22 and
RedHat 9).
* v1: {code}$(cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us) /
$(/sys/fs/cgroup/cpu/cpu.cfs_period_us){code}
* v2: {code}awk '{ cores=$1/$2; print cores==int(cores) ? cores : int(cores)+1
}' /sys/fs/cgroup/cpu.max{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
