Joe McDonnell created IMPALA-13147:
--------------------------------------
Summary: Add support for limiting the concurrency of link jobs
Key: IMPALA-13147
URL: https://issues.apache.org/jira/browse/IMPALA-13147
Project: IMPALA
Issue Type: Improvement
Components: Infrastructure
Affects Versions: Impala 4.5.0
Reporter: Joe McDonnell
Link jobs can use a lot of memory due to the amount of debug info. The level of
concurrency that is useful for compilation can be too high for linking. Running
a link-heavy command like buildall.sh -skiptests can run out of memory from
linking all of the backend tests / benchmarks.
It would be useful to be able to limit the number of concurrent link jobs.
There are two basic approaches:
When using the ninja generator for CMake, ninja supports having job pools with
limited parallelism. CMake has support for mapping link tasks to their own
pool. Here is an example:
{noformat}
set(CMAKE_JOB_POOLS compilation_pool=24 link_pool=8)
set(CMAKE_JOB_POOL_COMPILE compilation_pool)
set(CMAKE_JOB_POOL_LINK link_pool){noformat}
The makefile generator does not have equivalent functionality, but we could do
a more limited version where buildall.sh can split the -skiptests into two make
invocations. The first does all the compilation with full parallelism
(equivalent to -notests) and then the second make invocation does the backend
tests / benchmarks with a reduced parallelism.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
