Andrei Sekretenko created MESOS-10160:
-----------------------------------------
Summary: Split source files that require the larges amount of
memory for compilation.
Key: MESOS-10160
URL: https://issues.apache.org/jira/browse/MESOS-10160
Project: Mesos
Issue Type: Improvement
Reporter: Andrei Sekretenko
Mesos build is known to be memory-consuming and under-utilizing CPUs when built
on hosts with small amount of memory.
What is less known is the fact that the peak RSS of invocations of
compiler/linker seems to follow Paretolike distribution.
Top-20 clang++/clang invocations by max RSS in CMake RelWithDebInfo build of
mesos-tests on Linux:
{noformat}
4.43 GiB <linking mesos-tests> // almost never built in parallel with other
sources
3.99 GiB mesos/src/csi/v1_volume_manager.cpp
3.97 GiB mesos/src/csi/v0_volume_manager.cpp
3.74 GiB <linking libmesos>
3.25 GiB mesos/src/tests/slave_tests.cpp
3.00 GiB mesos/src/tests/api_tests.cpp
2.99 GiB mesos/src/tests/master_tests.cpp
2.72 GiB mesos/src/tests/slave_recovery_tests.cpp
2.56 GiB mesos/src/tests/storage_local_resource_provider_tests.cpp
2.51 GiB mesos/src/slave/slave.cpp
2.21 GiB mesos/src/slave/http.cpp
2.12 GiB mesos/src/slave/containerizer/mesos/containerizer.cpp
2.04 GiB mesos/src/tests/master_slave_reconciliation_tests.cpp
2.02 GiB mesos/src/slave/containerizer/docker.cpp
1.96 GiB mesos/src/master/master.cpp
1.90 GiB mesos/src/tests/reservation_tests.cpp
1.88 GiB mesos/src/tests/csi_client_tests.cpp
1.87 GiB mesos/src/tests/master_authorization_tests.cpp
1.83 GiB mesos/src/tests/containerizer/docker_containerizer_tests.cpp
1.82 GiB mesos/src/tests/master_allocator_tests.cpp
1.77 GiB mesos/src/tests/fault_tolerance_tests.cpp
1.75 GiB mesos/src/tests/partition_tests.cpp
1.72 GiB mesos/src/tests/operation_reconciliation_tests.cpp
1.71 GiB mesos/src/tests/master_draining_tests.cpp
1.70 GiB mesos/src/tests/resource_provider_manager_tests.cpp
1.69 GiB mesos/src/tests/containerizer/cni_isolator_tests.cpp
1.66 GiB mesos/src/resource_provider/storage/provider.cpp
1.65 GiB mesos/src/tests/master_quota_tests.cpp
1.62 GiB mesos/src/master/http.cpp
1.59 GiB mesos/src/tests/persistent_volume_tests.cpp
1.59 GiB mesos/src/tests/scheduler_tests.cpp
1.55 GiB mesos/src/tests/master_maintenance_tests.cpp
1.54 GiB mesos/src/tests/hierarchical_allocator_tests.cpp
1.54 GiB mesos/src/tests/gc_tests.cpp
1.53 GiB mesos/src/slave/containerizer/mesos/io/switchboard.cpp
1.53 GiB mesos/src/tests/master_validation_tests.cpp
1.53 GiB mesos/src/tests/log_tests.cpp
1.53 GiB mesos/src/tests/oversubscription_tests.cpp
1.50 GiB mesos/src/tests/slave_authorization_tests.cpp
1.49 GiB mesos/src/tests/resource_offers_tests.cpp
{noformat}
The presence of a few files of 3+ GiB is what makes clean build on standard AWS
instances (and typical laptops) memory-limited if one tries to avoid compiler
OOM.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
