Andrei Sekretenko created MESOS-10122:
-----------------------------------------
Summary: cmake+MSBuild is uncapable of building all Mesos sources
in parallel.
Key: MESOS-10122
URL: https://issues.apache.org/jira/browse/MESOS-10122
Project: Mesos
Issue Type: Bug
Components: build
Environment: When a library (in cmake's sense) contains several
sources with different paths but the same filename (for example,
slave/validation.cpp and resource_provider/validation.cpp), the build generated
by CMake for MSVC does not allow for building those files in parallel
(presumably, because the .obj files will be located in the same directory).
This has been tested observed with both cmake 3.9 and 3.17, with "Visual Studio
15 2017 Win64" generator.
It seems to be a known behaviour - see
https://stackoverflow.com/questions/7033855/msvc10-mp-builds-not-multicore-across-folders-in-a-project.
Two options for fixing this in a way that will work with these cmake/MSVC
configurations are:
- splitting the build into small static libraries (a library per directory)
- introducing an intermediate code-generation-like step optionally flattening
the directory structure (slave/validation.cpp -> slave_validation.cpp)
Both options have their drawbacks:
- The first will result in changing the layout the static build artifacts
(mesos.lib will be replaced with a ton of smaller libraries), that will pose
integration cahllenges, and potentially will result in worse parallelism.
- The second will result in being unable to use #include without a path
(right now there are three or four such #include's in the whole
Mesos/libprocess code buildable on Windows) in changed value of __FILE__ macro
(as a consequence, in the example above, `validation.cpp` in logs will be
replaced either with `slave_validation.cpp` or with
`resource_provider_validation.cpp`)
Note that the second approach will need to deal with potential collisions when
the source tree has filenames with underscores. If, for example, we had both
slave/validation.cpp and slave_validation.cpp, then either some additional
escaping will be needed when or, alternatively, such layout could be just
forbidden (and made to fail the build).
Preliminary testing shows that on a 8-core AWS instance flattening source trees
of libprocess, mesos-protobufs and libmesos results in clean build speedup from
54 minutes to 33 minutes.
Reporter: Andrei Sekretenko
Assignee: Andrei Sekretenko
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
