Re: [cmake-developers] CMake generates Makefiles that don't parallelize as much as they could.
Here's the sample code: https://github.com/NickNick/cmake-interface-includes/commits/master . The second commit breaks the build. I sort of understand why it does, but that means I can't use the INTERFACE-trick, so to say, even with static libraries. Thanks for linking the other bug. On Mon, Jul 28, 2014 at 4:19 PM, Brad King brad.k...@kitware.com wrote: On 07/24/2014 05:47 PM, Nick Overdijk wrote: I'm using target_include_directories of A to get some include directories into B well, so I can't use target_link_libraries(A INTERFACE B), Can you clarify this with sample code please? and I can't seem to use the OBJECT-way neither since B's sources won't compile without A's INTERFACE_INCLUDE_DIRECTORIES... Usage requirements were designed with target_link_libraries as the main way to propagate them, but OBJECT libraries do not allow use with target_link_libraries. This is an open problem with the design of the two features. See also discussion here: http://www.cmake.org/Bug/view.php?id=14970 -Brad -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] CMake generates Makefiles that don't parallelize as much as they could.
On 07/30/2014 05:31 AM, Nick Overdijk wrote: https://github.com/NickNick/cmake-interface-includes/commits/master Thanks. For reference, the summary is: cmake_minimum_required(VERSION 2.8.12) project(FOO CXX) add_library(foo foo/foo.cpp) target_include_directories(foo INTERFACE foo) add_library(bar bar/bar.cpp) target_link_libraries(bar foo) # reduces parallel compilation #target_link_libraries(bar INTERFACE foo) # compiles bar without foo reqs With CMake 3.0 you might be able to hack something up with INTERFACE libraries, but that will increase complexity. The INTERFACE workaround commented out in the above example is really only good when no usage requirements need to be propagated. Otherwise, I think you're stuck with the reduced parallelism until CMake can be taught to detect when it is safe to drop such dependencies as I explained earlier. FWIW, I've been building large projects this way for years and rarely been bothered by this. Usually each library has so many sources that parallelism within each target is enough. -Brad -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] CMake generates Makefiles that don't parallelize as much as they could.
True, I was just bothered by it when we got distcc running with a good amount of cores. Thanks for your effort. :) I'll try and make something with the things I've learned (I'm thinking of making some functions that propagate usage requirements yet link with INTERFACE or something along those lines, of course only usable for static libs). On Wed, Jul 30, 2014 at 4:51 PM, Brad King brad.k...@kitware.com wrote: On 07/30/2014 05:31 AM, Nick Overdijk wrote: https://github.com/NickNick/cmake-interface-includes/commits/master Thanks. For reference, the summary is: cmake_minimum_required(VERSION 2.8.12) project(FOO CXX) add_library(foo foo/foo.cpp) target_include_directories(foo INTERFACE foo) add_library(bar bar/bar.cpp) target_link_libraries(bar foo) # reduces parallel compilation #target_link_libraries(bar INTERFACE foo) # compiles bar without foo reqs With CMake 3.0 you might be able to hack something up with INTERFACE libraries, but that will increase complexity. The INTERFACE workaround commented out in the above example is really only good when no usage requirements need to be propagated. Otherwise, I think you're stuck with the reduced parallelism until CMake can be taught to detect when it is safe to drop such dependencies as I explained earlier. FWIW, I've been building large projects this way for years and rarely been bothered by this. Usually each library has so many sources that parallelism within each target is enough. -Brad -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] CMake generates Makefiles that don't parallelize as much as they could.
On 07/24/2014 05:47 PM, Nick Overdijk wrote: I'm using target_include_directories of A to get some include directories into B well, so I can't use target_link_libraries(A INTERFACE B), Can you clarify this with sample code please? and I can't seem to use the OBJECT-way neither since B's sources won't compile without A's INTERFACE_INCLUDE_DIRECTORIES... Usage requirements were designed with target_link_libraries as the main way to propagate them, but OBJECT libraries do not allow use with target_link_libraries. This is an open problem with the design of the two features. See also discussion here: http://www.cmake.org/Bug/view.php?id=14970 -Brad -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] CMake generates Makefiles that don't parallelize as much as they could.
I'm using target_include_directories of A to get some include directories
into B well, so I can't use target_link_libraries(A INTERFACE B), and I
can't seem to use the OBJECT-way neither since B's sources won't compile
without A's INTERFACE_INCLUDE_DIRECTORIES... Any suggestions?
On Wed, Jul 23, 2014 at 3:19 PM, Nick Overdijk n...@astrant.net wrote:
Crystal clear. Another layer of indirection eh? I'll see if I can work
with that... Thanks for the explanation.
On Wed, Jul 23, 2014 at 3:14 PM, Brad King brad.k...@kitware.com wrote:
On 07/23/2014 09:07 AM, Nick Overdijk wrote:
Oh wait, since a is in the interface of b, b will always be
accompanied by a, even though it's not a dependency.
Is that how it works?
Yes. If B is a static library then it does not really link so
its dependencies are only ever used transitively when something
else links to B. If B really links as a shared library though
then see the following.
If target B links to target A then CMake will not compile objects
in B until A is done. As explained in my previous link this is
because we allow A to contain custom commands that generate
headers used by B. The VS and Xcode IDE build systems offer only
this granularity since they organize compilation and linking rules
of a single target together. The Makefile generator was designed
this way too. Only the Ninja generator has a chance of increasing
parallelism if the extra ordering dependencies were dropped. We've
thought about how to do extra analysis to determine when there is
no such custom command to drop them but have not implemented anything
yet.
You might be able to use OBJECT libraries to increase parallelism
for all generators and with no changes to CMake:
set(CMAKE_POSITION_INDEPENDENT_CODE ON)
add_library(Aobjs OBJECT a1.c a2.c)
add_library(Bobjs OBJECT b1.c b2.c)
add_library(Cobjs OBJECT c1.c c2.c)
set(dummy_c dummy.c) # needed for VS 6 and Xcode
add_library(A SHARED $TARGET_OBJECTS:Aobjs ${dummy_c})
add_library(B SHARED $TARGET_OBJECTS:Bobjs ${dummy_c})
add_library(C SHARED $TARGET_OBJECTS:Cobjs ${dummy_c})
target_link_libraries(B PUBLIC A)
target_link_libraries(C PUBLIC B)
This way the object compilations will be completely independent
of one another and of the linking and ordering dependencies.
-Brad
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers
[cmake-developers] CMake generates Makefiles that don't parallelize as much as they could.
Hey guys, With this https://github.com/NickNick/cmake-dependency-waiting code here, why do b wait for a and c wait for b to be build? The object files could all be build in parallel right? Not doing it is making my distcc-cluster less and less useful the more nodes I add. Is there a way to fix or work around this? Cheers, -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] CMake generates Makefiles that don't parallelize as much as they could.
On 07/23/2014 08:00 AM, Nick Overdijk wrote: With this https://github.com/NickNick/cmake-dependency-waiting code here, why do b wait for a and c wait for b to be build? The object files could all be build in parallel right? Not doing it is making my distcc-cluster less and less useful the more nodes I add. Is there a way to fix or work around this? See here: http://www.cmake.org/Bug/view.php?id=14726#c35021 http://www.cmake.org/Bug/view.php?id=14726#c35023 -Brad -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] CMake generates Makefiles that don't parallelize as much as they could.
Thanks for the quick reply, but what if c needs b and b needs a? Adding INTERFACE will then break the build of course, right, since b isn't really linked to a... Or am I mistaken? On Wed, Jul 23, 2014 at 2:36 PM, Brad King brad.k...@kitware.com wrote: On 07/23/2014 08:00 AM, Nick Overdijk wrote: With this https://github.com/NickNick/cmake-dependency-waiting code here, why do b wait for a and c wait for b to be build? The object files could all be build in parallel right? Not doing it is making my distcc-cluster less and less useful the more nodes I add. Is there a way to fix or work around this? See here: http://www.cmake.org/Bug/view.php?id=14726#c35021 http://www.cmake.org/Bug/view.php?id=14726#c35023 -Brad -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] CMake generates Makefiles that don't parallelize as much as they could.
Oh wait, since a is in the interface of b, b will always be accompanied by a, even though it's not a dependency. Is that how it works? On Wed, Jul 23, 2014 at 2:58 PM, Nick Overdijk n...@astrant.net wrote: Thanks for the quick reply, but what if c needs b and b needs a? Adding INTERFACE will then break the build of course, right, since b isn't really linked to a... Or am I mistaken? On Wed, Jul 23, 2014 at 2:36 PM, Brad King brad.k...@kitware.com wrote: On 07/23/2014 08:00 AM, Nick Overdijk wrote: With this https://github.com/NickNick/cmake-dependency-waiting code here, why do b wait for a and c wait for b to be build? The object files could all be build in parallel right? Not doing it is making my distcc-cluster less and less useful the more nodes I add. Is there a way to fix or work around this? See here: http://www.cmake.org/Bug/view.php?id=14726#c35021 http://www.cmake.org/Bug/view.php?id=14726#c35023 -Brad -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] CMake generates Makefiles that don't parallelize as much as they could.
On 07/23/2014 09:07 AM, Nick Overdijk wrote:
Oh wait, since a is in the interface of b, b will always be
accompanied by a, even though it's not a dependency.
Is that how it works?
Yes. If B is a static library then it does not really link so
its dependencies are only ever used transitively when something
else links to B. If B really links as a shared library though
then see the following.
If target B links to target A then CMake will not compile objects
in B until A is done. As explained in my previous link this is
because we allow A to contain custom commands that generate
headers used by B. The VS and Xcode IDE build systems offer only
this granularity since they organize compilation and linking rules
of a single target together. The Makefile generator was designed
this way too. Only the Ninja generator has a chance of increasing
parallelism if the extra ordering dependencies were dropped. We've
thought about how to do extra analysis to determine when there is
no such custom command to drop them but have not implemented anything
yet.
You might be able to use OBJECT libraries to increase parallelism
for all generators and with no changes to CMake:
set(CMAKE_POSITION_INDEPENDENT_CODE ON)
add_library(Aobjs OBJECT a1.c a2.c)
add_library(Bobjs OBJECT b1.c b2.c)
add_library(Cobjs OBJECT c1.c c2.c)
set(dummy_c dummy.c) # needed for VS 6 and Xcode
add_library(A SHARED $TARGET_OBJECTS:Aobjs ${dummy_c})
add_library(B SHARED $TARGET_OBJECTS:Bobjs ${dummy_c})
add_library(C SHARED $TARGET_OBJECTS:Cobjs ${dummy_c})
target_link_libraries(B PUBLIC A)
target_link_libraries(C PUBLIC B)
This way the object compilations will be completely independent
of one another and of the linking and ordering dependencies.
-Brad
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] CMake generates Makefiles that don't parallelize as much as they could.
Crystal clear. Another layer of indirection eh? I'll see if I can work with
that... Thanks for the explanation.
On Wed, Jul 23, 2014 at 3:14 PM, Brad King brad.k...@kitware.com wrote:
On 07/23/2014 09:07 AM, Nick Overdijk wrote:
Oh wait, since a is in the interface of b, b will always be
accompanied by a, even though it's not a dependency.
Is that how it works?
Yes. If B is a static library then it does not really link so
its dependencies are only ever used transitively when something
else links to B. If B really links as a shared library though
then see the following.
If target B links to target A then CMake will not compile objects
in B until A is done. As explained in my previous link this is
because we allow A to contain custom commands that generate
headers used by B. The VS and Xcode IDE build systems offer only
this granularity since they organize compilation and linking rules
of a single target together. The Makefile generator was designed
this way too. Only the Ninja generator has a chance of increasing
parallelism if the extra ordering dependencies were dropped. We've
thought about how to do extra analysis to determine when there is
no such custom command to drop them but have not implemented anything
yet.
You might be able to use OBJECT libraries to increase parallelism
for all generators and with no changes to CMake:
set(CMAKE_POSITION_INDEPENDENT_CODE ON)
add_library(Aobjs OBJECT a1.c a2.c)
add_library(Bobjs OBJECT b1.c b2.c)
add_library(Cobjs OBJECT c1.c c2.c)
set(dummy_c dummy.c) # needed for VS 6 and Xcode
add_library(A SHARED $TARGET_OBJECTS:Aobjs ${dummy_c})
add_library(B SHARED $TARGET_OBJECTS:Bobjs ${dummy_c})
add_library(C SHARED $TARGET_OBJECTS:Cobjs ${dummy_c})
target_link_libraries(B PUBLIC A)
target_link_libraries(C PUBLIC B)
This way the object compilations will be completely independent
of one another and of the linking and ordering dependencies.
-Brad
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers
