Re: [CMake] cmake + ninja how to use several CPU cores?
On 2019-07-29 15:32-0400 Brad King wrote: [...]We don't know what `main.cpp` includes until after compiling it, by which point it is too late. It could have `#include "anything.txt"` for example. CMake must add these pessimistic dependencies to ensure a correct build. Is this the real reason why even the simplest no_custom_add_internal_library project (with nothing custom at all) still demonstrates the issue? Alan __ Alan W. Irwin Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __ Linux-powered Science __ -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: https://cmake.org/mailman/listinfo/cmake
Re: [CMake] cmake + ninja how to use several CPU cores?
On Mon, Jul 29, 2019 at 10:32 PM Brad King wrote: > > Even if we had that information we don't know what `main.cpp` includes > until after compiling it, by which point it is too late. It could have > `#include "anything.txt"` for example. CMake must add these pessimistic > dependencies to ensure a correct build. > > The only way to avoid this is to tell CMake that there are no dependencies > for compiling an object file by moving it to an object library that does > not depend on any of the targets with custom commands. > But my custom command OUTPUT is shared library, and in my case it would be strange to '#include' it into some source file. So what about EXPLICIT_DEPENDS_ONLY mentioned https://gitlab.kitware.com/cmake/cmake/issues/17097 to mark something that no way to be included into source file? -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: https://cmake.org/mailman/listinfo/cmake
Re: [CMake] cmake + ninja how to use several CPU cores?
On 2019-07-29 13:24-0400 Brad King wrote: On 7/29/19 11:50 AM, Dave Milter wrote: Only source code are generated, so main.cpp -> main.cpp.o doesn't depend on anything. But generated by cmake build.ninja still require link of extern_lib before starting main.cpp -> main.cpp.o If there are any custom commands in any targets on which main.cpp's target depends then CMake must pessimistically assume that the custom commands may generate a header needed to compile `main.cpp`. Hi Brad: Thanks for that clarification. Clearly David's test case for an external library and my own recently corrected test case for an internal library are not consistent with that assumption which explains the bad results we get for them. However, I think there is still a problem when a project is consistent with that assumption. See the attached no_custom_add_internal_library.tar.gz which contains an extremely simple test project with no custom commands at all (i.e., no wait and headers and source are pre-existing in all cases). Yet for this project I get the following results with ninja. irwin@merlin> ninja -j16 -v app [1/7] /usr/bin/c++ -Dinternal_lib_EXPORTS -I../lib -fPIC -MD -MT CMakeFiles/internal_lib.dir/lib/lib4.cpp.o -MF CMakeFiles/internal_lib.dir/lib/lib4.cpp.o.d -o CMakeFiles/internal_lib.dir/lib/lib4.cpp.o -c ../lib/lib4.cpp [2/7] /usr/bin/c++ -Dinternal_lib_EXPORTS -I../lib -fPIC -MD -MT CMakeFiles/internal_lib.dir/lib/lib1.cpp.o -MF CMakeFiles/internal_lib.dir/lib/lib1.cpp.o.d -o CMakeFiles/internal_lib.dir/lib/lib1.cpp.o -c ../lib/lib1.cpp [3/7] /usr/bin/c++ -Dinternal_lib_EXPORTS -I../lib -fPIC -MD -MT CMakeFiles/internal_lib.dir/lib/lib2.cpp.o -MF CMakeFiles/internal_lib.dir/lib/lib2.cpp.o.d -o CMakeFiles/internal_lib.dir/lib/lib2.cpp.o -c ../lib/lib2.cpp [4/7] /usr/bin/c++ -Dinternal_lib_EXPORTS -I../lib -fPIC -MD -MT CMakeFiles/internal_lib.dir/lib/lib3.cpp.o -MF CMakeFiles/internal_lib.dir/lib/lib3.cpp.o.d -o CMakeFiles/internal_lib.dir/lib/lib3.cpp.o -c ../lib/lib3.cpp [5/7] : && /usr/bin/c++ -fPIC-shared -Wl,-soname,libinternal_lib.so -o libinternal_lib.so CMakeFiles/internal_lib.dir/lib/lib1.cpp.o CMakeFiles/internal_lib.dir/lib/lib2.cpp.o CMakeFiles/internal_lib.dir/lib/lib3.cpp.o CMakeFiles/internal_lib.dir/lib/lib4.cpp.o && : [6/7] /usr/bin/c++ -I../lib -MD -MT CMakeFiles/app.dir/main.cpp.o -MF CMakeFiles/app.dir/main.cpp.o.d -o CMakeFiles/app.dir/main.cpp.o -c ../main.cpp [7/7] : && /usr/bin/c++ CMakeFiles/app.dir/main.cpp.o -o app -Wl,-rpath,/home/irwin/David.Milter/20190728/test_ninja/no_gen_add_internal_library/build_dir libinternal_lib.so && : irwin@merlin> grep cmake_object_order_depends_target_app build.ninja build cmake_object_order_depends_target_app: phony || cmake_object_order_depends_target_internal_lib build CMakeFiles/app.dir/main.cpp.o: CXX_COMPILER__app ../main.cpp || cmake_object_order_depends_target_app Alan __ Alan W. Irwin Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __ Linux-powered Science __ -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: https://cmake.org/mailman/listinfo/cmake
Re: [CMake] cmake + ninja how to use several CPU cores?
On 7/29/19 3:22 PM, Dave Milter wrote: >> Since then all objects in a target can start compiling independently >> so long as that target does not depend on any targets with custom >> commands. See also https://gitlab.kitware.com/cmake/cmake/issues/17097 > Is any way to tell cmake that add_custom_command is have no side effect, > and OUTPUT is exactly what it produces? Even if we had that information we don't know what `main.cpp` includes until after compiling it, by which point it is too late. It could have `#include "anything.txt"` for example. CMake must add these pessimistic dependencies to ensure a correct build. The only way to avoid this is to tell CMake that there are no dependencies for compiling an object file by moving it to an object library that does not depend on any of the targets with custom commands. -Brad -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: https://cmake.org/mailman/listinfo/cmake
Re: [CMake] cmake + ninja how to use several CPU cores?
On Mon, Jul 29, 2019 at 8:24 PM Brad King wrote: > > CMake 3.9 made this much better than it used to be: > > https://gitlab.kitware.com/cmake/cmake/merge_requests/430 > > Since then all objects in a target can start compiling independently > so long as that target does not depend on any targets with custom > commands. Is any way to tell cmake that add_custom_command is have no side effect, and OUTPUT is exactly what it produces? I see that there is Tests/RunCMake/Ninja/LooseObjectDepends.cmake in https://gitlab.kitware.com/cmake/cmake/merge_requests/430 is it negative test, or there is chance for fast compilation? -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: https://cmake.org/mailman/listinfo/cmake
Re: [CMake] cmake + ninja how to use several CPU cores?
On 2019-07-29 18:50+0300 Dave Milter wrote: On Mon, Jul 29, 2019 at 1:48 PM Bruce Stephens wrote: I think it's reasonable for CMake/Ninja to require the headers be generated, especially since main.cpp does include one of them (though CMake/Ninja doesn't know that until later). lib/lib1.cpp is more arguable, but I imagine there's no real distinction made: main.cpp.o depends on all the dependencies of the libraries. In my initial example headers files are not generated in cmake scripts. Only source code are generated, so main.cpp -> main.cpp.o doesn't depend on anything. But generated by cmake build.ninja still require link of extern_lib before starting main.cpp -> main.cpp.o Thanks, Bruce, for spotting that issue with the first version of my simple example which I oversimplified. I have now corrected it (see attached tarball) to follow the use case of pre-existing headers and an app that links to a library (internal for my corrected test case, and external for David's test case). The issue (compilation of main.cpp is unreasonably delayed for parallel builds until after the library (internal or external) is built) still shows up in my corrected test case for both -G"Ninja" and -G"Unix Makefiles" just like happens for David's test case for an external library. So the question remains why does CMake constrain parallel builds this way (to the point that it is a bottlneck for David's real project with external libraries) for this general use case? Alan __ Alan W. Irwin Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __ Linux-powered Science __ corrected_add_internal_library.tar.gz Description: tarball containing corrected test case of app with pre-existing headers and internal library -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: https://cmake.org/mailman/listinfo/cmake
Re: [CMake] cmake + ninja how to use several CPU cores?
On 7/29/19 11:50 AM, Dave Milter wrote: > Only source code are generated, so main.cpp -> main.cpp.o doesn't > depend on anything. > But generated by cmake build.ninja still require link of extern_lib > before starting main.cpp -> main.cpp.o If there are any custom commands in any targets on which main.cpp's target depends then CMake must pessimistically assume that the custom commands may generate a header needed to compile `main.cpp`. Object libraries can be used to break the dependencies that lead to the assumption. CMake 3.9 made this much better than it used to be: https://gitlab.kitware.com/cmake/cmake/merge_requests/430 Since then all objects in a target can start compiling independently so long as that target does not depend on any targets with custom commands. -Brad -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: https://cmake.org/mailman/listinfo/cmake
Re: [CMake] cmake + ninja how to use several CPU cores?
On Mon, Jul 29, 2019 at 1:48 PM Bruce Stephens wrote: > > I think it's reasonable for CMake/Ninja to require the headers be > generated, especially since main.cpp does include one of them (though > CMake/Ninja doesn't know that until later). lib/lib1.cpp is more > arguable, but I imagine there's no real distinction made: main.cpp.o > depends on all the dependencies of the libraries. > In my initial example headers files are not generated in cmake scripts. Only source code are generated, so main.cpp -> main.cpp.o doesn't depend on anything. But generated by cmake build.ninja still require link of extern_lib before starting main.cpp -> main.cpp.o -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: https://cmake.org/mailman/listinfo/cmake
Re: [CMake] cmake + ninja how to use several CPU cores?
You might find it interesting to look at build.ninja. By the looks of it there's a (phony) target guving the dependencies of the library. (That's called cmake_object_order_depends_target_internal_lib) and CMakeFiles/app.dir/main.cpp.o is depending on that. Presumably the idea is that there's no point trying to compile main.cpp before we could build the library, since (as in this case) that might require generated headers which main.cpp might need. If I get rid of the wait5 thing, then main.cpp gets compiled (using "ninja -v CMakeFiles/app.dir/main.cpp.o") without the library. Just the headers (and lib/lib1.cpp) get generated. I think it's reasonable for CMake/Ninja to require the headers be generated, especially since main.cpp does include one of them (though CMake/Ninja doesn't know that until later). lib/lib1.cpp is more arguable, but I imagine there's no real distinction made: main.cpp.o depends on all the dependencies of the libraries. On Mon, 29 Jul 2019 at 10:34, Alan W. Irwin wrote: > > On 2019-07-28 23:39-0700 Alan W. Irwin wrote: > > > @Both: I also plan to look at whether this issue exists for > > the internal library case so more later when I get that > > corresponding simple test project implemented. > > @Brad and David: > > I have now implemented a simple test project for the internal library > case. See the attached tarball (and please let me know if your input > mail software chain allows you to receive that tarball). In general I > think this test case improves somewhat on Dave's test project by > making the sleep more explicit at the CMake level, generating the > headers explicitly rather than including them in the tarball, > generating all header and source code files and the library (internal > in this case) within the build tree rather than source tree (which > considerably simplifies making a clean start), and by allowing the > user to specify the -DBUILD_SHARED_LIBS=ON (or OFF) cmake option to > choose whether the library is built shared on static. (Of course, > none of these improvements should affect the delayed compilation > issue.) > > For this new simple test project, here are the results for the -G"Ninja" case > (when -DBUILD_SHARED_LIBS=ON is specified): > > irwin@merlin> ninja -j16 -v app > [1/10] cd > /home/irwin/David.Milter/20190728/test_ninja/add_internal_library/build_dir/lib > && sh > /home/irwin/David.Milter/20190728/test_ninja/add_internal_library/lib/include_generator.sh > [2/10] cd > /home/irwin/David.Milter/20190728/test_ninja/add_internal_library/build_dir/lib > && sh > /home/irwin/David.Milter/20190728/test_ninja/add_internal_library/lib/lib_generator.sh > [3/10] cd > /home/irwin/David.Milter/20190728/test_ninja/add_internal_library/build_dir > && sleep 5 > [4/10] /usr/bin/c++ -Dinternal_lib_EXPORTS -Ilib -fPIC -MD -MT > CMakeFiles/internal_lib.dir/lib/lib1.cpp.o -MF > CMakeFiles/internal_lib.dir/lib/lib1.cpp.o.d -o > CMakeFiles/internal_lib.dir/lib/lib1.cpp.o -c lib/lib1.cpp > [5/10] /usr/bin/c++ -Dinternal_lib_EXPORTS -Ilib -fPIC -MD -MT > CMakeFiles/internal_lib.dir/lib/lib3.cpp.o -MF > CMakeFiles/internal_lib.dir/lib/lib3.cpp.o.d -o > CMakeFiles/internal_lib.dir/lib/lib3.cpp.o -c lib/lib3.cpp > [6/10] /usr/bin/c++ -Dinternal_lib_EXPORTS -Ilib -fPIC -MD -MT > CMakeFiles/internal_lib.dir/lib/lib4.cpp.o -MF > CMakeFiles/internal_lib.dir/lib/lib4.cpp.o.d -o > CMakeFiles/internal_lib.dir/lib/lib4.cpp.o -c lib/lib4.cpp > [7/10] /usr/bin/c++ -Dinternal_lib_EXPORTS -Ilib -fPIC -MD -MT > CMakeFiles/internal_lib.dir/lib/lib2.cpp.o -MF > CMakeFiles/internal_lib.dir/lib/lib2.cpp.o.d -o > CMakeFiles/internal_lib.dir/lib/lib2.cpp.o -c lib/lib2.cpp > [8/10] : && /usr/bin/c++ -fPIC-shared -Wl,-soname,libinternal_lib.so -o > libinternal_lib.so CMakeFiles/internal_lib.dir/lib/lib1.cpp.o > CMakeFiles/internal_lib.dir/lib/lib2.cpp.o > CMakeFiles/internal_lib.dir/lib/lib3.cpp.o > CMakeFiles/internal_lib.dir/lib/lib4.cpp.o && : > [9/10] /usr/bin/c++ -Ilib -MD -MT CMakeFiles/app.dir/main.cpp.o -MF > CMakeFiles/app.dir/main.cpp.o.d -o CMakeFiles/app.dir/main.cpp.o -c > ../main.cpp > [10/10] : && /usr/bin/c++ CMakeFiles/app.dir/main.cpp.o -o app > -Wl,-rpath,/home/irwin/David.Milter/20190728/test_ninja/add_internal_library/build_dir > libinternal_lib.so && : > > Note that the build of main.cpp.o is (unreasonably in my opinion) delayed to > step 9. > > And that build is similarly delayed in the -G"Unix Makefiles" case. > That is this newly implemented test case for the internal library > build shows exactly the same unexplained compilation delays for > main.cpp for both -G"Ninja" and -G"Unix Makefiles" as David's test > case for the external library build. > > @Brad and other CMake developers who might be lurking here: > > Can anybody explain why CMake has this delayed compilation "feature" > for apps that depend on internal or external libraries, and if there > is no compelling reason for it would it be straightforward to
Re: [CMake] cmake + ninja how to use several CPU cores?
On 2019-07-28 23:39-0700 Alan W. Irwin wrote: @Both: I also plan to look at whether this issue exists for the internal library case so more later when I get that corresponding simple test project implemented. @Brad and David: I have now implemented a simple test project for the internal library case. See the attached tarball (and please let me know if your input mail software chain allows you to receive that tarball). In general I think this test case improves somewhat on Dave's test project by making the sleep more explicit at the CMake level, generating the headers explicitly rather than including them in the tarball, generating all header and source code files and the library (internal in this case) within the build tree rather than source tree (which considerably simplifies making a clean start), and by allowing the user to specify the -DBUILD_SHARED_LIBS=ON (or OFF) cmake option to choose whether the library is built shared on static. (Of course, none of these improvements should affect the delayed compilation issue.) For this new simple test project, here are the results for the -G"Ninja" case (when -DBUILD_SHARED_LIBS=ON is specified): irwin@merlin> ninja -j16 -v app [1/10] cd /home/irwin/David.Milter/20190728/test_ninja/add_internal_library/build_dir/lib && sh /home/irwin/David.Milter/20190728/test_ninja/add_internal_library/lib/include_generator.sh [2/10] cd /home/irwin/David.Milter/20190728/test_ninja/add_internal_library/build_dir/lib && sh /home/irwin/David.Milter/20190728/test_ninja/add_internal_library/lib/lib_generator.sh [3/10] cd /home/irwin/David.Milter/20190728/test_ninja/add_internal_library/build_dir && sleep 5 [4/10] /usr/bin/c++ -Dinternal_lib_EXPORTS -Ilib -fPIC -MD -MT CMakeFiles/internal_lib.dir/lib/lib1.cpp.o -MF CMakeFiles/internal_lib.dir/lib/lib1.cpp.o.d -o CMakeFiles/internal_lib.dir/lib/lib1.cpp.o -c lib/lib1.cpp [5/10] /usr/bin/c++ -Dinternal_lib_EXPORTS -Ilib -fPIC -MD -MT CMakeFiles/internal_lib.dir/lib/lib3.cpp.o -MF CMakeFiles/internal_lib.dir/lib/lib3.cpp.o.d -o CMakeFiles/internal_lib.dir/lib/lib3.cpp.o -c lib/lib3.cpp [6/10] /usr/bin/c++ -Dinternal_lib_EXPORTS -Ilib -fPIC -MD -MT CMakeFiles/internal_lib.dir/lib/lib4.cpp.o -MF CMakeFiles/internal_lib.dir/lib/lib4.cpp.o.d -o CMakeFiles/internal_lib.dir/lib/lib4.cpp.o -c lib/lib4.cpp [7/10] /usr/bin/c++ -Dinternal_lib_EXPORTS -Ilib -fPIC -MD -MT CMakeFiles/internal_lib.dir/lib/lib2.cpp.o -MF CMakeFiles/internal_lib.dir/lib/lib2.cpp.o.d -o CMakeFiles/internal_lib.dir/lib/lib2.cpp.o -c lib/lib2.cpp [8/10] : && /usr/bin/c++ -fPIC-shared -Wl,-soname,libinternal_lib.so -o libinternal_lib.so CMakeFiles/internal_lib.dir/lib/lib1.cpp.o CMakeFiles/internal_lib.dir/lib/lib2.cpp.o CMakeFiles/internal_lib.dir/lib/lib3.cpp.o CMakeFiles/internal_lib.dir/lib/lib4.cpp.o && : [9/10] /usr/bin/c++ -Ilib -MD -MT CMakeFiles/app.dir/main.cpp.o -MF CMakeFiles/app.dir/main.cpp.o.d -o CMakeFiles/app.dir/main.cpp.o -c ../main.cpp [10/10] : && /usr/bin/c++ CMakeFiles/app.dir/main.cpp.o -o app -Wl,-rpath,/home/irwin/David.Milter/20190728/test_ninja/add_internal_library/build_dir libinternal_lib.so && : Note that the build of main.cpp.o is (unreasonably in my opinion) delayed to step 9. And that build is similarly delayed in the -G"Unix Makefiles" case. That is this newly implemented test case for the internal library build shows exactly the same unexplained compilation delays for main.cpp for both -G"Ninja" and -G"Unix Makefiles" as David's test case for the external library build. @Brad and other CMake developers who might be lurking here: Can anybody explain why CMake has this delayed compilation "feature" for apps that depend on internal or external libraries, and if there is no compelling reason for it would it be straightforward to remove it? Alan __ Alan W. Irwin Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __ Linux-powered Science __ add_internal_library.tar.gz Description: tarball for newly implemented simple test case for internal library + app -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to
Re: [CMake] cmake + ninja how to use several CPU cores?
Hi to Dave and Brad: @Dave: As you are probably already aware Brad is one of the primary CMake developers so I think he should be in on this discussion. @Brad: Using Dave's simple test project I confirmed the compilation delay issue below in detail for both the -G"Ninja" case AND the -G"Unix Makefiles" case on Debian Buster. Would you please take a look at this simple test case to see if you can find the explanation for this issue? After years of ignoring CMake + Ninja because of the Fortran issues for that combination, I have just realized you have a fork of Ninja (not yet merged upstream) which apparently works well with CMake for the Fortran case. So I am in the process of trying that Ninja fork out for all components of PLplot (including our Fortran binding and examples), and I plan to report to you those results in a separate thread. Which explains my renewed interest in Ninja (in this case not your fork since no Fortran is involved) for this current thread. @Both: I also plan to look at whether this issue exists for the internal library case so more later when I get that corresponding simple test project implemented. More details below concerning the external library case On 2019-07-29 03:36+0300 Dave Milter wrote: Hi, On Mon, Jul 29, 2019 at 12:01 AM Alan W. Irwin wrote: On 2019-07-28 19:03+0300 Dave Milter wrote: To help answer your question, I see no fundamental issues with your code above. If the latter is really true, that appears to me to be a CMake bug (although I cannot make the final judgement call on that since I am an experienced CMake user rather than a CMake developer). And if so, does that same problem (unnecessary delay in compilation rather than linking) occur for the following simplified case? In fact my code in previous email is demo code. It already contains "sleep" inside lib_generator.sh to emulate real project. @Dave: Great minds think alike concerning clearly demonstrating this issue with the help of the sleep command. :-) I am just not sure is this good idea to send attachment into this mailing list. Actually, for obvious reasons of convenience, gzipped tarball attachments should never be an issue on mailing lists (like the CMake ones) which are relevant to discussion of software. (There are typically mailing-list limits with regard to size, but they are normally quite generous so I have never run into this issue on the CMake mailing lists. Therefore, as expected, your attachment got through to me without issues. Also, I double-checked your description of that attachment which is critical for the input mail chains of some users. In your case that description (which you can configure with your mail software) was "application/GZIP" which is obviously fine for my input mail chain case and presumably most others on this list who want to look at that attachment. @Brad: have you had any trouble seeing that attachment without your input mail chain stripping it out? See attachment, it is possible only to build on Linux/Mac, because of usage bash/gcc for lib_geneator.sh And I am pretty sure that problem in cmake (or my usage of cmake), because of build.ninja generated by cmake contains such rules: ``` build cmake_object_order_depends_target_app: phony || extern_lib_target build CMakeFiles/app.dir/main.cpp.o: CXX_COMPILER__app ../main.cpp || cmake_object_order_depends_target_app ``` "||" is "order-only dependencies" according to https://ninja-build.org/manual.html#_syntax_example . So cmake instruct ninja do not build main.cpp.o untill extern_lib_target build. @Dave: Thanks very much for that simple test project. @Both: I configured and built it using the Debian Buster = Stable system versions of cmake (version 3.13.4) and ninja (version 1.8.2) using the following commands I. The -G"Ninja" case # Fresh start: # IMPORTANT: first cd to the top-level directory of the source tree. Then # Get rid of the previously generated external library (if any) to # insure the 5 second pause associated with that generation always # occurs. rm -f extern_lib/libextern.so # Start with an empty build tree rm -rf build_dir mkdir build_dir cd build_dir # Configure time cmake -G"Ninja" .. >& cmake.out ==> real0m1.084s user0m0.845s sys 0m0.232s # Just to be specific (although ninja normally figures this out for itself for parallel builds) # build with -j16 since I have 16-thread hardware. Also use -v option to create (slightly more) # verbose output. time ninja -j16 -v all ==> [1/3] cd /home/irwin/David.Milter/20190728/test_ninja/add_library_imported/extern_lib && sh ./lib_generator.sh Generating... Generating...DONE [2/3] /usr/bin/c++ -isystem ../extern_lib/gen -MD -MT CMakeFiles/app.dir/main.cpp.o -MF CMakeFiles/app.dir/main.cpp.o.d -o CMakeFiles/app.dir/main.cpp.o -c ../main.cpp [3/3] : && /usr/bin/c++ CMakeFiles/app.dir/main.cpp.o -o app
Re: [CMake] cmake + ninja how to use several CPU cores?
Hi, On Mon, Jul 29, 2019 at 12:01 AM Alan W. Irwin wrote: > > On 2019-07-28 19:03+0300 Dave Milter wrote: > > To help answer your question, I see no fundamental issues with your code > above. > > If the latter is really true, that appears to me to be a CMake bug > (although I cannot make the final judgement call on that since I am an > experienced CMake user rather than a CMake developer). And if so, does that > same problem (unnecessary delay in compilation rather than linking) > occur for the following simplified case? > In fact my code in previous email is demo code. It already contains "sleep" inside lib_generator.sh to emulate real project. I am just not sure is this good idea to send attachment into this mailing list. See attachment, it is possible only to build on Linux/Mac, because of usage bash/gcc for lib_geneator.sh And I am pretty sure that problem in cmake (or my usage of cmake), because of build.ninja generated by cmake contains such rules: ``` build cmake_object_order_depends_target_app: phony || extern_lib_target build CMakeFiles/app.dir/main.cpp.o: CXX_COMPILER__app ../main.cpp || cmake_object_order_depends_target_app ``` "||" is "order-only dependencies" according to https://ninja-build.org/manual.html#_syntax_example . So cmake instruct ninja do not build main.cpp.o untill extern_lib_target build. add_library_imported.tar.gz Description: application/gzip -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: https://cmake.org/mailman/listinfo/cmake
Re: [CMake] cmake + ninja how to use several CPU cores?
On 2019-07-28 19:03+0300 Dave Milter wrote:
Hi,
I heard that ninja has great feature it allows build continue without
wainting full link.
So if you have library Lib and executable App,
source code of App may build in parallel with source code of Lib,
and sync only link stage. While other build systems force build of Lib,
and only then start build of any object files of App.
But is it possible to use this feature with cmake?
Here my cmake code to emulate bottleneck of our build:
```
cmake_minimum_required(VERSION 3.12 FATAL_ERROR)
project(imported)
set(EXTERN_LIB_FILE ${CMAKE_CURRENT_SOURCE_DIR}/extern_lib/libextern.so)
set(EXTERN_LIB_INCLUDES ${CMAKE_CURRENT_SOURCE_DIR}/extern_lib/gen)
add_custom_command(OUTPUT ${EXTERN_LIB_FILE}
COMMAND sh ./lib_generator.sh
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/extern_lib)
add_custom_target(extern_lib_target DEPENDS ${EXTERN_LIB_FILE})
add_library(extern_lib SHARED IMPORTED GLOBAL)
add_dependencies(extern_lib extern_lib_target)
set_target_properties(extern_lib
PROPERTIES
IMPORTED_LOCATION ${EXTERN_LIB_FILE}
INTERFACE_INCLUDE_DIRECTORIES ${EXTERN_LIB_INCLUDES})
add_executable(app main.cpp)
target_link_libraries(app extern_lib)
```
we use not C/C++ library as part of our source tree,
and it builds some time,
so I expect that ninja can run "main.cpp" -> main.o compilation during
"extern_lib" build,
but this is not happens.
Is cmake has problem with add_custom_command and ninja,
or the whole idea is wrong, and cmake + ninja can not work in this way,
and I should use some other project generator instead of cmake for this?
Hi Dave:
To help answer your question, I see no fundamental issues with your code above.
The
add_dependencies(extern_lib extern_lib_target)
command in your above code means that the external project that is
created by building the extern_lib_target target must be *entirely*
completed before the extern_lib target
is built. Furthermore, the
target_link_libraries(app extern_lib)
command means extern_lib must be built before the app is linked which, of
course, is a complete necessity.
However, I don't see anything in your CMake code that demands that the
compilation of main.cpp must be delayed until extern_lib is built. So
can you double-check that the bottleneck is not actually the
(necessary until external_lib is built) delay of linking of app rather
than some unneccessary delay in the compilation of main.cpp?
If the latter is really true, that appears to me to be a CMake bug
(although I cannot make the final judgement call on that since I am an
experienced CMake user rather than a CMake developer). And if so, does that
same problem (unnecessary delay in compilation rather than linking)
occur for the following simplified case?
cmake_minimum_required(VERSION 3.12 FATAL_ERROR)
project(test_internal_library)
# Delay 10 seconds so that it is obvious if main.cpp compilation is
# being unnecessarily delayed.
# You should introduce boolean CMake logic here to
# use the equivalent (see
#
)
# "timeout 10" command for the Windows case.
add_custom_target(delay
COMMAND sleep 10
)
add_library(intern_lib SHARED ${)
# Add a 10 second delay until intern_lib is built
add_dependency(intern_lib delay)
add_executable(app main.cpp)
target_link_libraries(app intern_lib)
Also, please organize both test cases as tarballs containing simple
examples (including a simplified lib_generator.sh for the first case,
a 10 second delay inserted in that case as well, and simplified source
code for library (e.g., containing one function to return the "hello,
world" string and a simplified app (a main routine to print out that
string) for both cases). Such self-contained test projects make it
much more convenient for others to confirm the issue you have
discovered on the platforms they have access to. For example, if you
supply such self-contained test cases, I would be happy to run both
test cases here on my Linux platform for both ninja and make. (I have
had a lot of success with parallel builds for the make case on Linux
which is why I would like to also test make just in case it turns out
you have found a ninja-only CMake bug.)
Alan
__
Alan W. Irwin
Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.org); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__
Linux-powered Science
__
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
[CMake] cmake + ninja how to use several CPU cores?
Hi,
I heard that ninja has great feature it allows build continue without
wainting full link.
So if you have library Lib and executable App,
source code of App may build in parallel with source code of Lib,
and sync only link stage. While other build systems force build of Lib,
and only then start build of any object files of App.
But is it possible to use this feature with cmake?
Here my cmake code to emulate bottleneck of our build:
```
cmake_minimum_required(VERSION 3.12 FATAL_ERROR)
project(imported)
set(EXTERN_LIB_FILE ${CMAKE_CURRENT_SOURCE_DIR}/extern_lib/libextern.so)
set(EXTERN_LIB_INCLUDES ${CMAKE_CURRENT_SOURCE_DIR}/extern_lib/gen)
add_custom_command(OUTPUT ${EXTERN_LIB_FILE}
COMMAND sh ./lib_generator.sh
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/extern_lib)
add_custom_target(extern_lib_target DEPENDS ${EXTERN_LIB_FILE})
add_library(extern_lib SHARED IMPORTED GLOBAL)
add_dependencies(extern_lib extern_lib_target)
set_target_properties(extern_lib
PROPERTIES
IMPORTED_LOCATION ${EXTERN_LIB_FILE}
INTERFACE_INCLUDE_DIRECTORIES ${EXTERN_LIB_INCLUDES})
add_executable(app main.cpp)
target_link_libraries(app extern_lib)
```
we use not C/C++ library as part of our source tree,
and it builds some time,
so I expect that ninja can run "main.cpp" -> main.o compilation during
"extern_lib" build,
but this is not happens.
Is cmake has problem with add_custom_command and ninja,
or the whole idea is wrong, and cmake + ninja can not work in this way,
and I should use some other project generator instead of cmake for this?
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
