Re: [Boost-cmake] Analysis of the current CMake system
on Wed Mar 04 2009, Bill Hoffman bill.hoffman-AT-kitware.com wrote: David Abrahams wrote: Another new feature in CTest/CDash development heads which may be of interest to Boost is support for subproject labeling. See their main dashboard page here: http://trilinos-dev.sandia.gov/cdash/index.php?project=Trilinosdate=2009-02-27 Functionally, it's pretty darned good. Cool! Room for improvement: * I think we'd want a view that's limited to issues, so a subproject doesn't show up if it's all green (or has only warnings, configurable according to project). See http://www.boost.org/development/tests/trunk/developer/issues.html You can do that now from here: http://trilinos-dev.sandia.gov/cdash/index.php?project=Trilinosdisplay=projectdate=2009-02-27 Click on Show Filters, and you can create a filter that shows only errors, warnings or other items that you want. The idea will be to be able to save the queries and create your own custom views for different purposes. Nice capability; not a great interface. If I want to see just things with errors, I need a filter that says +--+ +--+ ++ |build errors | |is not| || +--+ +--+ ++ * It doesn't make good use of space in my browser window unless I make the browser really narrow. It would take some design work, but it would be good to be able to get a bigger picture view at a glance, especially when the browser window is large. We are open to suggestions. I'll give it some thought. There is a cdash mailing list here: http://www.cdash.org/cdash/help/mailing.html * It's kinda ugly, but I suppose we can tweak the CSS ourselves. Hey! (we actually hired a graphics designer for that...) Sorry! -- Dave Abrahams BoostPro Computing http://www.boostpro.com ___ Boost-cmake mailing list Boost-cmake@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-cmake
Re: [Boost-cmake] Analysis of the current CMake system
David Abrahams wrote: Click on Show Filters, and you can create a filter that shows only errors, warnings or other items that you want. The idea will be to be able to save the queries and create your own custom views for different purposes. Nice capability; not a great interface. If I want to see just things with errors, I need a filter that says +--+ +--+ ++ |build errors | |is not| || +--+ +--+ ++ I often click on the errors column label to sort by error count. Then all the zeros go to the bottom. -Brad ___ Boost-cmake mailing list Boost-cmake@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-cmake
Re: [Boost-cmake] Analysis of the current CMake system
troy d. straszheim wrote:
So the first order of business would be to remove this kind of thing:
set(CMAKE_CXX_COMPILE_OBJECT \${PYTHON_EXECUTABLE}\
\${BOOST_TEST_DRIVER}\ CMAKE_CURRENT_BINARY_DIR cxx_compile_object
OBJECT ${CMAKE_CXX_COMPILE_OBJECT} )
We're 100% agreed on the need for this, as far as I can see.
A couple weeks ago I committed changes in CMake CVS HEAD to provide an
interface like this. It works only for Makefile generators, but is now
a builtin feature.
Troy, if you build CMake from CVS and run with --help-properties you
will see these properties:
RULE_LAUNCH_COMPILE
Specify a launcher for compile rules.
Makefile generators prefix compiler commands with the given launcher
command line. This is intended to allow launchers to intercept build
problems with high granularity. Non-Makefile generators currently
ignore this property.
RULE_LAUNCH_CUSTOM
Specify a launcher for custom rules.
Makefile generators prefix custom commands with the given launcher
command line. This is intended to allow launchers to intercept build
problems with high granularity. Non-Makefile generators currently
ignore this property.
RULE_LAUNCH_LINK
Specify a launcher for link rules.
Makefile generators prefix link and archive commands with the given
launcher command line. This is intended to allow launchers to
intercept build problems with high granularity. Non-Makefile
generators currently ignore this property.
CTest provides a launcher interface too. Look in Modules/CTest.cmake for
this code:
IF(CTEST_USE_LAUNCHERS)
SET(CTEST_LAUNCH_COMPILE \${CMAKE_CTEST_COMMAND}\ --launch --target-name TARGET_NAME --build-dir
CMAKE_CURRENT_BINARY_DIR --output OBJECT --source SOURCE --language LANGUAGE --)
SET(CTEST_LAUNCH_LINK\${CMAKE_CTEST_COMMAND}\ --launch --target-name TARGET_NAME --build-dir
CMAKE_CURRENT_BINARY_DIR --output TARGET --target-type TARGET_TYPE --language LANGUAGE --)
SET(CTEST_LAUNCH_CUSTOM \${CMAKE_CTEST_COMMAND}\ --launch --target-name TARGET_NAME
--build-dir CMAKE_CURRENT_BINARY_DIR --output OUTPUT --)
SET_PROPERTY(GLOBAL PROPERTY RULE_LAUNCH_COMPILE ${CTEST_LAUNCH_COMPILE})
SET_PROPERTY(GLOBAL PROPERTY RULE_LAUNCH_LINK ${CTEST_LAUNCH_LINK})
SET_PROPERTY(GLOBAL PROPERTY RULE_LAUNCH_CUSTOM ${CTEST_LAUNCH_CUSTOM})
ENDIF(CTEST_USE_LAUNCHERS)
When a project sets CTEST_USE_LAUNCHERS to true, CTest uses this interface
to run all Make rules through an (undocumented) API 'ctest --launch'. It
allows us to report full information about failed build rules (working dir,
full command line, stdout, stderr, return code, etc.) with no log scraping.
The launcher also does some simple scraping of compiler output to detect
warnings (needed for some Windows compilers which warn on stdout).
The current CDash release will not understand CTest build submissions that
use this launcher interface, so one would need CDash from its SVN trunk to
try it. However, you should be able to use the above code as an example
to create your python/Trac reporting stuff.
-Brad
___
Boost-cmake mailing list
Boost-cmake@lists.boost.org
http://lists.boost.org/mailman/listinfo.cgi/boost-cmake
Re: [Boost-cmake] Analysis of the current CMake system
Ingo Albrecht wrote: Note that vcbuild (the command line driver for VS builds) has command line arguments for specifying strings to prefix log messages at various log levels with. This should make log scraping of the compilation much more reliable, although it still disgusts me. This does not work for CTest though because it tests using cmake scripts. Running vcbuild is certainly no alternative for trying the build in the IDE, but it should be sufficient for continuous integration. Further, I have to note that command-line VS builds should be supported for one simple reason: nmake does not support parallel builds and probably never will. This makes VS the easiest way of running a parallel build on Windows (locally or distributed with additional tools). GNU make from MSYS is out of the question because MSYS seems far from production-grade. Should IDE builds be considered support-worthy, it would of course be necessary to test manually before releases. I hope that some of this helps Interesting, thanks for this. Would the 'cwrap' approach (where the compiler is called via a wrapper) work under vcbuild? (Forgive me, I don't do windows). The only reason I paid attention to NMAKE is that it was somewhat familiar to me. -t ___ Boost-cmake mailing list Boost-cmake@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-cmake
Re: [Boost-cmake] Analysis of the current CMake system
troy d. straszheim wrote: Further, I have to note that command-line VS builds should be supported for one simple reason: nmake does not support parallel builds and probably never will. This makes VS the easiest way of running a parallel build on Windows (locally or distributed with additional tools). GNU make from MSYS is out of the question because MSYS seems far from production-grade. Should IDE builds be considered support-worthy, it would of course be necessary to test manually before releases. I hope that some of this helps Interesting, thanks for this. Would the 'cwrap' approach (where the compiler is called via a wrapper) work under vcbuild? (Forgive me, I don't do windows). The only reason I paid attention to NMAKE is that it was somewhat familiar to me. No, that would not help. vcbuild uses the project files. CTest can already run devenv from the command line which is almost the same thing. I think with VS you really have to test the project files for performance, and to make sure they work. We are working on a way of extracting the build information from the log files produced by VS. -Bill ___ Boost-cmake mailing list Boost-cmake@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-cmake
Re: [Boost-cmake] Analysis of the current CMake system
on Mon Jan 19 2009, Brad King brad.king-AT-kitware.com wrote: Hi Dave, I think some of the confusion is because my original posting proposed *two* different approaches to testing: 1.) Build all the (run) tests for one library into a single executable. Each test gets its own TU but the tests are linked together. Execution of the tests is delayed until test time at which point CTest runs the executable over and over with different arguments for each test. That approach has promise for us, but it would take some investment because, in general, our test executables are more granular than that. 2.) Build the tests as individual executables, but not until *test* time. The idea is that we drive testing with CTest, but each test is a *recursive invocation* of CTest with its --build-and-test feature. This feature drives the build of a test source file through the native toolchain as part of running the test. The output of the test includes all the native toolchain stuff and the test executable's output. However, every test is its own separate recursive invocation of ctest, so its output is recorded separately from other tests. Seems plausible. Run-tests can use either #1 or #2. Compile-only tests should use #2 since the interesting part of the test is the compilation, and compile-fail tests can clearly not be linked to other tests. Right, I think. David Abrahams wrote: on Thu Jan 15 2009, Brad King brad.king-AT-kitware.com wrote: The question here is whether one wants to test with the same tools users might use to build the project. If one user's tool doesn't provide per-rule information then we need log-scraping to test it. Except that I contest your premise that no intrinsic per-rule information support implies log scraping. If there is support for the use of replacement tools (cl-wrapper instead of cl), you can also avoid log scraping. My argument is simply that if there *were no way* to get per-rule info from a native build tool then log scraping is necessary. 'course I fully agree that it may be possible to avoid it with sufficient effort for every native tool. Log scraping has been working well enough for us that we've not been motivated to put in this effort. Well, if someone else is maintaining it, I might be convinced not to care whether you do the job by log scraping or by reading tea leaves. ;-) If you can point me at documentation about how to do this in VS I'd love to see it. I know the Intel compiler does it, but that is a full-fledged plugin that even supports its own project file format. We would probably need funding to do something that heavy-weight. I know nothing about how to do that. I think Eric Niebler might be able to help you; he's done this kind of tool integration in the past. Frankly I'm not sure what logfile scraping has to do with the structural problems you've mentioned. I'm only referring to the test part of the anti-logscraping code. The python command wrappers are there to avoid log scraping, Sorry, I'm not up on the details of the system, so I don't know what python command wrappers refers to. Currently in boost every test compilation command line is invoked through a python command that wraps around the real command. In the current system this is necessary to avoid log scraping since the tests are done during the main build. OK. but if the tests were run through CTest then no log scraping would be needed. Now I'm really confused. On one hand, you say it's necessary to run tests through the native toolchains, and that implies log scraping. On the other, you suggest running tests through CTest and say that doesn't imply log scraping. I must be misinterpreting something. Could you please clarify? See approach #2 above. OK. * Boost developers need the ability to change something in their libraries and then run a test that checks everything in Boost that could have been affected by that change without rebuilding and re-testing all of Boost (i.e. incremental retesting). How does the current solution solve that problem (either Boost.Build or the current CMake system)? Boost.Build does it by making test results into targets that depend on successful runs of up-to-date test executables. Test executables are targets that depend on boost library binaries and headers. CTest will need some work to make this totally minimal. Basically it is missing timestamps to avoid re-running tests, which is probably why Troy put the tests into the build in the current system. Prolly. If you can make the changes, I'm on board. As things stand now, the above approaches work as follows. Approach #1 will compile/link test executables during the main build with full dependencies. Approach #2 will drive the individual native build system for every test which has its own dependencies. Both approaches will still run every test executable though. I'm sure we can
Re: [Boost-cmake] Analysis of the current CMake system
on Wed Jan 14 2009, troy d. straszheim troy-AT-resophonic.com wrote: Hi Brad, There is a lot to discuss here. I'll go back later and make specific comments. It'd be great to talk in person at boostcon, (boostcon rocks, by the way.) I understand/agree with a lot of your points (especially bulkiness, and the need to reduce the number of toplevel targets), in most cases because I've learned more about cmake since I implemented what is currently on the boost trunk. Brad King wrote: [snip] In summary, I'd like to help you folks address these issues. Some of the work will be in Boost's CMake code and some in CMake itself. The work will benefit both projects. We can arrange to meet at BoostCon, but we can probably get alot of discussion done on this list before then. BTW, can anyone suggest a preferred format for a BoostCon session from the boost-cmake-devs' point of view? I don't personally see a formal presentation to boost-cmake devs as being useful, there just aren't enough of us (last I checked there were three). Who are you counting? I don't think I've done anything substantial with Boost-CMake but would still be *very* interested in such a talk. -- Dave Abrahams BoostPro Computing http://www.boostpro.com ___ Boost-cmake mailing list Boost-cmake@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-cmake
Re: [Boost-cmake] Analysis of the current CMake system
On Wed, Jan 14, 2009 at 6:16 PM, Doug Gregor doug.gre...@gmail.com wrote: On Wed, Jan 14, 2009 at 3:02 PM, Beman Dawes bda...@acm.org wrote: On Wed, Jan 14, 2009 at 11:52 AM, Brad King brad.k...@kitware.com wrote: .. One of the goals of CMake is to let developers use their favorite native tools. Horrors! As a boost developer, the last thing in the world I want is to have to know anything about a platform's native tools. I just want to be able to enter the CMake equivalent of bjam in the directory I'm testing, and have it build and run all tests for all installed compilers. Perhaps with appropriate arguments if I only want to test with subset of the compilers, or a single test, or pass in some compiler arguments. This is exactly the argument that got us into our current build-tools mess. We've always placed so much emphasis on making things easy for Boost *developers* that we've made them extremely tough for Boost *users*. This feature---the ability to run bjam once and run everything across multiple compilers---is responsible for the majority of the damage, because we've been architecting bjam for multiple compilers at the expense of the common case of a single system compiler. We had this discussion before and decided it would be fine for boost developers if what they were running was actually just a script that just called the underlying CMake setups multiple times. But at the level of reporting, both local and on the web, there has to be a single summary available that brings together all test results. Have you tried helping a Boost newbie go through the process of building and installing Boost lately? It's extremely painful, but we don't see that pain because we've all gone through the initial hurdles of getting bjam setup just right for our own configurations. I certainly see the pain - I'm the one who does the builds for a client of mine. It is a painful mess now, that's for sure. Any progress on pre-built binaries? That's the wrong thing to optimize: we need to optimize for the case where a new user downloads Boost and wants to build/use it immediately. Those users only care about a single compiler---the one they use for their day-to-day work---and would greatly benefit from being able to use their platform-specific tools (Visual Studio, xCode, whatever). True, but they have to build several variants for that compiler, and on some platforms the 32 and 64 bit flavors are more like two separate compilers. If we're going to go through the effort of introducing a new build system, we need to focus on the user experience first and foremost. If you don't give the developers tools they can use, they won't switch to CMake. If you don't give the release managers tools they can use, they won't switch to CMake. And of course the users needs have to be met too. It is a three-legged stool. All three legs have to bear the weight, or it topples. --Beman ___ Boost-cmake mailing list Boost-cmake@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-cmake
Re: [Boost-cmake] Analysis of the current CMake system
on Thu Jan 15 2009, Beman Dawes bdawes-AT-acm.org wrote: Have you tried helping a Boost newbie go through the process of building and installing Boost lately? It's extremely painful, but we don't see that pain because we've all gone through the initial hurdles of getting bjam setup just right for our own configurations. I certainly see the pain - I'm the one who does the builds for a client of mine. It is a painful mess now, that's for sure. Any progress on pre-built binaries? Boostpro Computing provides them for Windows. We've just done the extra work necessary to make it fairly easy to push out a new release, and we're expecting to post a 1.37 release in the next 24 hours. That's the wrong thing to optimize: we need to optimize for the case where a new user downloads Boost and wants to build/use it immediately. Those users only care about a single compiler---the one they use for their day-to-day work---and would greatly benefit from being able to use their platform-specific tools (Visual Studio, xCode, whatever). True, but they have to build several variants for that compiler, and on some platforms the 32 and 64 bit flavors are more like two separate compilers. I think the number of actual end-users who need to build multiple variants is relatively small. Even if a company is shipping multiple versions of a product, end users are typically doing the compile-edit-debug cycle with a single compiler and variant. If we're going to go through the effort of introducing a new build system, we need to focus on the user experience first and foremost. If you don't give the developers tools they can use, they won't switch to CMake. If you don't give the release managers tools they can use, they won't switch to CMake. And of course the users needs have to be met too. It is a three-legged stool. All three legs have to bear the weight, or it topples. Good. Maybe you could be explicit about the needs that, in your view, are brought by each group? -- Dave Abrahams BoostPro Computing http://www.boostpro.com ___ Boost-cmake mailing list Boost-cmake@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-cmake
[Boost-cmake] Analysis of the current CMake system
Hi Folks, I'm considering attending BoostCon 2009 to provide developer-level CMake expertise, and I'm looking into proposing a session as Hartmut requested. In preparation I've downloaded and tried the current system and read back through some of the discussion on this list: http://thread.gmane.org/gmane.comp.lib.boost.cmake/4 http://thread.gmane.org/gmane.comp.lib.boost.cmake/10 The current system feels very bulky compared to CMake-generated build systems I've used for projects of comparable size. This is primarily due to the use of top-level targets for the tests. IMO the efforts to provide test-to-library dependencies and to avoid log-scraping have steered the system off-course. One of the goals of CMake is to let developers use their favorite native tools. These include Visual Studio and Xcode along with Make tools. In order to support these tools CMake's build model separates high-level targets from low-level file dependencies. The add_executable(), add_library(), and add_custom_target() commands create high-level targets. Each target contains file-level rules to build individual sources. In generated VS and Xcode projects the high-level targets become top-level items in the GUIs. These IDEs define file-level rules inside each target. In generated Makefiles there is a two-level system in which the outer level knows only about inter-dependencies among high-level targets and the inner level loads the file-level rules inside each target. This design yields fast builds because each make process sees a relatively small set of rules (and makes automatic dependency scanning easy and reliable). It also makes the representation of build rules in the IDEs tractable. The key is that there should be relatively few high-level targets compared to the number of file-level rules. Currently Boost's CMake system creates about two high-level targets for each test. One is the add_executable() to build the test and the other is the add_custom_target() to run the test. This results in a very large number of high-level targets. The generated Xcode and VS projects are simply too large for the IDEs to load (I waited 10 minutes for VS to try), which already defeats one purpose of using CMake. This leaves the Makefile generators as the only option. The large number of high-level targets places many rules in the outer make level which leads to very long startup times (look at CMakeFiles/Makefile2, which make needs to parse many times). For example, I run time make type_traits-rank_test VERBOSE=1 and get 52.49s user 0.31s system 96% cpu 54.595 total but only about 1s of that time was actually spent running the compiler and linker. Boost needs to build libraries, documentation, and other files to be placed in the install tree. Rules to build these parts can fit in relatively few high-level targets and should certainly use them. This is currently done when not building the tests (BUILD_TESTING=OFF). The problem lies in building the tests. These do not belong in high-level targets for the reasons described above. I'd like to help you create an alternative solution. There are four kinds of tests: boost_test_run boost_test_run_fail boost_test_compile boost_test_compile_fail Let's first consider the run and run_fail tests. In our projects we typically link all tests for a given package into one (or a few) executable(s). The executable's main() dispatches the individual tests based on name. For example, one might manually run the test mentioned above like this: bin/type_traits_tests rank_test This reduces the number of top-level targets in the build to one per library to be tested. It also reduces total link time and disk usage (especially for static linking and when many tests share common template instantiations). However, there are some disadvantages: (a) If one test fails to compile none of its tests can run (b) A bad test may accidentally link due to symbols from another test Problem (a) is not a big deal IMO. If a test doesn't compile the library it tests has a serious problem and needs manual attention anyway. Problem (b) may or may not be a big problem for Boost (it isn't for us). However, there is an alternative tied to the treatement of compile_fail tests. Let's now consider the compile and compile_fail tests. The compile tests could be built into a single executable along with the run and run_fail tests above, but the compile_fail tests cannot. Boost's current solution drives the test compilation by generating an actual compiler command line. It bypasses CMake's knowledge of the native build tools and tries to run its own command line which may not work on all compilers. Furthermore, this is not always representative of how users build programs against boost (they may use CMake, or create their own VS or Xcode project files). CTest provides an explicit feature, its --build-and-test mode, specifically for testing sample external projects built against
Re: [Boost-cmake] Analysis of the current CMake system
Hi Brad, There is a lot to discuss here. I'll go back later and make specific comments. It'd be great to talk in person at boostcon, (boostcon rocks, by the way.) I understand/agree with a lot of your points (especially bulkiness, and the need to reduce the number of toplevel targets), in most cases because I've learned more about cmake since I implemented what is currently on the boost trunk. Brad King wrote: [snip] In summary, I'd like to help you folks address these issues. Some of the work will be in Boost's CMake code and some in CMake itself. The work will benefit both projects. We can arrange to meet at BoostCon, but we can probably get alot of discussion done on this list before then. BTW, can anyone suggest a preferred format for a BoostCon session from the boost-cmake-devs' point of view? I don't personally see a formal presentation to boost-cmake devs as being useful, there just aren't enough of us (last I checked there were three). I'd suggest we just sit down together... there are plenty of conference rooms available at all times. The boost-cmake-for-users talk could of course reflect whatever we get done between now and then. -t ___ Boost-cmake mailing list Boost-cmake@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-cmake
Re: [Boost-cmake] Analysis of the current CMake system
On Wed, Jan 14, 2009 at 3:02 PM, Beman Dawes bda...@acm.org wrote: On Wed, Jan 14, 2009 at 11:52 AM, Brad King brad.k...@kitware.com wrote: .. One of the goals of CMake is to let developers use their favorite native tools. Horrors! As a boost developer, the last thing in the world I want is to have to know anything about a platform's native tools. I just want to be able to enter the CMake equivalent of bjam in the directory I'm testing, and have it build and run all tests for all installed compilers. Perhaps with appropriate arguments if I only want to test with subset of the compilers, or a single test, or pass in some compiler arguments. This is exactly the argument that got us into our current build-tools mess. We've always placed so much emphasis on making things easy for Boost *developers* that we've made them extremely tough for Boost *users*. This feature---the ability to run bjam once and run everything across multiple compilers---is responsible for the majority of the damage, because we've been architecting bjam for multiple compilers at the expense of the common case of a single system compiler. Have you tried helping a Boost newbie go through the process of building and installing Boost lately? It's extremely painful, but we don't see that pain because we've all gone through the initial hurdles of getting bjam setup just right for our own configurations. That's the wrong thing to optimize: we need to optimize for the case where a new user downloads Boost and wants to build/use it immediately. Those users only care about a single compiler---the one they use for their day-to-day work---and would greatly benefit from being able to use their platform-specific tools (Visual Studio, xCode, whatever). If we're going to go through the effort of introducing a new build system, we need to focus on the user experience first and foremost. - Doug ___ Boost-cmake mailing list Boost-cmake@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-cmake
