On 09/14/2017 11:33 PM, Edward Diener wrote: > Boost Build has tests for running an application successfully or not, > for compiling one or more source files successfully or not, and for > building one or more source files into an exe or not. These tests in > Boost Build are the run/run-fail, compile/compile-fail, and > link/link-fail rules.
Good summary. One major difference is that Boost Build directly controls the entire build graph and is natively aware of expected failure cases. CMake needs to generate an external build system e.g. as Visual Studio `.vcxproj` files with no good way to represent expected failure directly. This is a technical reason CMake has never gained this functionality, but perhaps it could be overcome with a careful design. See below. > CMake/CTest has the exact equivalent to the run/run-fail rule in its > add_test framework While `add_test` can do run/run-fail it doesn't do so given just a source file like Boost Build does. `add_test` expects to be given a command referring to some executable that already exists or is built elsewhere in the project via `add_executable`. Therefore some work would be needed even to reproduce Boost's run/run-fail infrastructure. > Hopefully CMake developers will get the message and make the necessary > update to CMake/CTest. We understand the value of the tests but simply don't have the need for them in our own use cases. This is a non-technical reason CMake does not have this functionality, but it doesn't stand in the way of others contributing a solution. > Is there any consensus that these other two types of tests might be > valuable for CMake/CTest, or any way to make this happen ? I agree they would be useful. >From some quick brainstorming, one idea to deal with the technical challenge raised above is to add a new "BUILD_TEST" target property to tell CMake that a target is meant only to be built during testing. We allow dependencies among BUILD_TEST targets on each other and on normal targets, but not from normal targets to BUILD_TEST targets. That will give a separable part of the build graph to generate only for testing. Then add_test can create tests that try to build one of the BUILD_TEST targets and expect one of the compile/compile-fail, link/link-fail, or run/run-fail cases. For generators like Visual Studio we could lave the BUILD_TEST targets out of the main `.sln` file so they wouldn't show up when users load the projects. Instead they would be placed in a second `.sln` file used only for testing. Further design work is likely needed but this is a start. --------------------------------------------------------------------- IMO it will be better to first prototype this externally to gain some experience with it and work with existing CMake releases. One can do this by using the `export()` command to make all targets in the build tree available to externally-built projects (which is useful on its own). With that a few approaches can be used to drive testing as an external build. Then have a custom target that configures the test-only part of the project externally but doesn't build it. The test-only project is externally-built and can load the targets file created by `export()` above. Then use `add_test` to create tests that drive the build on parts of the external test-only project and check for appropriate success/failure and output matching. -Brad -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
