On Tue, Aug 23, 2016 at 4:00 PM, Rolf Eike Beer <e...@sf-mail.de> wrote:
> Am Dienstag, 23. August 2016, 10:06:01 schrieb Craig Scott: > > Cheeky way to get me more involved in contributing, but okay, I'll bite. > ;) > > Switching discussion to the dev list. > > > > So how would you want the feature to work? I'd suggest an initial set of > > requirements something like the following: > > > > - Need to support the ability to define multiple setup and/or tear > down > > tasks. > > - It should be possible to specify dependencies between setup tasks > and > > between tear down tasks. > > - Individual tests need to be able to indicate which setup and/or tear > > down tasks they require, similar to the way DEPENDS is used to specify > > dependencies between test cases. > > - When using ctest --rerun-failed, ctest should automatically invoke > any > > setup or tear down tasks required by the test cases that will be > re-run. > > - Setup or tear down tasks which reference executable targets should > > substitute the actual built executable just like how > add_custom_command() > > does. > > -need a way to mark if 2 tests with the same setup/teardown can share > those or > if they need to run for every of them > -the default for each test is "no s/t", which means it can't be run with > any > of the above in parallel (especially for compatibillity)[1] > -need a way to tell if a test doesn't care about those > > 1) think of a database connector test: the test that will check what > happens > if no DB is present will fail if the setup step "start DB" was run, but not > the teardown > So maybe that requires being able to specify that tests for resource XXX and resource YYY cannot be executed concurrently. Maybe that's a separate change that could be made independent of this proposed improvement, since it would apply even for existing CMake functionality. I see the value, I'm just trying to sort out what is really needed from what is nice-to-have but could be done as a subsequent improvement later. > > > Some open questions: > > > > - Should setup and tear down tasks be defined in pairs, or should they > > completely independent (this would still require the ability to > specify a > > dependency of a tear down task on a setup task)? > > The test could be "shutdown daemon" or "delete files" so I would keep them > separated. They may be created by the same command, so they could be > batched > anyway. > Agreed, it seems clear now that keeping them separate is preferable. > > > - Should the setup and tear down tasks be defined by a new CTest/CMake > > command or extend an existing mechanism (e.g. add_custom_command())? > > Don't clutter existing commands more than needed. If it's something new, > then > create a new command (they could still share C++ code). If it's basically > the > same as anything existing at the end then use that. > See my other email reply just now. I think re-using the existing commands and concepts and adding the RESOURCE_SETUP and RESOURCE_CLEANUP test properties might be the most seamless from an end user perspective. I might change my mind once I dig into the CMake source code though. ;) > > > - If no test case has a dependency on a setup or tear down task, > should > > that task be skipped? Perhaps tasks need to have a flag which > indicates > > whether they always run or only if a test case depends on it. > > Keep backward compatibility. I.e. if I now add a new test with s/t, then > every > other test should still run (and succeed) as before. > Definitely. Existing projects should receive zero impact from any changes made. New functionality should be opt-in. > > > - What terminology to use? Things like GoogleTest use terms like test > > *fixtures* for this sort of thing. The terms setup and tear down are a > > bit imprecise and cumbersome, so we would probably need something > better > > than those. > > - Would it make sense for the ctest command line to support disabling > > setup and/or tear down steps? I can see some potential scenarios where > > this may be desirable, but maybe this is getting too ambitious for a > > starting set of requirements. > > IMHO that doesn't make sense. One could think about an option to disable > the > s/t merging, i.e. that they are really called alone for every test. > To reduce complexity, I'm gravitating that way too. If you define a setup/cleanup task, then why allow disabling it? If developers really want that, they could wrap defining the setup/cleanup definitions inside an if() block controlled by a CMake option or something similar. > > > - What should happen if a setup or tear down task fails? How would > > failure be detected? How would such failures impact things like a > CDash > > test report, etc.? > > Then the test fails, just the same as it now does when it can't find the > executable. > Seems sensible. -- Craig Scott Melbourne, Australia http://crascit.com
-- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers