How is the babeltrace test suite on the master and stable-2.0 branches intended to be used?
Effectively there are four test suites, one for each definition of TESTS in the tests/ directory. If one of them has a failure, 'make check' will not proceed to the next unless the -k/--keep-going flag is used. Is this intended? I unwittingly was running only the tests under tests/cli (some of which always fail) until I discovered this. Defining multiple test suites this way seems to be unusual; other projects I've checked just have one top-level definition of TESTS. I'm aware of the rationale for doing it this way: commit 6ca1931cb32ca2eb33252896d2a42a4c48af436a Author: Philippe Proulx <[email protected]> Date: Fri May 5 16:45:11 2017 -0400 tests: put TESTS list in each Makefile.am So that you can do `make check` in any subdirectory to run only those tests. But it still feels non-idiomatic to me, and I'm wondering if there's a better way. There are other ways to limit the set of tests to run, detailed here: https://www.gnu.org/software/automake/manual/html_node/Parallel-Test-Harness.html Here are my current results from running all test suites; let me know if you want more detail and I'll follow up; there are likely several different issues to investigate. $ find tests/ -name test-suite.log -exec head {} + ==> tests/lib/test-suite.log <== ===================================================== babeltrace 2.0.0-pre1: tests/lib/test-suite.log ===================================================== # TOTAL: 1200 # PASS: 1195 # SKIP: 0 # XFAIL: 0 # FAIL: 1 # XPASS: 0 ==> tests/bindings/python/bt2/test-suite.log <== ===================================================================== babeltrace 2.0.0-pre1: tests/bindings/python/bt2/test-suite.log ===================================================================== # TOTAL: 11152 # PASS: 0 # SKIP: 0 # XFAIL: 0 # FAIL: 11151 # XPASS: 0 ==> tests/cli/test-suite.log <== ===================================================== babeltrace 2.0.0-pre1: tests/cli/test-suite.log ===================================================== # TOTAL: 153 # PASS: 143 # SKIP: 0 # XFAIL: 0 # FAIL: 7 # XPASS: 0 ==> tests/plugins/test-suite.log <== ========================================================= babeltrace 2.0.0-pre1: tests/plugins/test-suite.log ========================================================= # TOTAL: 2 # PASS: 0 # SKIP: 0 # XFAIL: 0 # FAIL: 0 # XPASS: 0 (BTW, since the tests don't pass/skip/xfail 100%, 'make distcheck' is failing.) _______________________________________________ lttng-dev mailing list [email protected] https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
