On 2015-11-04, Kornel Benko wrote: > [-- Type: text/plain, Encoding: 7bit --]
> Am Mittwoch, 4. November 2015 um 14:36:19, schrieb Vincent van Ravesteijn > <v...@lyx.org> >> > In my view, suspension is orthogonal to reversion: >> > >> > - normal: we want the test to pass >> > revert: we want the test to fail >> You mean: revert is that "it is known to fail", but we haven't fixed it yet. > Yes. We hope to do it eventually. We have also tests that correctly fail: "it is known to fail", there is nothing to fix, and we must ensure that it continues to fail. The third point is, what prevents us from ignoring this test. >> > - normal: run the test >> > suspend: skip the test temporarily >> > ignore: skip the test permanently >> > >> Suspending means: "The outcome is noisy, so skip it until someone >> looks into it and makes the test better." > Sort of, if we are careful enough about which test should go there. Not only, with "suspending" I also mean "The outcome is of no value for finding new bugs or regressions until someone solves the known bug ...". However, we usually know what the outcome should be if the bug is solved: if the expected outcome is "pass", this test should not be inverted. > For the actual committed cmake build, suspended test means: > a.) Test is one of export tests > b.) Test is failing, and therefore it is part of revertedTests > c.) In 'normal' use like 'ctest -L export' it is hidden > d.) Test gets the ctest-label 'suspended' > You can run all suspended tests with 'ctest -L suspended'. > Or, for a known testname (say xyzzy) you can use 'ctest -R xyzzy'. >> Ignore: Skipping a test permanently is the same as just removing the >> test.. ?? > This is effectively the same here. Because the testcase should be > ignored (it is part of ignoredTests) it will not be added with > add_test(). E.g. it is not known to ctest. However, as the test rules are something like: "test exporting all our manuals to PDF (pdflatex)", we need to specify cases where a manual is known not to compile, e.g. because it relies on non-TeX fonts or special features of LuaTeX. Günter
