Am Mittwoch, 4. November 2015 um 15:36:45, schrieb Guenter Milde <mi...@users.sf.net> > On 2015-11-04, Kornel Benko wrote: > > > [-- Type: text/plain, Encoding: 7bit --] > > > Am Mittwoch, 4. November 2015 um 14:36:19, schrieb Vincent van Ravesteijn > > <v...@lyx.org> > >> > In my view, suspension is orthogonal to reversion: > >> > > >> > - normal: we want the test to pass > >> > revert: we want the test to fail > > >> You mean: revert is that "it is known to fail", but we haven't fixed it > >> yet. > > > Yes. We hope to do it eventually. > > We have also tests that correctly fail: > "it is known to fail", there is nothing to fix, and we must ensure that it > continues to fail. > The third point is, what prevents us from ignoring this test. > > >> > - normal: run the test > >> > suspend: skip the test temporarily > >> > ignore: skip the test permanently > >> > > > >> Suspending means: "The outcome is noisy, so skip it until someone > >> looks into it and makes the test better." > > > Sort of, if we are careful enough about which test should go there. > > Not only, with "suspending" I also mean "The outcome is of no value for > finding new bugs or regressions until someone solves the known bug ...".
OK.
> However, we usually know what the outcome should be if the bug is solved: if
> the expected outcome is "pass", this test should not be inverted.
>
Here we disagree. Matter of taste I suppose.
For me the test fails _now_. We don't care now (because we know what's going on
etc.).
Therefore the test is to be inverted as to not catch unwanted attention.
> > For the actual committed cmake build, suspended test means:
> > a.) Test is one of export tests
> > b.) Test is failing, and therefore it is part of revertedTests
> > c.) In 'normal' use like 'ctest -L export' it is hidden
> > d.) Test gets the ctest-label 'suspended'
> > You can run all suspended tests with 'ctest -L suspended'.
> > Or, for a known testname (say xyzzy) you can use 'ctest -R xyzzy'.
>
> >> Ignore: Skipping a test permanently is the same as just removing the
> >> test.. ??
>
> > This is effectively the same here. Because the testcase should be
> > ignored (it is part of ignoredTests) it will not be added with
> > add_test(). E.g. it is not known to ctest.
>
> However, as the test rules are something like: "test exporting all our
> manuals to PDF (pdflatex)", we need to specify cases where a manual is
> known not to compile, e.g. because it relies on non-TeX fonts or special
> features of LuaTeX.
I was describing the machinery how our ctests currently works. Not politics,
that is how we organize
our config files (ignoredTests, revertedTests, suspendedTests).
> Günter
>
Kornel
signature.asc
Description: This is a digitally signed message part.
