On 2015-11-04, Kornel Benko wrote:
> Am Mittwoch, 4. November 2015 um 09:30:03, schrieb Guenter Milde
> <mi...@users.sf.net>
>> On 2015-11-02, Kornel Benko wrote:
>> > Am 2. November 2015 um 08:36:05, schrieb Guenter Milde <mi...@users.sf.net>
>> >> On 2015-11-02, Scott Kostyshak wrote:
>> >> > On Sun, Nov 01, 2015 at 10:36:17PM +0100, Kornel Benko wrote:
>> >> >> Am 1. November 2015 um 21:27:11, schrieb Guenter Milde
>> >> >> > Could we introduce a new status "suspended" - meaning just skip the
>> >> >> > test until a known bug is solved as we know the result is
>> >> >> > insignificant until then?
...
>> >> Candidates for "suspended" tests are
>> >> * "fragile" documents (many packages, ERT, heavy preamble, ...),
>> >> * "fragile" export routes (XeTeX/LuaTeX with TeX fonts),
>> >> * non-default export routes
>> >> and especially combinations of these.
>> > OK, here is my suggestion
>> > 1.) We add the appropriate tests into revertedTests
>> Why? This would undo one important benefit of "suspended" tests:
> I was not clear about this. These tests will not be run with e.g.
> 'ctest -L export' (see the '-L' param). In this sense we get 'clean'
> export tests.
May be I was not clear either:
In my view, suspension is orthogonal to reversion:
- normal: we want the test to pass
revert: we want the test to fail
- normal: run the test
suspend: skip the test temporarily
ignore: skip the test permanently
In other words: while "normal" vs. "reverted" will be used to specify the
expected test result, ignore and suspend will be used if we established
the reason why a test currently gives the wrong result but cannot
immediately solve the issue.
Then we have
* suspended reverted tests: should fail but currently passes, e.g.
wrong output/dataloss that does not lead to error.
* suspended normal tests: should pass but currently fails, e.g.
ru/Intro.lyx with XeTeX+TeX-fonts
With the benefit:
>> >> Suspending instead of reverting frees us from the need to
>> >> reassess them if a change in the "real life documents" or a fix
>> >> makes them pass again. Instead, they could/should be revised at a
>> >> time we have fixed major known problems and no release
>> >> pressure...
...
> So what do you propose for such tests (84% of export tests)?
> ctest -L export -N | wc => 3719
> ctest -L export -N |egrep '/(doc|examples)/'| wc => 3153
> 3153 / 3719 => 84.78%
I don't have any clue which tests are hidden behind these commands. The
general recipe is:
* inspect the file,
* run the export command "by hand",
* try to find the reason for the test failure result
If the current "inversion state" does not match the expected test outcome
(pass/fail), correct it.
If it is "easyfix", fix.
If fixing »the right way« takes time, suspend (stating the established
reason).
The idea is, to de-couple the test analysis (from failure to bug report)
from the repair of found problems. This will hopefully allow us
to get the test suite clean without hackish quick-fixes.
> For now committed 1.b at 538228d
Thanks
Günter
