On 2016-09-23, Kornel Benko wrote: > Am Freitag, 23. September 2016 um 07:29:14, schrieb Guenter Milde > <mi...@users.sf.net>
... > If we could considerably reduce the number of failings, we can also > omit regexes and instead use full test names. 1. We cannot significantly reduce the number of failings. This may have worked for the original ~300 test cases but now we have 5407. Even with the same failure-rate we would get 10 times more problematic cases. In addition, there is a large number of failing exports due to the non-standard export routes tested but not supported by LaTeX for several packages and document classes. (We may ignore them if we do not expect future support.) See below for statistics. 2. Full test names would not solve the problem of a test case with two (or more) problems making it both, unreliable and failing. > Yes, unfortunately we cannot make tests selective to specific failure. There is also no need to do this. However, we could create independent labels for unreliable and inverted tests or just "allow" matches in both files (actually, there is nothing in the documentation telling that this is not allowed). Statistics: Currently, we have 265 problematic tests (ctest -N -L "inverted|suspended|unreliable") 146 tests that are known to fail, including 57 tests requiring attention -L todo 14 tests with a track number -L lyxbugs 61 tests we cannot fix -L texissues 119 unreliable tests, including 62 tests with extra requirements -L nonstandard 46 passing test with wrong output -L wrong_output 10 with result depending on TeXLive version -L varying_versions The 57 "todo" tests are to be investigated and sorted -- only a few of them will be "easyfix". This means we have about 200 problematic test cases that will stay so for a longer time and should adapt to this. Günter
