Dear Scott and Kornel, On 2015-11-02, Scott Kostyshak wrote: > On Sun, Nov 01, 2015 at 10:36:17PM +0100, Kornel Benko wrote: >> Am Sonntag, 1. November 2015 um 21:27:11, schrieb Guenter Milde >> <mi...@users.sf.net>
>> > Could we introduce a new status "suspended" - meaning just skip the >> > test until a known bug is solved as we know the result is >> > insignificant until then? > As Kornel mentions, we have "ignoredTests" for this. Not exactly, see below. >> We already have such (specified in file "ignoredTests"). But as this >> tests are never executed, nobody cares for them anymore. >> The tests here are such, that we know, we never resolve them. > I agree. We should not use ignored tests for temporary issues. Once we > put something in ignoredTests chances are strong that we will forget > about them for a long time. This is why I said "suspended": Suggestion: Specified in file "suspendedTests") with the reason for suspending (bug report, commit that turned hidden problems into export failure, ...) These tests are normally skipped, but they are not forgotten. The tests here are such, that we know, we can resolve them but their failure is a minor issue that can be postponed (comparable to enhancement requests in Trac). Suspending instead of reverting also frees us from the need to reassess them if a change in the "real life documents" or a fix makes them pass again. Instead, they could/should be revised at a time we have fixed major known problems and no release pressure... Candidates for "suspended" tests are * "fragile" documents (many packages, ERT, heavy preamble, ...), * "fragile" export routes (XeTeX/LuaTeX with TeX fonts), * non-default export routes and especially combinations of these. Günter
