commit 3ec76c472ee4a51e727d4c2f03050c9494dead6c Author: Günter Milde <mi...@lyx.org> Date: Mon Jan 18 10:25:23 2016 +0100
ctest autotest documentation update diff --git a/lib/doc/Development.lyx b/lib/doc/Development.lyx index fec0266..9e14346 100644 --- a/lib/doc/Development.lyx +++ b/lib/doc/Development.lyx @@ -1462,6 +1462,12 @@ reference "subsec:Interpreting-export-tests" \end_layout \begin_layout Paragraph +\begin_inset CommandInset label +LatexCommand label +name "par:ctest-options" + +\end_inset + Running the tests \end_layout @@ -1556,6 +1562,17 @@ status collapsed \end_inset Exclude the tests whose labels match the given regular expression. + Cannot be combined with +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +-L +\end_layout + +\end_inset + +. \end_layout \begin_layout Standard @@ -1613,7 +1630,7 @@ print-labels \end_layout \begin_layout Standard -Other relevant options are: +Other useful options are: \end_layout \begin_layout Labeling @@ -1914,45 +1931,6 @@ A good question is why do we enable the tests for non-default formats? The \end_layout \begin_layout Standard -What action should you take if a test fails? This depends: -\end_layout - -\begin_layout Standard -Generally, if a change breaks compilation for the target format (for the - manuals pdf2) without solving some important other issue, fix or revert - the commit that led to failure. - If a change breaks compilation for some non-target format (for the manuals - everything except pdf2), invert the test. - -\end_layout - -\begin_layout Standard -A special case is given, if the export was succeeding before but showing - garbled text in the PDF output. - Try to establish, that when the compilation succeeded before the resulting - PDF was good. - Otherwise, it is in fact an improvement when a test fails. - Now it might fail with a clear message of "language xyz not supported". - It is always good to check manually why something fails and if it passes - if the PDF output is good. -\end_layout - -\begin_layout Standard -Sometimes a test is fixed as side-effect of some change. - We should uninvert a test (remove it from the -\begin_inset Flex Code -status collapsed - -\begin_layout Plain Layout -suspiciousTests -\end_layout - -\end_inset - - file) in order to preserve the fix. -\end_layout - -\begin_layout Standard When a test or several tests fail, consider checking the files in the \begin_inset Flex Code status collapsed @@ -2009,27 +1987,258 @@ CTestCostData.txt \end_layout \begin_layout Paragraph +What action should you take if a test fails? +\end_layout + +\begin_layout Standard +\paragraph_spacing single +It is always good to check manually why something fails and if it passes + if the PDF output is good. +\end_layout + +\begin_layout Itemize +Generally, if a change breaks compilation for the target format (for the + manuals pdf2) without solving some important other issue, +\emph on +fix or revert the commit +\emph default + that led to failure. +\end_layout + +\begin_layout Itemize +If it is not possible to (immediately) fix the failure but there are reasons + not to revert the commit (e.g. + it fixes another more important issue), +\emph on +invert +\emph default + the failing test case (see +\begin_inset CommandInset ref +LatexCommand ref +reference "par:Inverted-tests" + +\end_inset + +). +\end_layout + +\begin_layout Itemize +If an +\emph on +inverted +\emph default + test case fails because the export now works, +\emph on +uninvert +\emph default + the test by removing the labeling pattern from +\begin_inset Quotes eld +\end_inset + +suspiciousTests +\begin_inset Quotes erd +\end_inset + +) (see +\begin_inset CommandInset ref +LatexCommand ref +reference "par:Inverted-tests" + +\end_inset + +). +\end_layout + +\begin_layout Itemize +If the export did not fail previously but led to wrong output (PDF, say), + it is in fact an improvement when the test now fails, label it as +\begin_inset Quotes eld +\end_inset + +unreliable:wrong:output +\begin_inset Quotes erd +\end_inset + + ( +\begin_inset CommandInset ref +LatexCommand ref +reference "par:Unreliable-tests" + +\end_inset + +). +\end_layout + +\begin_layout Paragraph +\begin_inset CommandInset label +LatexCommand label +name "par:Inverted-tests" + +\end_inset + Inverted tests \end_layout \begin_layout Standard -These tests fail if the export does +Test cases whose name matches a pattern in the file +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +development/autotests/suspiciousTests +\end_layout + +\end_inset + + get the label \emph on -not +inverted \emph default - return an error. - +. + They get also the test property +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +WILL_FAIL +\end_layout + +\end_inset + +, i.e. + they are reported as failing if the export works without error +\begin_inset Flex URL +status collapsed + +\begin_layout Plain Layout + +https://cmake.org/cmake/help/v3.0/command/set_tests_properties.html +\end_layout + +\end_inset + +. +\end_layout + +\begin_layout Standard +Add failing cases to this file, if they cannot be solved +\begin_inset Quotes eld +\end_inset + +immediately +\begin_inset Quotes erd +\end_inset + + but it is expected that the export will work in a foreseeable future, e.g. + low priority issues like failures to export to a non-target format (for + the manuals everything except pdf2). +\end_layout + +\begin_layout Standard +The following sublabels are currently present in +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +suspiciousTests +\end_layout + +\end_inset + +: \end_layout \begin_layout Description -inverted Export fails, but the test cases are subject to be examined. - It is expected that the export will work in a foreseeable future. - They are labeled 'inverted'. +todo test failures that require attention: +\end_layout + +\begin_deeper +\begin_layout Itemize +minor issues to explore and properly sort later, +\end_layout + +\begin_layout Itemize +easyfix issues, +\end_layout + +\begin_layout Itemize +LyX bugs to report at trac (move pattern to section "lyxbugs" once done). +\end_layout + +\end_deeper +\begin_layout Description +lyxbugs LyX bugs with a Trac number. +\end_layout + +\begin_layout Description +ert Export failures due to "raw" LaTeX use in ERT or preamble code. +\end_layout + +\begin_deeper +\begin_layout Standard +"Wontfix" if demonstrating correct use and OK in the default output format. +\end_layout + +\end_deeper +\begin_layout Description +texissues Export fails due to LaTeX limitations like non-ASCII characters + in verbatim or listings, incompatible packages, ... +\end_layout + +\begin_deeper +\begin_layout Standard +"Wontfix" if documents demonstrate correct use in the default output format: +\end_layout + +\begin_layout Itemize +If the source can be made more robust without becoming "hackish", fix the + source, +\end_layout + +\begin_layout Itemize +if LyX could be enhanced to care for a permanent TeX limitation, file a + ticket at trac and add a pattern under lyxbugs, +\end_layout + +\begin_layout Itemize +otherwise, add a pattern here. \end_layout +\end_deeper \begin_layout Description -suspended Some inverted tests are labeled 'suspended'. - This means, they are not executed using +attic Documents in the attic. + (Kept for reference and format conversion test.) +\end_layout + +\begin_layout Subparagraph +suspended +\end_layout + +\begin_layout Standard +Test cases whose name additionally matches a pattern in the file +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +development/autotests/suspendedTests +\end_layout + +\end_inset + + get the label +\emph on +suspended +\emph default +(instead of +\emph on +export +\emph default +and +\emph on + inverted +\emph default +). + This means they are not executed using \begin_inset Flex Code status collapsed @@ -2050,6 +2259,18 @@ ctest -L inverted \end_inset . + However, they also get the test property +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +WILL_FAIL +\end_layout + +\end_inset + +, i.e. + they are reported as failing if the export works without error. From time to time they still have to be checked using \begin_inset Flex Code status collapsed @@ -2063,7 +2284,6 @@ ctest -L suspended . \end_layout -\begin_deeper \begin_layout Standard These tests are suspended, because the export fails for known reasons which cannot ATM be resolved. @@ -2084,16 +2304,46 @@ status collapsed \end_inset parameter nothing changes. - Suspended or not, tests will be executed depending only on the regexes - parameters given to the ctest command. + Suspended or not, tests will be executed depending only on the selectiong + regular expression given to the ctest command (see +\begin_inset CommandInset ref +LatexCommand ref +reference "par:ctest-options" + +\end_inset + +). \end_layout -\end_deeper \begin_layout Paragraph +\begin_inset CommandInset label +LatexCommand label +name "par:Unreliable-tests" + +\end_inset + Unreliable tests \end_layout \begin_layout Standard +Test cases whose name matches a pattern in the file +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +development/autotests/unreliableTests +\end_layout + +\end_inset + + get the label +\emph on +unreliable +\emph default +. +\end_layout + +\begin_layout Standard These tests are not executed using \begin_inset Flex Code status collapsed @@ -2121,11 +2371,24 @@ ctest -L inverted \begin_layout Standard They pass or fail for various reasons not related to LyX (nonstandard, erratic) or pass but should rather fail (wrong output). + +\begin_inset Note Note +status collapsed + +\begin_layout Plain Layout +*invalid* tests (wrong output) are not *unreliable*. + # Use "unfit" or "unapplicable" as better label and name of pattern file? + +\end_layout + +\end_inset + + \end_layout \begin_layout Description nonstandard Documents with additional requirements, e.g. - a class or package file not on CTAN. + a class or package file not in TeXLive. \begin_inset Note Note status open @@ -2159,7 +2422,7 @@ TODO: use \emph on erratic \emph default - only for the phase-of-moon dependency. + only for the phase-of-moon dependency? \end_layout \end_inset @@ -2186,6 +2449,11 @@ output Export does not fail but the resulting document has errors. \begin_deeper \begin_layout Standard \paragraph_spacing single +\begin_inset Note Note +status open + +\begin_layout Plain Layout +\paragraph_spacing single These tests are actually not \emph on unreliable @@ -2197,6 +2465,11 @@ invalid (not measuring what they should measure). \end_layout +\end_inset + + +\end_layout + \end_deeper \begin_layout Paragraph Export test filtering @@ -3223,40 +3496,6 @@ Associated test results can be examined in ctest-log directory in files of the form 'LastFailed.*URLS.log' \end_layout -\begin_layout Subsubsection -Test labels -\end_layout - -\begin_layout Standard -ctest label commands: -\end_layout - -\begin_layout Description -\SpecialChar nobreakdash -\SpecialChar nobreakdash -print-labels shows all assigned labels -\end_layout - -\begin_layout Description -\SpecialChar nobreakdash -L -\begin_inset space ~ -\end_inset - -labelname executes all tests to which this label is assigned to. - A test may have more that one label. -\end_layout - -\begin_layout Description -\SpecialChar nobreakdash -j -\begin_inset space ~ -\end_inset - -number executes tests in parallel using 'number' simultaneously processes. - Some tests are marked as 'sequencial', for them this parameter has no effect. -\end_layout - \begin_layout Section Development policies \end_layout