On Monday 27 June 2011, Stefano Lattarini wrote:
>
> Attached is the squash-in with the fixes you suggested. I hope to
> be soon able to finish up this part of the documentation; I'll post
> and additional squash-in when I'm done.
>
OK the documentation is basically complete now (it still lacks a
working example I'd like to add in the `doc/' subdir, similar to
what is done with the "amhello" package). I've pushed it to the
temp branch; attached is the diff w.r.t. the previous status, for
reference.
Regards,
Stefano
diff --git a/ChangeLog b/ChangeLog
index 6acd090..628d879 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,4 +1,4 @@
-2011-06-25 Stefano Lattarini <stefano.lattar...@gmail.com>
+2011-06-27 Stefano Lattarini <stefano.lattar...@gmail.com>
docs: document custom test drivers and protocols
* doc/automake.texi (Simple Tests): Note that the TESTS_ENVIRONMENT
@@ -10,10 +10,12 @@
(Declaring Custom Test Drivers in @file{Makefile.am}): Likewise.
(APIs for Custom Test Drivers): Likewise.
(Options): Update description of color-tests.
- * lib/am/check (recheck, recheck-html): Minor adjustments to better
- conform to the documentation (this should cause no semantic changes
- w.r.t. the former behaviour); minor improvements and extensions to
- existing comments.
+ * lib/am/check ($(TEST_SUITE_LOG)): Remove comments that have been
+ moved into the manual.
+ (recheck, recheck-html): Minor adjustments to better conform to the
+ documentation (this should cause no semantic changes w.r.t. the
+ former behaviour); minor improvements and extensions to existing
+ comments.
* tests/test-driver-create-log-dir.test: New test.
* tests/test-driver-strip-vpath.test: Likewise.
* tests/test-driver-global-log.test: Likewise.
diff --git a/doc/automake.texi b/doc/automake.texi
index 8a6871c..39192ba 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -9119,9 +9119,9 @@ for new features or to satisfy additional portability requirements.
The main characteristic of these APIs is that they are designed to share
as much infrastructure, semantics, and implementation details as possible
-with the plain @option{parallel-tests} harness. So everything said in
-the previous section should apply here too, unless explicitly stated
-otherwise.
+with the @option{parallel-tests} harness and its default driver. So
+everything said in the previous section should apply here too, unless
+explicitly stated otherwise.
@menu
* Command-line arguments for test drivers::
@@ -9135,9 +9135,10 @@ otherwise.
A custom driver can rely on various command-line options and arguments
being passed to it automatically by the Automake's @option{parallel-tests}
-harness. It is @emph{mandatory} that it understands all of them (even if
-the exact interpretation of the associated semantics can change between a
-test driver and another, and even be a no-op in some drivers).
+harness. It is @emph{mandatory} that it understands all of them (even
+if the exact interpretation of the associated semantics can legitimately
+change between a test driver and another, and even be a no-op in some
+drivers).
@noindent
Here is the list of options:
@@ -9151,10 +9152,10 @@ results (@pxref{Console output during testsuite runs}).
@item --log-file=@var{PATH}
The log file the test driver must create. If it has a directory component
(as in e.g., @file{sub/foo.log}), the Automake harness will ensure that
-such directory exist before the test driver is called.
+such directory exist @emph{before} the test driver is called.
@item --color-tests=@{yes|no@}
-Whether the console output should be colored or not (@pxref{Simple
-tests and color-tests} to learn when this option gets activated and
+Whether the console output should be colorized or not (@pxref{Simple
+tests and color-tests}, to learn when this option gets activated and
when it doesn't).
@item --expect-failure=@{yes|no@}
Whether the tested program is expected to fail.
@@ -9181,23 +9182,14 @@ consistency and a more pleasant user experience.
@node Log files generation and test results recording
@subsubsection Log files generation and test results recording
-One of the main features of the new testsuite harness is the ability to
-support test protocols that allow a single test script to run more
-test cases, @emph{each with its distinct result}. In order for the
-testsuite summary report to be correct in this case, the test driver
-implementing support for the protocol must follow the following
-convention.
-
-TODO: In order to have a correct count of test cases in the final
-testsuite summary, the driver must use special syntax in the created
-.log file (see below, `:test-result:').
-
The test driver must correctly generate the file specified by the
-@option{--log-file} option (even when the tested program fails, of
-course!). This file is quite free-form, but still it has to conform
-to the following conventions:
+@option{--log-file} option (of course even when the tested program
+fails or crashes). This file is quite free-form, but still it has
+to conform to the following conventions, in order to work with the
+@option{parallel-tests} harness and take advantage of its features.
@itemize @bullet
+
@item
@cindex Global test script result
@cindex Test result, global
@@ -9209,9 +9201,10 @@ in the first line of the @file{.log} file.
If this line does not begin with either @code{PASS:}, @code{XFAIL:} or
@code{SKIP:}, the test script will be considered failed, and it will be
re-run by @code{make recheck}. Also, if this line does not begin with
-either @code{PASS:} or @code{XFAIL:}, the content of and its @file{.log}
-file will be copied into the global @file{test-suite.log} (in order to
-help in debugging and bug-report analysis).
+either @code{PASS:} or @code{XFAIL:}, the content of the @file{.log} file
+will be copied into the global @file{test-suite.log} (in order to help in
+debugging and bug-report analysis).
+
@item
@cindex Single test case result
@cindex Test result, single
@@ -9220,67 +9213,90 @@ help in debugging and bug-report analysis).
One of the main features of the new testsuite harness is the ability to
support test protocols that allow a single test script to run more
test cases, @emph{each with its distinct result}. In order for the
-testsuite report to be correct in this case, the test driver must
+testsuite summary to be correct in this case, the test driver must
register @emph{each} such result in the generated @file{.log} file, using
-the @code{:test-result:} reStructuredText field.
-
-FIXME: currently, that field is recognized anywhere in the @file{.log}
-file; this is possibly prone to generating spurious results, in case
-of verbose tests. We should allow a special @code{:test-result:} that
-stops any further parsing (maybe a line @code{:test-result:END} will
-do?).
-
-FIXME: we are probably going to change the current semantics of
-testsuite summary generation, in order to allow test protocols with
-only one result per test case not to be forced to also declare that
-result with @code{:test-results:}; that would be a repetition, since
-the result is already declared in the @file{.log} first line.
+the @code{:test-result:} reStructuredText field. Otherwise, only the
+global test result will be considered (as it is the case of the default
+@option{parallel-tests} driver).
+
+@c Keep this in sync with lib/am/check-am:$(TEST_SUITE_LOG).
+The only recognized test results are currently @code{PASS}, @code{XFAIL},
+@code{SKIP}, @code{FAIL} and @code{XPASS}. These results, when declared
+with @code{:test-result:}, can be optionally followed by text holding
+the name and/or a brief description of the corresponding test; the
+@option{parallel-tests} harness will ignore such extra text when
+generating @file{test-suite.log} and preparing the testsuite summary.
+
+For example, if a @file{.log} file contains the following lines:
+@example
+:test-result: PASS server starts
+:test-result: PASS HTTP/1.1 request
+:test-result: FAIL HTTP/1.0 request
+:test-result: SKIP HTTPS request (TLS library wasn't available)
+:test-result: PASS server stops
+@end example
+it means that the corresponding test script contributes with five test
+results to the testsuite summary (three of these tests being successful,
+one failed, and one skipped).
+
+@c FIXME: currently, the @code{:test-result:} field is recognized anywhere
+@c in the @file{.log} file; this is possibly prone to generating spurious
+@c results, in case of verbose tests. We should allow a special
+@c @code{:test-result:} that stops any further parsing (maybe a line
+@c @code{:test-result:END} will do?).
+
@end itemize
-@emph{TODO}: Explain why supporting the Global test result declaration
-in the first line is useful also for test protocols that allow more
-test cases per test script.
-
-@c
-@c * A test driver is free (but not forced) to register multiple test
-@c results for a single test script, through the use of a custom
-@c reStructuredText field ``test-result''. Each of this results will
-@c be accounted for separately in the final testsuite summary;
-@c TODO: give an example
-@c TODO: a special field to terminate parsing of .log?
-@c But note that the task of reporting this results on the screen is
-@c left to the individual test driver though; more about this below.
-@c For example, if the test script executes
-@c two test cases, one successful and one failing, and skip another
-@c test case, the driver should end up writing the following in the
-@c test log:
-@c :test-result: PASS [name of passed test, and/or details about it]
-@c :test-result: FAIL [name of failed test, and/or details about it]
-@c :test-result: SKIP [name of skipped test, and/or details about it]
-@c The above lines (each of which *must* be followed by a blank line
-@c in order for the HTML output generation to work) are used only
-@c when generating the `test-suite.log' from the individual test
-@c logs, and can be placed in any order and position within the logs
-@c itself.
-@c
+@noindent
+Finally, note that the declaration of the global test result in the first
+line, apart from being needed for backward-compatibility with the default
+@option{parallel-tests} driver, can be useful also for test protocols
+that allow more test results per test script: using it, a custom test
+driver is allowed to decide what the global outcome of the test script
+is in case of conflicting test results within the script. For example,
+if a test script reports 8 successful test cases and 2 skipped test
+cases, some drivers might report that globally as a SKIP, while others
+as a PASS.
@node Console output during testsuite runs
@subsubsection Console output during testsuite runs
-TODO.
-@c * explain what kind of console output the driver is expected to produce;
-@c it can have some leeway here, but IMHO should provide a "look & feel"
-@c similar to the existing "parallel-tests" output.
-@c * tell that this is where the --color option is to be influential.
+A custom test driver has also the task of displaying, on the standard
+output, the test results as soon as they become available. Depending on
+the protocol in use, it can also display the reasons for failures and
+skips, and, more generally, any useful diagnostic output (but remember
+that each line on the screen is precious, so that cluttering the screen
+with overly verbose information is bad idea). The exact format of this
+progress output is left up completely to the test drive; in fact, a custom
+test driver might @emph{theoretically} even decide not to do any such
+report, leaving it all to the testsuite summary (that would be a very
+lousy idea, of course, and serves only to illustrate the flexibility that
+is granted here).
+
+Remember that consistency is good; so, if possible, try to be consistent
+with the output of the built-in Automake test drivers, providing a similar
+``look & feel''. In particular, the testsuite progress output should be
+colorized when the @option{--color-tests} is passed to the driver. On the
+other end, if you are using a known and widespread test protocol with
+well-established implementations, being consistent with those
+implementations' output might be a good idea too.
+
+@c TODO: Give an example, maybe inspired to py.test-style output.
+@c TODO: That is a good idea because it shows a test driver that allows
+@c TODO: for different levels of verbosity in the progress output (could
+@c TODO: be implemented either using a driver cmdline flag, or an
+@c TODO: environment variable, or both).
@node HTML generation from testsuite logs
@subsubsection HTML generation from testsuite logs
If HTML testsuite output (with @code{check-html}) is to be supported,
the generated @file{.log} files must contain syntactically valid
-reStructuredText. If this is not the case, the HTML generation will not
-work, but all the other functionalities of the Automake testsuite harness
-should remain untouched, and continue to work correctly.
+reStructuredText (among the other things, this means that every
+@code{:test-result:} line must be followed by a blank line). If this
+is not the case, the HTML generation will not work, although all the
+other functionalities of the Automake testsuite harness should remain
+untouched, and continue to work correctly.
@node DejaGnu Tests
@section DejaGnu Tests
diff --git a/lib/am/check.am b/lib/am/check.am
index cbf0acb..b02d6c5 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -205,16 +205,8 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
echo; \
echo ".. contents:: :depth: 2"; \
echo; \
-## Here we assume that the test driver writes a proper summary for the
-## test script on the first line. Requiring this is not a limitation,
-## but a feature, since this way a custom test driver is allowed to decide
-## what the outcome is in case of conflicting testcase results in a test
-## script. For example, if a test script reports 8 successful testcases
-## and 2 skipped testcases, some drivers might report that globally as a
-## SKIP, while others as a PASS.
-## FIXME: This should be documented in the automake manual. The above
-## FIXME: explanation is indeed more appropriate for the manual than for
-## FIXME: comments in code.
+## Here we assume that the test driver writes a proper "summarizing
+## result" for the test script on the first line.
for f in $$list; do \
test -r $$f && read line < $$f || line=; \
case $$line in \
