On Monday 27 June 2011, Ralf Wildenhues wrote:
> Hi Stefano,
>
Hi Ralf, thanks for the review.
> * Stefano Lattarini wrote on Sun, Jun 26, 2011 at 02:20:48PM CEST:
> > I've pushed the documentation work I've done so far to the temporary
> > public branch "GSoC/experimental/docs-custom-test-drivers", in case
> > anyone wants to take a look at it (I had hoped to finish it this morning,
> > but I didn't, and I'll be AFK until this evening, so there's no point in
> > delaying even more). Notice that that branch is bound to be amended and
> > rebased, and finally deleted (it won't be merged in the 'test-protocols'
> > branch, rather its changes will be re-applied on the top of that).
>
> A few comments from a cursory look:
>
>
> > +2011-06-25 Stefano Lattarini <stefano.lattar...@gmail.com>
> > +
> > + docs: document custom test drivers and protocols
> > + * doc/automake.texi (Simple Tests): Note that the TESTS_ENVIRONMENT
> > + use suggested here is not portable to 'parallel-tests'.
> > + (Simple Tests using parallel-tests): Document new restrictions on
> > + the uses of TESTS_ENVIRONMENT and AM_TESTS_ENVIRONMENT.
> > + (Custom Test Drivers): New section and node.
> > + (Overview of Custom Test Drivers Support): New subsection.
> > + (Declaring Custom Test Drivers in @file{Makefile.am}): Likewise.
> > + (APIs for Custom Test Drivers): Likewise.
> > + (Options): Update description of color-tests.
> > + * lib/am/check (recheck, recheck-html): Minor adjustments to better
> > + conform to the documentation (this should cause no semantic changes
> > + w.r.t. the former behaviour); minor improvements and extensions to
> > + existing comments.
> > + * tests/test-driver-create-log-dir.test: New test.
> > + * tests/test-driver-strip-vpath.test: Likewise.
> > + * tests/test-driver-global-log.test: Likewise.
> > + * tests/test-driver-recheck.test: Likewise.
> > + * tests/Makefile.am (TESTS): Update.
>
> > --- a/doc/automake.texi
> > +++ b/doc/automake.texi
>
> > @@ -8691,10 +8708,12 @@ Simple Tests
> > TESTS = foo.pl bar.pl baz.pl
> > @end example
> >
> > -Note that the @option{parallel-tests} driver provides a more elegant
> > -way to achieve the same effect, freeing the @code{TESTS_ENVIRONMENT}
> > -variable for the user to override (@pxref{Simple Tests using
> > -parallel-tests}).
> > +It's important to note that this last use of @code{TESTS_ENVIRONMENT}
> > +is invalid when the @option{parallel-tests} option is active; however,
> > +the @option{parallel-tests} driver provides a more elegant way to
> > +achieve the same effect, with the further benefit of freeing the
> > +@code{TESTS_ENVIRONMENT} variable for the user
> > +(@pxref{Simple Tests using parallel-tests}).
> >
> >
> > @cindex Tests, expected failure
> > @@ -8739,7 +8758,9 @@ Simple Tests using parallel-tests
> > the @code{check_*} variables are honored, and the environment variable
> > @env{srcdir} is set during test execution. Also, @code{TESTS_ENVIRONMENT}
> > is still honored, but is complemented by a new developer-reserved variable
> > -@code{AM_TESTS_ENVIRONMENT} (described below).
> > +@code{AM_TESTS_ENVIRONMENT} (described below), and its allowed uses are
> > +somewhat restricted (other features of the @option{parallel-tests} driver
> > +compensate for this, though).
> >
> > This test driver is still experimental and may undergo changes in order
> > to satisfy additional portability requirements.
> > @@ -8812,12 +8833,13 @@ Simple Tests using parallel-tests
> > but should be reserved for the user.
> >
> > @vindex AM_TESTS_ENVIRONMENT
> > -The @code{AM_TESTS_ENVIRONMENT} variable can be used to run initialization
> > -code and set environment variables for the tests' runs. The user can
> > -still employ the @code{TESTS_ENVIRONMENT} variable to override settings
> > -from @code{AM_TESTS_ENVIRONMENT}. Note that, for implementation reasons,
> > -if the @code{AM_TESTS_ENVIRONMENT} variable is set, its contents
> > -@emph{must} be terminated by a semicolon.
> > +The @code{AM_TESTS_ENVIRONMENT} and @code{TESTS_ENVIRONMENT} variables can
> > +be used to run initialization code and set environment variables for the
> > +test scripts. The former variable is developer-reserved, and can be
> > +defined in the @file{Makefile.am}, while the latter is reserved for the
>
> How about s/reserved // here, to reflect prior usage?
>
Well, I do think we should now discourage such prior usage...
> > +user, which can employ it to extend or override the settings in the
> > +former. For implementation reasons, if the @code{AM_TESTS_ENVIRONMENT}
>
> s/\. /& /
>
Fixed.
> > +variable is set, its contents @emph{must} be terminated by a semicolon.
>
> Why is that necessarily so?
>
Otherwise the user couldn't safely use TESTS_ENVIRONMENT for overriding
AM_TESTS_ENVIRONMENT; for example, with "foo=0 foo=1 COMMAND ...", some
shells (e.g., bash and dash) will export foo to `0' in the environment
of COMMAND, while other shells (e.g., Solaris /bin/sh) will export foo
to `1'. And BTW, it was you that pointed this out to me originally ;-)
> > @@ -8834,6 +8856,23 @@ Simple Tests using parallel-tests
> > AM_TESTS_ENVIRONMENT = exec 9>&2; warn_fileno=9; export warn_fileno;
> > @end example
> >
> > +It's important to note that, differently from what we've seen for the
> > +serial testsuite driver (@pxref{Simple Tests using parallel-tests}), the
> > +@code{AM_TESTS_ENVIRONMENT} and @code{TESTS_ENVIRONMENT} variables
> > +@emph{cannot} be use to define a custom test driver; the
> > +@code{LOG_COMPILER} and @code{LOG_FLAGS} (or their extension-specific
> > +counterparts) should be used instead:
> > +
> > +@example
> > +## This is WRONG!
> > +AM_TESTS_ENVIRONMENT = PERL5LIB='$(srcdir)/lib' $(PERL) -Mstrict -w
> > +
> > +## Do this instead.
> > +AM_TESTS_ENVIRONMENT = PERL5LIB='$(srcdir)/lib'; export PERL5LIB;
> > +LOG_COMPILER = $(PERL)
> > +AM_LOG_FLAGS = -Mstrict -w
> > +@end example
> > +
> > @trindex mostlyclean
> > @trindex check-html
> > @vindex RST2HTML
> > @@ -8959,6 +8998,290 @@ Simple Tests using parallel-tests
> > In case of doubt you may want to require to use GNU @command{make},
> > or work around the issue with inference rules to generate the tests.
> >
> > +@node Custom Test Drivers
> > +@section Custom Test Drivers
> > +
> > +@menu
> > +* Overview of Custom Test Drivers Support::
> > +* Declaring Custom Test Drivers::
> > +* API for Custom Test Drivers::
> > +@end menu
> > +
> > +@node Overview of Custom Test Drivers Support
> > +@subsection Overview of Custom Test Drivers Support
> > +
> > +Starting form Automake version 1.12, the @option{parallel-tests} harness
> > +has been augmented to allow the package authors to use third-party custom
>
> s/has been augmented to allow/allows/
> s/authors/author/
>
> It is usually easier to read some text a couple of years down the road
> if it doesn't reflect transition details that aren't necessary to know
> after the fact.
>
OK.
> > +test drivers, in case the
> > +@c FIXME: this should become "default ones" once we have TAP and Subunit
> > +default one is inadequate for their purposes.
> > +@c FIXME: add this once we have TAP and Subunit
> > +@c "or do not support their testing protocol of choice."
> > +
> > +A custom test driver is expected to properly run the test programs passed
> > +to it (bearing in mind that they can have command-line arguments too, which
> > +can legitimately happen within the Automake testsuite harness framework),
>
> s/(.*)/, including the command-line arguments passed to it/
>
Yes, simpler and clearer. Fixed
> > +to analyze their execution and outcome, to create the @file{.log} files
> > +associated to these test runs, and to display the test results on the
> > +console.
>
> > It is the sole responsibility of the author of the test driver
> > +to ensure that it implements all the above steps meaningfully and
> > +correctly; Automake isn't and can't be of any help here.
>
> I'm wondering whether it's better to omit this sentence. It sounds like
> a teacher talking to me; in programming, I am always expected to do
> things correctly.
>
Yes, but often your tools offer diagnostic that helps to prevent some, or
many, mistakes. In case the tools can't or won't do that (like here),
extra warnings and "be careful" exhortations in the documentation are
IMHO warranted.
> > On the other
> > +hand, the Automake-provided code for testsuite summary generation offers
> > +support for test drivers allowing several test results per test script,
> > +if they take care of registering such results properly (@pxref{Log files
>
> s/of registering/to register/
>
OK.
> > +generation and test results recording}).
> > +
> > +The exact details of how test scripts' results are to be determined and
> > +analyzed is left to the individual drivers. Some drivers might only
> > +consider the test script exit status (this is done for example by the
> > +default test driver used by the @option{parallel-tests} harness, described
> > +in the previous section). Other drivers might implement more complex and
> > +advanced test protocols, which might require them to parse and interpreter
> > +the output emitted by the test script they're running (examples of such
> > +protocols are TAP and SubUnit).
> > +
> > +It's very important to note that, even when using custom test drivers,
> > +most of the @option{parallel-tests} infrastructure described in the
> > +previous section remains in place; this comprises:
>
> s/comprises/includes/
> (the alternative is "this is comprised of" and needs an exhaustive list)
>
I went for 's/comprises/includes/'.
> > +@itemize
> > +@item
> > +concurrency through the use of @command{make}'s option @option{-j};
> > +@item
> > +per-test @file{.log} files, and generation of a summary @file{.log}
> > +file from them;
> > +@item
> > +@code{recheck} target and lazy reruns of tests;
> > +@item
> > +inter-test dependencies;
> > +@item
> > +support @code{check_*} variables (@code{check_PROGRAMS},
> > +@code{check_LIBRARIES}, ...);
> > +@item
> > +use of @code{VERBOSE} environment variable to get verbose output on
> > +testsuite failures;
> > +@item
> > +definition and honoring of @code{TESTS_ENVIRONMENT} and
> > +@code{AM_TESTS_ENVIRONMENT} variables;
> > +@item
> > +definition of generic and extension-specific @code{LOG_COMPILER} and
> > +@code{LOG_FLAGS} variables.
> > +@end itemize
> > +
>
> @noindent?
>
You mean before the following lines? (I guess you do)
> > +On the other hand, the exact semantics of how (and if)
> > +@option{color-tests}, @code{XFAIL_TESTS}, and hard errors are supported
> > +and handled is left to the individual test drivers.
> > +
> > +@c TODO: We should really add a working example in the doc/ directory,
> > +@c TODO: and reference if from here.
>
> Good idea. It could be a simplified form of the default test driver.
>
> > +@node Declaring Custom Test Drivers
> > +@subsection Declaring Custom Test Drivers in @file{Makefile.am}
>
> Karl recommends keeping node and section names the same (modulo
> formatting and comma), or at least very similar.
>
And I agree with him (the above inconsistency was probably due to an
incomplete renaming, even if I can't remember exactly). Will fix.
> > +@vindex _LOG_DRIVER
> > +@vindex _LOG_DRIVER_FLAGS
> > +@vindex LOG_DRIVER
> > +@vindex LOG_DRIVER_FLAGS
> > +@vindex @var{ext}_LOG_DRIVER
> > +@vindex @var{ext}_LOG_DRIVER_FLAGS
> > +@vindex AM_@var{ext}_LOG_DRIVER_FLAGS
> > +@vindex AM_LOG_DRIVER_FLAGS
> > +Custom testsuite drivers are declared by defining the make variables
> > +@code{LOG_DRIVER} or @code{@var{ext}_LOG_DRIVER} (where @var{ext} must
> > +be declared in @code{TEST_EXTENSIONS}). They must be defined to
> > +programs or scripts that will be used to drive the execution, logging,
> > +and outcome report of the tests with corresponding extensions (or of
> > +those with no registered extension in the case of @code{LOG_DRIVER}).
> > +Clearly, multiple distinct test drivers can be declared in the same
> > +@file{Makefile.am}. Note moreover that the @code{LOG_DRIVER} variables
> > +are @emph{not} a substitute for the @code{LOG_COMPILER} variables: the
> > +two sets of variables can, and often do, usefully and legitimately
> > +coexist.
> > +
> > +@c TODO: We should really be able to point to a clarifying example here!
>
> Yep. As a reader, this leaves me confused. Why should there be two
> separate script instances?
>
> Actually, why should there (other than maybe for backward
> compatibility)? I mean, not just in the documentation, but in the
> Automake code as well? Can't we simplify this? Thanks.
>
Oh no! They are really meant to fullfill two orthogonal and complementary
needs. Let's see if an example can help to clarify the situation.
Assume you are testing a static code checker for C; in this case, you
might to write most of your test cases as C files with expected diagnostic
embedded in special comments, as in e.g.:
...
if (a = b) /*! did you mean "a == b" here? */
...
Then you can add `.c' in TEST_EXTENSIONS, and define C_LOG_COMPILER as
a script that extracts the expected diagnostic from a C file (keeping
track of line numbers too), runs the code checker on that same C file,
and compares the diagnostic emitted by the code checker with the one
extracted from the file.
As you can see, this is useful as-is, and doesn't force the developer
to write a custom test driver, which would also have to take care of
generating the `.log', doing console output (with possible coloring),
handling expected failures disabilitation of hard errors, etc. -- all
without any gain, because the default test driver already behaves in
a perfectly adequate way in these respects.
So far, so good. At this point, however, each `.c' test will count
as a whole failure even if, say, 90% of the diagnostic it gives is
correct. This is clearly suboptimal. So the developer might modify
his $(C_LOG_COMPILER) script to emit TAP output instead of just
returning a single pass/fail status, and then define C_LOG_DRIVER to
a script (hopefully Automake provided ;-) that handles TAP. If the
$(C_LOG_COMPILER) script has been written with care, this could be a
simple and localized change. OTOH, without having the possibility
of keeping $(C_LOG_COMPILER) and $(C_LOG_DRIVER) separate, the
developer would have to write a monolitic test driver from scratch,
and it would have to assume the roles and functionalities of both
$(C_LOG_COMPILER) and $(C_LOG_DRIVER). Not nice.
> > +The developer-reserved variable @code{AM_LOG_DRIVER_FLAGS} and the
> > +user-reserved variable @code{LOG_DRIVER_FLAGS} can be used to define
> > +flags that will be passed to each invocation of @code{LOG_DRIVER}
> > +(with the user-defined flags obviously taking precedence over the
> > +developer-reserved ones). Similarly, for each extension @var{ext}
> > +declared in @code{TEST_EXTENSIONS}, flags listed in
> > +@code{AM_@var{ext}_LOG_DRIVER_FLAGS} and
> > +@code{@var{ext}_LOG_DRIVER_FLAGS} will be passed to
> > +invocations of @code{@var{ext}_LOG_DRIVER}.
> > +
> > +@node API for Custom Test Drivers
> > +@subsection API for Custom Test Drivers
> > +
> > +Here we'll describe the APIs currently offered to the authors of custom
> > +test drivers.
>
> Just describe things, don't describe what you describe.
> (This is OK as a TODO note of course)
>
OK.
> > Note that they are still somewhat experimental, and might
> > +undergo changes and tightenings in the future, to accommodate for new
> > +features or to satisfy additional portability requirements.
> > +
> > +Also, note that the main characteristic of these APIs is that they are
>
> repetition; s/Also, note that t/T/
>
Fixed.
> > +designed to share as much infrastructure, semantics, and implementation
> > +details as possible with the plain @option{parallel-tests} harness. So
> > +everything said in the previous section should apply here too, unless
> > +explicitly noted otherwise.
>
> repetition; s/noted/stated/
>
OK.
> > +@menu
> > +* Command-line arguments for test drivers::
> > +* Log files generation and test results recording::
> > +* Console output during testsuite runs::
> > +* HTML generation from testsuite logs::
> > +@end menu
> > +
> > +@node Command-line arguments for test drivers
> > +@subsubsection Command-line arguments for test drivers
> > +
> > +A custom driver can rely on various command-line options and arguments
> > +being passed to it automatically by the Automake's @option{parallel-tests}
> > +harness. It is @emph{mandatory} that it understands all of them (even if
> > +the exact interpretation of the associated semantics can change between a
> > +test driver and another, and even be a no-op in some drivers).
> > +
> > +@noindent
> > +Here is the list of options:
> > +
> > +@table @option
> > +@item --test-name=@var{NAME}
> > +The name of the test, with VPATH prefix (if any) removed. This can have a
> > +suffix and a directory component (as in e.g., @file{sub/foo.test}), and is
> > +mostly meant to be used in console reports about testsuite advancements and
> > +results (@pxref{Console output during testsuite runs}).
> > +@item --log-file=@var{PATH}
> > +The log file the test driver must create. If it has a directory component
> > +(as in e.g., @file{sub/foo.log}), the Automake harness will ensure that
> > +such directory exist before the test driver is called.
> > +@item --color-tests=@{yes|no@}
> > +Whether the console output should be colored or not (@pxref{Simple
> > +tests and color-tests} to learn when this option gets activated and
> > +when it doesn't).
> > +@item --expect-failure=@{yes|no@}
> > +Whether the tested program is expected to fail.
> > +@item --enable-hard-errors=@{yes|no@}
> > +Whether "hard errors" in the tested program should be treated differently
>
> Prefer ``quotes'' to "quotes" unless the quotes are part of a code
> snippet. Several instances
>
OK (but I remember you disapproving the use of ``quotes'' in a previous mail;
did I get that wrong?)
> > +from normal failures or not (the default should be "@code{yes}"). The
> > exact
>
> s/"//g
>
Fixed.
> > +meaning of "hard error" is highly dependent from the test protocols or
> > +conventions in use.
> > +@item --
> > +Explicitly terminate the list of options.
> > +@end table
> > +
> > +The first of the remaining arguments passed to the test driver is the
> > +program to be run, and the other arguments are command-line options
> > +and arguments for this program.
> > +
> > +Note that the exact semantics attached to the @option{--color-tests},
> > +@option{--expect-failure} and @option{--enable-hard-errors} options are
> > +left up to the individual test drivers. Still, having a behaviour
> > +compatible or at least similar to that provided by the default
> > +@option{parallel-tests} driver is advised, as that would offer a better
> > +consistency and a more pleasant user experience.
> > +
> > +@node Log files generation and test results recording
> > +@subsubsection Log files generation and test results recording
> > +
> > +One of the main features of the new testsuite harness is the ability to
> > +support test protocols that allow a single test script to run more
> > +test cases, @emph{each with its distinct result}. In order for the
> > +testsuite summary report to be correct in this case, the test driver
> > +implementing support for the protocol must follow the following
> > +convention.
> > +
> > +TODO: In order to have a correct count of test cases in the final
> > +testsuite summary, the driver must use special syntax in the created
> > +.log file (see below, `:test-result:').
> > +
> > +The test driver must correctly generate the file specified by the
> > +@option{--log-file} option (even when the tested program fails, of
> > +course!). This file is quite free-form, but still it has to conform
> > +to the following conventions:
> > +
> > +@itemize @bullet
> > +@item
> > +@cindex Global test script result
> > +@cindex Test result, global
> > +The @emph{global test result} for the whole test script must be placed
> > +in the first line of the @file{.log} file.
> > +@c The following semantics is checked by tests 'test-driver-recheck.test'
> > +@c and 'test-driver-global-log.test'. Please keep them in sync whenever
> > +@c the following is changed.
> > +If this line does not begin with either @code{PASS:}, @code{XFAIL:} or
> > +@code{SKIP:}, the test script will be considered failed, and it will be
> > +re-run by @code{make recheck}. Also, if this line does not begin with
> > +either @code{PASS:} or @code{XFAIL:}, the content of and its @file{.log}
> > +file will be copied into the global @file{test-suite.log} (in order to
> > +help in debugging and bug-report analysis).
> > +@item
> > +@cindex Single test case result
> > +@cindex Test result, single
> > +@cindex @code{:test-result:} reStructuredText field
> > +@cindex reStructuredText field, @code{:test-result:}
> > +One of the main features of the new testsuite harness is the ability to
> > +support test protocols that allow a single test script to run more
> > +test cases, @emph{each with its distinct result}. In order for the
> > +testsuite report to be correct in this case, the test driver must
> > +register @emph{each} such result in the generated @file{.log} file, using
> > +the @code{:test-result:} reStructuredText field.
> > +
> > +FIXME: currently, that field is recognized anywhere in the @file{.log}
> > +file; this is possibly prone to generating spurious results, in case
> > +of verbose tests. We should allow a special @code{:test-result:} that
> > +stops any further parsing (maybe a line @code{:test-result:END} will
> > +do?).
>
> Well, but usually you only know the test outcome at the very end.
> Requiring the result to be early in the test will make file generation
> more complicated than it would need to be otherwise.
>
True; but with the semantic I'm proposing you can be 100% sure not to
have spurious test results (as everything after ":test-result:END" is
ignored); how'd you achive such safety if you want to allow test results
go at the end of the log instead? Safety which you youself harped on
BTW (and in hindsight I agree that you were perfectly right in doing so).
> > +FIXME: we are probably going to change the current semantics of
> > +testsuite summary generation, in order to allow test protocols with
> > +only one result per test case not to be forced to also declare that
> > +result with @code{:test-results:}; that would be a repetition, since
> > +the result is already declared in the @file{.log} first line.
> > +@end itemize
>
> Good question.
>
> > +@emph{TODO}: Explain why supporting the Global test result declaration
> > +in the first line is useful also for test protocols that allow more
> > +test cases per test script.
> > +
> > +@c
> > +@c * A test driver is free (but not forced) to register multiple test
> > +@c results for a single test script, through the use of a custom
> > +@c reStructuredText field ``test-result''. Each of this results will
> > +@c be accounted for separately in the final testsuite summary;
> > +@c TODO: give an example
> > +@c TODO: a special field to terminate parsing of .log?
> > +@c But note that the task of reporting this results on the screen is
> > +@c left to the individual test driver though; more about this below.
> > +@c For example, if the test script executes
> > +@c two test cases, one successful and one failing, and skip another
> > +@c test case, the driver should end up writing the following in the
> > +@c test log:
> > +@c :test-result: PASS [name of passed test, and/or details about it]
> > +@c :test-result: FAIL [name of failed test, and/or details about it]
> > +@c :test-result: SKIP [name of skipped test, and/or details about it]
> > +@c The above lines (each of which *must* be followed by a blank line
> > +@c in order for the HTML output generation to work) are used only
> > +@c when generating the `test-suite.log' from the individual test
> > +@c logs, and can be placed in any order and position within the logs
> > +@c itself.
> > +@c
> > +
> > +@node Console output during testsuite runs
> > +@subsubsection Console output during testsuite runs
> > +
> > +TODO.
> > +@c * explain what kind of console output the driver is expected to
> > produce;
> > +@c it can have some leeway here, but IMHO should provide a "look & feel"
> > +@c similar to the existing "parallel-tests" output.
> > +@c * tell that this is where the --color option is to be influential.
> > +
> > +@node HTML generation from testsuite logs
> > +@subsubsection HTML generation from testsuite logs
> > +
> > +If the HTML testsuite output (with @code{check-html}) is to be supported,
>
> s/the //
>
Fixed.
> > +the generated @file{.log} files must contain syntactically valid
> > +reStructuredText. If this is not the case, the HTML generation won't
>
> will not
>
Aren't "won't" and "will not" equivalent? I'll do the fix anyway, because
I don't have any preference here; but I'm just curious.
> > +work, but all the other functionalities of the Automake testsuite harness
> > +should remain untouched, and continue to work correctly.
>
> > --- a/lib/am/check.am
> > +++ b/lib/am/check.am
> > @@ -317,7 +317,8 @@ recheck recheck-html:
> > ## test script on the first line. See the comments in the rules of
> > ## $(TEST_SUITE_LOG) above for why we consider this acceptable and even
> > ## advisable.
> > - case $$line in FAIL*|XPASS*) echo $$f;; esac; \
> > + case $$line in PASS:*|XFAIL:*|SKIP:*);; *) echo $$f;; esac; \
> > +## A test whose log is unreadable is to be caonsidered failed.
>
> typo
>
Oops, sorry. Fixed.
> Why is the comment below the code?
>
Because if refers to the code in the following "else" branch.
> What about XPASSes?
>
They are basically treated like failures (as is now everything else that
is not an explicit PASS; XFAIL or SKIP). No semantic change here.
> Can test drivers implement other kinds of results?
>
They could be easily extended to do so (and I think we'll need a new
"ERROR" outcome to cater for, e.g., "Bail out!" directives in TAP).
But making the current test harness more flexible in this regard is
ortoghonal to the current changes, and while I'd love to go down that
road too, it will have to wait (even if it would be nice to have it
before 1.12).
> (See dejagnu for examples)
>
> Thanks,
> Ralf
>
Attached is the squash-in with the fixes you suggested. I hope to
be soon able to finish up this part of the documentation; I'll post
and additional squash-in when I'm done.
Thanks,
Stefano
diff --git a/doc/automake.texi b/doc/automake.texi
index 4dafb00..8a6871c 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -8838,7 +8838,7 @@ be used to run initialization code and set environment variables for the
test scripts. The former variable is developer-reserved, and can be
defined in the @file{Makefile.am}, while the latter is reserved for the
user, which can employ it to extend or override the settings in the
-former. For implementation reasons, if the @code{AM_TESTS_ENVIRONMENT}
+former. For implementation reasons, if the @code{AM_TESTS_ENVIRONMENT}
variable is set, its contents @emph{must} be terminated by a semicolon.
@example
@@ -9011,24 +9011,23 @@ or work around the issue with inference rules to generate the tests.
@subsection Overview of Custom Test Drivers Support
Starting form Automake version 1.12, the @option{parallel-tests} harness
-has been augmented to allow the package authors to use third-party custom
-test drivers, in case the
+allows the package authors to use third-party custom test drivers, in case
+the
@c FIXME: this should become "default ones" once we have TAP and Subunit
default one is inadequate for their purposes.
@c FIXME: add this once we have TAP and Subunit
@c "or do not support their testing protocol of choice."
-A custom test driver is expected to properly run the test programs passed
-to it (bearing in mind that they can have command-line arguments too, which
-can legitimately happen within the Automake testsuite harness framework),
+A custom test driver is expected to properly run the test programs
+passed to it, including the command-line arguments passed to it,
to analyze their execution and outcome, to create the @file{.log} files
associated to these test runs, and to display the test results on the
console. It is the sole responsibility of the author of the test driver
to ensure that it implements all the above steps meaningfully and
-correctly; Automake isn't and can't be of any help here. On the other
+correctly; Automake isn't and can't be of any help here. On the other
hand, the Automake-provided code for testsuite summary generation offers
support for test drivers allowing several test results per test script,
-if they take care of registering such results properly (@pxref{Log files
+if they take care to register such results properly (@pxref{Log files
generation and test results recording}).
The exact details of how test scripts' results are to be determined and
@@ -9042,7 +9041,7 @@ protocols are TAP and SubUnit).
It's very important to note that, even when using custom test drivers,
most of the @option{parallel-tests} infrastructure described in the
-previous section remains in place; this comprises:
+previous section remains in place; this includes:
@itemize
@item
@@ -9055,7 +9054,7 @@ file from them;
@item
inter-test dependencies;
@item
-support @code{check_*} variables (@code{check_PROGRAMS},
+support for @code{check_*} variables (@code{check_PROGRAMS},
@code{check_LIBRARIES}, ...);
@item
use of @code{VERBOSE} environment variable to get verbose output on
@@ -9068,6 +9067,7 @@ definition of generic and extension-specific @code{LOG_COMPILER} and
@code{LOG_FLAGS} variables.
@end itemize
+@noindent
On the other hand, the exact semantics of how (and if)
@option{color-tests}, @code{XFAIL_TESTS}, and hard errors are supported
and handled is left to the individual test drivers.
@@ -9076,7 +9076,7 @@ and handled is left to the individual test drivers.
@c TODO: and reference if from here.
@node Declaring Custom Test Drivers
-@subsection Declaring Custom Test Drivers in @file{Makefile.am}
+@subsection Declaring Custom Test Drivers
@vindex _LOG_DRIVER
@vindex _LOG_DRIVER_FLAGS
@@ -9113,16 +9113,15 @@ invocations of @code{@var{ext}_LOG_DRIVER}.
@node API for Custom Test Drivers
@subsection API for Custom Test Drivers
-Here we'll describe the APIs currently offered to the authors of custom
-test drivers. Note that they are still somewhat experimental, and might
-undergo changes and tightenings in the future, to accommodate for new
-features or to satisfy additional portability requirements.
+Note that @emph{the APIs described here are still somewhat experimental},
+and might undergo changes and tightenings in the future, to accommodate
+for new features or to satisfy additional portability requirements.
-Also, note that the main characteristic of these APIs is that they are
-designed to share as much infrastructure, semantics, and implementation
-details as possible with the plain @option{parallel-tests} harness. So
-everything said in the previous section should apply here too, unless
-explicitly noted otherwise.
+The main characteristic of these APIs is that they are designed to share
+as much infrastructure, semantics, and implementation details as possible
+with the plain @option{parallel-tests} harness. So everything said in
+the previous section should apply here too, unless explicitly stated
+otherwise.
@menu
* Command-line arguments for test drivers::
@@ -9160,9 +9159,9 @@ when it doesn't).
@item --expect-failure=@{yes|no@}
Whether the tested program is expected to fail.
@item --enable-hard-errors=@{yes|no@}
-Whether "hard errors" in the tested program should be treated differently
-from normal failures or not (the default should be "@code{yes}"). The exact
-meaning of "hard error" is highly dependent from the test protocols or
+Whether ``hard errors'' in the tested program should be treated differently
+from normal failures or not (the default should be @code{yes}). The exact
+meaning of ``hard error'' is highly dependent from the test protocols or
conventions in use.
@item --
Explicitly terminate the list of options.
@@ -9277,9 +9276,9 @@ TODO.
@node HTML generation from testsuite logs
@subsubsection HTML generation from testsuite logs
-If the HTML testsuite output (with @code{check-html}) is to be supported,
+If HTML testsuite output (with @code{check-html}) is to be supported,
the generated @file{.log} files must contain syntactically valid
-reStructuredText. If this is not the case, the HTML generation won't
+reStructuredText. If this is not the case, the HTML generation will not
work, but all the other functionalities of the Automake testsuite harness
should remain untouched, and continue to work correctly.
diff --git a/lib/am/check.am b/lib/am/check.am
index 711d225..cbf0acb 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -318,7 +318,7 @@ recheck recheck-html:
## $(TEST_SUITE_LOG) above for why we consider this acceptable and even
## advisable.
case $$line in PASS:*|XFAIL:*|SKIP:*);; *) echo $$f;; esac; \
-## A test whose log is unreadable is to be caonsidered failed.
+## A test whose log is unreadable is to be considered failed.
else echo $$f; fi; \
done | tr '\012\015' ' '`; \
## This apparently useless munging helps to avoid a nasty bug (a
