Re: [FYI] New public temporary branch, about documentation for custom test drivers
On Tuesday 28 June 2011, Stefano Lattarini wrote:
On Monday 27 June 2011, Ralf Wildenhues wrote:
* Stefano Lattarini wrote on Mon, Jun 27, 2011 at 01:11:51PM CEST:
+If the @code{AM_TESTS_ENVIRONMENT} variable is set, its contents
+@emph{must} be terminated by a semicolon.
Why is that necessarily so?
Otherwise the user couldn't safely use TESTS_ENVIRONMENT for overriding
AM_TESTS_ENVIRONMENT; for example, with foo=0 foo=1 COMMAND ..., some
shells (e.g., bash and dash) will export foo to `0' in the environment
of COMMAND, while other shells (e.g., Solaris /bin/sh) will export foo
to `1'. And BTW, it was you that pointed this out to me originally ;-)
See, I simply didn't think of TESTS_ENVIRONMENT. So, the documentation
should make it clear that variables actually need to be exported (not
just set); it can't hurt to say why the semicolon is useful: so that
the user can set TESTS_ENVIRONMENT to override variables further.
I like this, and I'll make the change. But notice that the documentation
about AM_TESTS_ENVIRONMENT we're referring to was already in master, so
that the right thing to do is to fix it there, merge master to the branch
'test-protocols', and rebase the documentation branch on that.
Here it is the patch I've applied to master.
From 3e334a272fa601bebb5896e25cfb63f34822a275 Mon Sep 17 00:00:00 2001
Message-Id:
3e334a272fa601bebb5896e25cfb63f34822a275.1309333773.git.stefano.lattar...@gmail.com
From: Stefano Lattarini stefano.lattar...@gmail.com
Date: Wed, 29 Jun 2011 09:44:14 +0200
Subject: [PATCH] docs: explain why AM_TESTS_ENVIRONMENT must be
semicolon-terminated
* doc/automake.texi (Simple Tests using parallel-tests): Ditto, and
related adjustments.
Suggestion by Ralf Wildenhues.
---
ChangeLog |7 +++
doc/automake.texi |8
2 files changed, 11 insertions(+), 4 deletions(-)
diff --git a/ChangeLog b/ChangeLog
index 7ce..64c2bbb 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,10 @@
+2011-06-29 Stefano Lattarini stefano.lattar...@gmail.com
+
+ docs: explain why AM_TESTS_ENVIRONMENT must be semicolon-terminated
+ * doc/automake.texi (Simple Tests using parallel-tests): Ditto, and
+ related adjustments.
+ Suggestion by Ralf Wildenhues.
+
2011-06-29 Stefano Lattarini stefano.lattar...@gmail.com
docs: fix unportable example of AM_TESTS_ENVIRONMENT usage
diff --git a/doc/automake.texi b/doc/automake.texi
index d60bfe4..da2e2b4 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -8740,10 +8740,10 @@ but should be reserved for the user.
The @code{AM_TESTS_ENVIRONMENT} variable can be used to run initialization
code and set environment variables for the tests' runs. The user can
still employ the @code{TESTS_ENVIRONMENT} variable to override settings
-from @code{AM_TESTS_ENVIRONMENT}. Note that, for implementation reasons,
-if the @code{AM_TESTS_ENVIRONMENT} variable is set, its contents
-@emph{must} be terminated by a semicolon. Here is an example of a
-slightly elaborate definition:
+from @code{AM_TESTS_ENVIRONMENT}; for that to work portably, however,
+the contents of a non-empty @code{AM_TESTS_ENVIRONMENT} @emph{must} be
+terminated by a semicolon. Here is an example of a slightly elaborate
+definition:
@example
AM_TESTS_ENVIRONMENT = \
--
1.7.2.3
Re: [FYI] New public temporary branch, about documentation for custom test drivers
On Tuesday 28 June 2011, Stefano Lattarini wrote:
Please go ahead after addressing my issues above. The questions I wrote
should still be discussed (and might need changing stuff again) but that
shouldn't hold up the patch.
OK, will push this evening or tomorrow.
I've pushed to 'test-protocols' now, and I've deleted the temporary branch
'GSoC/experimental/docs-custom-test-drivers'. Attached is the patch that
has been pushed, for reference.
Regards,
Stefano
From 1402741313cf6c2fe8611eb294f9a8111c994a32 Mon Sep 17 00:00:00 2001
Message-Id: 1402741313cf6c2fe8611eb294f9a8111c994a32.1309354565.git.stefano.lattar...@gmail.com
From: Stefano Lattarini stefano.lattar...@gmail.com
Date: Wed, 22 Jun 2011 21:59:34 +0200
Subject: [PATCH] docs: document custom test drivers and protocols
* doc/automake.texi (Simple Tests): Note that the TESTS_ENVIRONMENT
use suggested here is not portable to 'parallel-tests'.
(Simple Tests using parallel-tests): Document new restrictions on
the uses of TESTS_ENVIRONMENT and AM_TESTS_ENVIRONMENT.
(Custom Test Drivers): New section and node.
(Overview of Custom Test Drivers Support): New subsection.
(Declaring Custom Test Drivers in @file{Makefile.am}): Likewise.
(APIs for Custom Test Drivers): Likewise.
(Options): Update description of color-tests.
* lib/am/check ($(TEST_SUITE_LOG)): Remove comments that have been
moved into the manual.
(recheck, recheck-html): Minor adjustments to better conform to the
documentation (this should cause no semantic changes w.r.t. the
former behaviour); minor improvements and extensions to existing
comments.
* tests/test-driver-create-log-dir.test: New test.
* tests/test-driver-strip-vpath.test: Likewise.
* tests/test-driver-global-log.test: Likewise.
* tests/test-driver-recheck.test: Likewise.
* tests/Makefile.am (TESTS): Update.
---
ChangeLog | 24 ++
doc/automake.texi | 378 +++--
lib/Automake/tests/Makefile.in|2 +-
lib/am/check.am | 15 +-
tests/Makefile.am |4 +
tests/Makefile.in |6 +-
tests/test-driver-create-log-dir.test | 74 +++
tests/test-driver-global-log.test | 110 ++
tests/test-driver-recheck.test| 126 +++
tests/test-driver-strip-vpath.test| 96 +
10 files changed, 804 insertions(+), 31 deletions(-)
create mode 100755 tests/test-driver-create-log-dir.test
create mode 100755 tests/test-driver-global-log.test
create mode 100755 tests/test-driver-recheck.test
create mode 100755 tests/test-driver-strip-vpath.test
diff --git a/ChangeLog b/ChangeLog
index c99944a..6203f89 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,27 @@
+2011-06-29 Stefano Lattarini stefano.lattar...@gmail.com
+
+ docs: document custom test drivers and protocols
+ * doc/automake.texi (Simple Tests): Note that the TESTS_ENVIRONMENT
+ use suggested here is not portable to 'parallel-tests'.
+ (Simple Tests using parallel-tests): Document new restrictions on
+ the uses of TESTS_ENVIRONMENT and AM_TESTS_ENVIRONMENT.
+ (Custom Test Drivers): New section and node.
+ (Overview of Custom Test Drivers Support): New subsection.
+ (Declaring Custom Test Drivers in @file{Makefile.am}): Likewise.
+ (APIs for Custom Test Drivers): Likewise.
+ (Options): Update description of color-tests.
+ * lib/am/check ($(TEST_SUITE_LOG)): Remove comments that have been
+ moved into the manual.
+ (recheck, recheck-html): Minor adjustments to better conform to the
+ documentation (this should cause no semantic changes w.r.t. the
+ former behaviour); minor improvements and extensions to existing
+ comments.
+ * tests/test-driver-create-log-dir.test: New test.
+ * tests/test-driver-strip-vpath.test: Likewise.
+ * tests/test-driver-global-log.test: Likewise.
+ * tests/test-driver-recheck.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+
2011-06-29 Stefano Lattarini stefano.lattar...@gmail.com
docs: explain why AM_TESTS_ENVIRONMENT must be semicolon-terminated
diff --git a/doc/automake.texi b/doc/automake.texi
index 481ada4..a26f2a5 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -319,9 +319,23 @@ Support for test suites
* Simple Tests::Listing programs and scripts in @code{TESTS}
* Simple Tests using parallel-tests:: More powerful test driver
+* Custom Test Drivers:: Writing and using custom test drivers
* DejaGnu Tests:: Interfacing with the external testing framework
* Install Tests:: Running tests on installed packages
+Custom Test Drivers
+
+* Overview of Custom Test Drivers Support::
+* Declaring Custom Test Drivers::
+* API for Custom Test Drivers::
+
+API for Custom Test Drivers
+
+* Command-line arguments for test drivers::
+* Log files generation and test results recording::
+* Testsuite progress output::
+* HTML generation from testsuite logs::
+
Changing Automake's Behavior
Re: [FYI] New public temporary branch, about documentation for custom test drivers
On Monday 27 June 2011, Ralf Wildenhues wrote:
* Stefano Lattarini wrote on Mon, Jun 27, 2011 at 01:11:51PM CEST:
On Monday 27 June 2011, Ralf Wildenhues wrote:
* Stefano Lattarini wrote on Sun, Jun 26, 2011 at 02:20:48PM CEST:
+The @code{AM_TESTS_ENVIRONMENT} and @code{TESTS_ENVIRONMENT} variables
can
+be used to run initialization code and set environment variables for
the
+test scripts. The former variable is developer-reserved, and can be
+defined in the @file{Makefile.am}, while the latter is reserved for the
How about s/reserved // here, to reflect prior usage?
Well, I do think we should now discourage such prior usage...
Yeah, we should. It's very wide-spread however.
Yes, and that's why I'm proposing to discourage it only in the
documentation, instead of also with automake-time warnings (as I'd
do in an ideal world).
And when you think in
terms of a developer eager to provide code portable to several Automake
versions, then changing that is very far down the road.
Painfully true. That's why I'll lobby for the introduction, starting from
1.11.2, of autoconf macros and/or Automake constructs that will allow the
developers to prob Automake version and features. If we had them today,
many of the problems you're highlighting here would become non-issues.
gnulib is in
this position, Libtool is, and several projects that don't provide
macros for other projects or Makefile.am fragments also are eager to
work with a range of Automake releases (often starting from 1.9.6 on).
+variable is set, its contents @emph{must} be terminated by a semicolon.
Why is that necessarily so?
Otherwise the user couldn't safely use TESTS_ENVIRONMENT for overriding
AM_TESTS_ENVIRONMENT; for example, with foo=0 foo=1 COMMAND ..., some
shells (e.g., bash and dash) will export foo to `0' in the environment
of COMMAND, while other shells (e.g., Solaris /bin/sh) will export foo
to `1'. And BTW, it was you that pointed this out to me originally ;-)
See, I simply didn't think of TESTS_ENVIRONMENT. So, the documentation
should make it clear that variables actually need to be exported (not
just set); it can't hurt to say why the semicolon is useful: so that
the user can set TESTS_ENVIRONMENT to override variables further.
I like this, and I'll make the change. But notice that the documentation
about AM_TESTS_ENVIRONMENT we're referring to was already in master, so
that the right thing to do is to fix it there, merge master to the branch
'test-protocols', and rebase the documentation branch on that.
+to analyze their execution and outcome, to create the @file{.log} files
+associated to these test runs, and to display the test results on the
+console.
It is the sole responsibility of the author of the test driver
+to ensure that it implements all the above steps meaningfully and
+correctly; Automake isn't and can't be of any help here.
I'm wondering whether it's better to omit this sentence. It sounds like
a teacher talking to me; in programming, I am always expected to do
things correctly.
Yes, but often your tools offer diagnostic that helps to prevent some, or
many, mistakes. In case the tools can't or won't do that (like here),
extra warnings and be careful exhortations in the documentation are
IMHO warranted.
OK. How about s/sole // then? We might actually want to (or be able
to) provide help at some point in the future.
Agreed, I'll make the change.
[SNIP]
+Custom testsuite drivers are declared by defining the make variables
+@code{LOG_DRIVER} or @code{@var{ext}_LOG_DRIVER} (where @var{ext} must
+be declared in @code{TEST_EXTENSIONS}). They must be defined to
+programs or scripts that will be used to drive the execution, logging,
+and outcome report of the tests with corresponding extensions (or of
+those with no registered extension in the case of @code{LOG_DRIVER}).
+Clearly, multiple distinct test drivers can be declared in the same
+@file{Makefile.am}. Note moreover that the @code{LOG_DRIVER} variables
+are @emph{not} a substitute for the @code{LOG_COMPILER} variables: the
+two sets of variables can, and often do, usefully and legitimately
+coexist.
+
+@c TODO: We should really be able to point to a clarifying example
here!
Yep. As a reader, this leaves me confused. Why should there be two
separate script instances?
Actually, why should there (other than maybe for backward
compatibility)? I mean, not just in the documentation, but in the
Automake code as well? Can't we simplify this? Thanks.
Oh no! They are really meant to fullfill two orthogonal and complementary
needs. Let's see if an example can help to clarify the situation.
Assume you are testing a static code checker for C; in this case, you
might to write most of your test cases as C files with expected
Re: [FYI] New public temporary branch, about documentation for custom test drivers
Hi Stefano,
* Stefano Lattarini wrote on Sun, Jun 26, 2011 at 02:20:48PM CEST:
I've pushed the documentation work I've done so far to the temporary
public branch GSoC/experimental/docs-custom-test-drivers, in case
anyone wants to take a look at it (I had hoped to finish it this morning,
but I didn't, and I'll be AFK until this evening, so there's no point in
delaying even more). Notice that that branch is bound to be amended and
rebased, and finally deleted (it won't be merged in the 'test-protocols'
branch, rather its changes will be re-applied on the top of that).
A few comments from a cursory look:
+2011-06-25 Stefano Lattarini stefano.lattar...@gmail.com
+
+ docs: document custom test drivers and protocols
+ * doc/automake.texi (Simple Tests): Note that the TESTS_ENVIRONMENT
+ use suggested here is not portable to 'parallel-tests'.
+ (Simple Tests using parallel-tests): Document new restrictions on
+ the uses of TESTS_ENVIRONMENT and AM_TESTS_ENVIRONMENT.
+ (Custom Test Drivers): New section and node.
+ (Overview of Custom Test Drivers Support): New subsection.
+ (Declaring Custom Test Drivers in @file{Makefile.am}): Likewise.
+ (APIs for Custom Test Drivers): Likewise.
+ (Options): Update description of color-tests.
+ * lib/am/check (recheck, recheck-html): Minor adjustments to better
+ conform to the documentation (this should cause no semantic changes
+ w.r.t. the former behaviour); minor improvements and extensions to
+ existing comments.
+ * tests/test-driver-create-log-dir.test: New test.
+ * tests/test-driver-strip-vpath.test: Likewise.
+ * tests/test-driver-global-log.test: Likewise.
+ * tests/test-driver-recheck.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -8691,10 +8708,12 @@ Simple Tests
TESTS = foo.pl bar.pl baz.pl
@end example
-Note that the @option{parallel-tests} driver provides a more elegant
-way to achieve the same effect, freeing the @code{TESTS_ENVIRONMENT}
-variable for the user to override (@pxref{Simple Tests using
-parallel-tests}).
+It's important to note that this last use of @code{TESTS_ENVIRONMENT}
+is invalid when the @option{parallel-tests} option is active; however,
+the @option{parallel-tests} driver provides a more elegant way to
+achieve the same effect, with the further benefit of freeing the
+@code{TESTS_ENVIRONMENT} variable for the user
+(@pxref{Simple Tests using parallel-tests}).
@cindex Tests, expected failure
@@ -8739,7 +8758,9 @@ Simple Tests using parallel-tests
the @code{check_*} variables are honored, and the environment variable
@env{srcdir} is set during test execution. Also, @code{TESTS_ENVIRONMENT}
is still honored, but is complemented by a new developer-reserved variable
-@code{AM_TESTS_ENVIRONMENT} (described below).
+@code{AM_TESTS_ENVIRONMENT} (described below), and its allowed uses are
+somewhat restricted (other features of the @option{parallel-tests} driver
+compensate for this, though).
This test driver is still experimental and may undergo changes in order
to satisfy additional portability requirements.
@@ -8812,12 +8833,13 @@ Simple Tests using parallel-tests
but should be reserved for the user.
@vindex AM_TESTS_ENVIRONMENT
-The @code{AM_TESTS_ENVIRONMENT} variable can be used to run initialization
-code and set environment variables for the tests' runs. The user can
-still employ the @code{TESTS_ENVIRONMENT} variable to override settings
-from @code{AM_TESTS_ENVIRONMENT}. Note that, for implementation reasons,
-if the @code{AM_TESTS_ENVIRONMENT} variable is set, its contents
-@emph{must} be terminated by a semicolon.
+The @code{AM_TESTS_ENVIRONMENT} and @code{TESTS_ENVIRONMENT} variables can
+be used to run initialization code and set environment variables for the
+test scripts. The former variable is developer-reserved, and can be
+defined in the @file{Makefile.am}, while the latter is reserved for the
How about s/reserved // here, to reflect prior usage?
+user, which can employ it to extend or override the settings in the
+former. For implementation reasons, if the @code{AM_TESTS_ENVIRONMENT}
s/\. / /
+variable is set, its contents @emph{must} be terminated by a semicolon.
Why is that necessarily so?
@@ -8834,6 +8856,23 @@ Simple Tests using parallel-tests
AM_TESTS_ENVIRONMENT = exec 92; warn_fileno=9; export warn_fileno;
@end example
+It's important to note that, differently from what we've seen for the
+serial testsuite driver (@pxref{Simple Tests using parallel-tests}), the
+@code{AM_TESTS_ENVIRONMENT} and @code{TESTS_ENVIRONMENT} variables
+@emph{cannot} be use to define a custom test driver; the
+@code{LOG_COMPILER} and @code{LOG_FLAGS} (or their extension-specific
+counterparts) should be used instead:
+
+@example
+## This is WRONG!
+AM_TESTS_ENVIRONMENT =
Re: [FYI] New public temporary branch, about documentation for custom test drivers
On Monday 27 June 2011, Ralf Wildenhues wrote:
Hi Stefano,
Hi Ralf, thanks for the review.
* Stefano Lattarini wrote on Sun, Jun 26, 2011 at 02:20:48PM CEST:
I've pushed the documentation work I've done so far to the temporary
public branch GSoC/experimental/docs-custom-test-drivers, in case
anyone wants to take a look at it (I had hoped to finish it this morning,
but I didn't, and I'll be AFK until this evening, so there's no point in
delaying even more). Notice that that branch is bound to be amended and
rebased, and finally deleted (it won't be merged in the 'test-protocols'
branch, rather its changes will be re-applied on the top of that).
A few comments from a cursory look:
+2011-06-25 Stefano Lattarini stefano.lattar...@gmail.com
+
+ docs: document custom test drivers and protocols
+ * doc/automake.texi (Simple Tests): Note that the TESTS_ENVIRONMENT
+ use suggested here is not portable to 'parallel-tests'.
+ (Simple Tests using parallel-tests): Document new restrictions on
+ the uses of TESTS_ENVIRONMENT and AM_TESTS_ENVIRONMENT.
+ (Custom Test Drivers): New section and node.
+ (Overview of Custom Test Drivers Support): New subsection.
+ (Declaring Custom Test Drivers in @file{Makefile.am}): Likewise.
+ (APIs for Custom Test Drivers): Likewise.
+ (Options): Update description of color-tests.
+ * lib/am/check (recheck, recheck-html): Minor adjustments to better
+ conform to the documentation (this should cause no semantic changes
+ w.r.t. the former behaviour); minor improvements and extensions to
+ existing comments.
+ * tests/test-driver-create-log-dir.test: New test.
+ * tests/test-driver-strip-vpath.test: Likewise.
+ * tests/test-driver-global-log.test: Likewise.
+ * tests/test-driver-recheck.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -8691,10 +8708,12 @@ Simple Tests
TESTS = foo.pl bar.pl baz.pl
@end example
-Note that the @option{parallel-tests} driver provides a more elegant
-way to achieve the same effect, freeing the @code{TESTS_ENVIRONMENT}
-variable for the user to override (@pxref{Simple Tests using
-parallel-tests}).
+It's important to note that this last use of @code{TESTS_ENVIRONMENT}
+is invalid when the @option{parallel-tests} option is active; however,
+the @option{parallel-tests} driver provides a more elegant way to
+achieve the same effect, with the further benefit of freeing the
+@code{TESTS_ENVIRONMENT} variable for the user
+(@pxref{Simple Tests using parallel-tests}).
@cindex Tests, expected failure
@@ -8739,7 +8758,9 @@ Simple Tests using parallel-tests
the @code{check_*} variables are honored, and the environment variable
@env{srcdir} is set during test execution. Also, @code{TESTS_ENVIRONMENT}
is still honored, but is complemented by a new developer-reserved variable
-@code{AM_TESTS_ENVIRONMENT} (described below).
+@code{AM_TESTS_ENVIRONMENT} (described below), and its allowed uses are
+somewhat restricted (other features of the @option{parallel-tests} driver
+compensate for this, though).
This test driver is still experimental and may undergo changes in order
to satisfy additional portability requirements.
@@ -8812,12 +8833,13 @@ Simple Tests using parallel-tests
but should be reserved for the user.
@vindex AM_TESTS_ENVIRONMENT
-The @code{AM_TESTS_ENVIRONMENT} variable can be used to run initialization
-code and set environment variables for the tests' runs. The user can
-still employ the @code{TESTS_ENVIRONMENT} variable to override settings
-from @code{AM_TESTS_ENVIRONMENT}. Note that, for implementation reasons,
-if the @code{AM_TESTS_ENVIRONMENT} variable is set, its contents
-@emph{must} be terminated by a semicolon.
+The @code{AM_TESTS_ENVIRONMENT} and @code{TESTS_ENVIRONMENT} variables can
+be used to run initialization code and set environment variables for the
+test scripts. The former variable is developer-reserved, and can be
+defined in the @file{Makefile.am}, while the latter is reserved for the
How about s/reserved // here, to reflect prior usage?
Well, I do think we should now discourage such prior usage...
+user, which can employ it to extend or override the settings in the
+former. For implementation reasons, if the @code{AM_TESTS_ENVIRONMENT}
s/\. / /
Fixed.
+variable is set, its contents @emph{must} be terminated by a semicolon.
Why is that necessarily so?
Otherwise the user couldn't safely use TESTS_ENVIRONMENT for overriding
AM_TESTS_ENVIRONMENT; for example, with foo=0 foo=1 COMMAND ..., some
shells (e.g., bash and dash) will export foo to `0' in the environment
of COMMAND, while other shells (e.g., Solaris /bin/sh) will export foo
to `1'. And BTW, it was you that pointed this out to me originally ;-)
@@ -8834,6 +8856,23 @@ Simple Tests using
Re: [FYI] New public temporary branch, about documentation for custom test drivers
On Monday 27 June 2011, Stefano Lattarini wrote:
On Monday 27 June 2011, Stefano Lattarini wrote:
Attached is the squash-in with the fixes you suggested. I hope to
be soon able to finish up this part of the documentation; I'll post
and additional squash-in when I'm done.
OK the documentation is basically complete now (it still lacks a
working example I'd like to add in the `doc/' subdir, similar to
what is done with the amhello package). I've pushed it to the
temp branch; attached is the diff w.r.t. the previous status, for
reference.
Regards,
Stefano
And please consider also this further squash-in as done.
Sorry for the noise,
Stefano
-*-*-*-
diff --git a/doc/automake.texi b/doc/automake.texi
index 39192ba..6b7b761 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -332,7 +332,7 @@ API for Custom Test Drivers
* Command-line arguments for test drivers::
* Log files generation and test results recording::
-* Console output during testsuite runs::
+* Testsuite progress output::
* HTML generation from testsuite logs::
Changing Automake's Behavior
@@ -9126,7 +9126,7 @@ explicitly stated otherwise.
@menu
* Command-line arguments for test drivers::
* Log files generation and test results recording::
-* Console output during testsuite runs::
+* Testsuite progress output::
* HTML generation from testsuite logs::
@end menu
@@ -9148,7 +9148,7 @@ Here is the list of options:
The name of the test, with VPATH prefix (if any) removed. This can have a
suffix and a directory component (as in e.g., @file{sub/foo.test}), and is
mostly meant to be used in console reports about testsuite advancements and
-results (@pxref{Console output during testsuite runs}).
+results (@pxref{Testsuite progress output}).
@item --log-file=@var{PATH}
The log file the test driver must create. If it has a directory component
(as in e.g., @file{sub/foo.log}), the Automake harness will ensure that
@@ -9258,8 +9258,8 @@ if a test script reports 8 successful test cases and 2
skipped test
cases, some drivers might report that globally as a SKIP, while others
as a PASS.
-@node Console output during testsuite runs
-@subsubsection Console output during testsuite runs
+@node Testsuite progress output
+@subsubsection Testsuite progress output
A custom test driver has also the task of displaying, on the standard
output, the test results as soon as they become available. Depending on
Re: [FYI] New public temporary branch, about documentation for custom test drivers
On Monday 27 June 2011, Stefano Lattarini wrote:
Attached is the squash-in with the fixes you suggested. I hope to
be soon able to finish up this part of the documentation; I'll post
and additional squash-in when I'm done.
OK the documentation is basically complete now (it still lacks a
working example I'd like to add in the `doc/' subdir, similar to
what is done with the amhello package). I've pushed it to the
temp branch; attached is the diff w.r.t. the previous status, for
reference.
Regards,
Stefano
diff --git a/ChangeLog b/ChangeLog
index 6acd090..628d879 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,4 +1,4 @@
-2011-06-25 Stefano Lattarini stefano.lattar...@gmail.com
+2011-06-27 Stefano Lattarini stefano.lattar...@gmail.com
docs: document custom test drivers and protocols
* doc/automake.texi (Simple Tests): Note that the TESTS_ENVIRONMENT
@@ -10,10 +10,12 @@
(Declaring Custom Test Drivers in @file{Makefile.am}): Likewise.
(APIs for Custom Test Drivers): Likewise.
(Options): Update description of color-tests.
- * lib/am/check (recheck, recheck-html): Minor adjustments to better
- conform to the documentation (this should cause no semantic changes
- w.r.t. the former behaviour); minor improvements and extensions to
- existing comments.
+ * lib/am/check ($(TEST_SUITE_LOG)): Remove comments that have been
+ moved into the manual.
+ (recheck, recheck-html): Minor adjustments to better conform to the
+ documentation (this should cause no semantic changes w.r.t. the
+ former behaviour); minor improvements and extensions to existing
+ comments.
* tests/test-driver-create-log-dir.test: New test.
* tests/test-driver-strip-vpath.test: Likewise.
* tests/test-driver-global-log.test: Likewise.
diff --git a/doc/automake.texi b/doc/automake.texi
index 8a6871c..39192ba 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -9119,9 +9119,9 @@ for new features or to satisfy additional portability requirements.
The main characteristic of these APIs is that they are designed to share
as much infrastructure, semantics, and implementation details as possible
-with the plain @option{parallel-tests} harness. So everything said in
-the previous section should apply here too, unless explicitly stated
-otherwise.
+with the @option{parallel-tests} harness and its default driver. So
+everything said in the previous section should apply here too, unless
+explicitly stated otherwise.
@menu
* Command-line arguments for test drivers::
@@ -9135,9 +9135,10 @@ otherwise.
A custom driver can rely on various command-line options and arguments
being passed to it automatically by the Automake's @option{parallel-tests}
-harness. It is @emph{mandatory} that it understands all of them (even if
-the exact interpretation of the associated semantics can change between a
-test driver and another, and even be a no-op in some drivers).
+harness. It is @emph{mandatory} that it understands all of them (even
+if the exact interpretation of the associated semantics can legitimately
+change between a test driver and another, and even be a no-op in some
+drivers).
@noindent
Here is the list of options:
@@ -9151,10 +9152,10 @@ results (@pxref{Console output during testsuite runs}).
@item --log-file=@var{PATH}
The log file the test driver must create. If it has a directory component
(as in e.g., @file{sub/foo.log}), the Automake harness will ensure that
-such directory exist before the test driver is called.
+such directory exist @emph{before} the test driver is called.
@item --color-tests=@{yes|no@}
-Whether the console output should be colored or not (@pxref{Simple
-tests and color-tests} to learn when this option gets activated and
+Whether the console output should be colorized or not (@pxref{Simple
+tests and color-tests}, to learn when this option gets activated and
when it doesn't).
@item --expect-failure=@{yes|no@}
Whether the tested program is expected to fail.
@@ -9181,23 +9182,14 @@ consistency and a more pleasant user experience.
@node Log files generation and test results recording
@subsubsection Log files generation and test results recording
-One of the main features of the new testsuite harness is the ability to
-support test protocols that allow a single test script to run more
-test cases, @emph{each with its distinct result}. In order for the
-testsuite summary report to be correct in this case, the test driver
-implementing support for the protocol must follow the following
-convention.
-
-TODO: In order to have a correct count of test cases in the final
-testsuite summary, the driver must use special syntax in the created
-.log file (see below, `:test-result:').
-
The test driver must correctly generate the file specified by the
-@option{--log-file} option (even when the tested program fails, of
-course!). This file is quite free-form, but still it has to conform
-to the following conventions:
+@option{--log-file} option (of course even when the tested program
+fails
Re: [FYI] New public temporary branch, about documentation for custom test drivers
* Stefano Lattarini wrote on Mon, Jun 27, 2011 at 01:11:51PM CEST:
On Monday 27 June 2011, Ralf Wildenhues wrote:
* Stefano Lattarini wrote on Sun, Jun 26, 2011 at 02:20:48PM CEST:
+The @code{AM_TESTS_ENVIRONMENT} and @code{TESTS_ENVIRONMENT} variables
can
+be used to run initialization code and set environment variables for the
+test scripts. The former variable is developer-reserved, and can be
+defined in the @file{Makefile.am}, while the latter is reserved for the
How about s/reserved // here, to reflect prior usage?
Well, I do think we should now discourage such prior usage...
Yeah, we should. It's very wide-spread however. And when you think in
terms of a developer eager to provide code portable to several Automake
versions, then changing that is very far down the road. gnulib is in
this position, Libtool is, and several projects that don't provide
macros for other projects or Makefile.am fragments also are eager to
work with a range of Automake releases (often starting from 1.9.6 on).
+variable is set, its contents @emph{must} be terminated by a semicolon.
Why is that necessarily so?
Otherwise the user couldn't safely use TESTS_ENVIRONMENT for overriding
AM_TESTS_ENVIRONMENT; for example, with foo=0 foo=1 COMMAND ..., some
shells (e.g., bash and dash) will export foo to `0' in the environment
of COMMAND, while other shells (e.g., Solaris /bin/sh) will export foo
to `1'. And BTW, it was you that pointed this out to me originally ;-)
See, I simply didn't think of TESTS_ENVIRONMENT. So, the documentation
should make it clear that variables actually need to be exported (not
just set); it can't hurt to say why the semicolon is useful: so that
the user can set TESTS_ENVIRONMENT to override variables further.
+to analyze their execution and outcome, to create the @file{.log} files
+associated to these test runs, and to display the test results on the
+console.
It is the sole responsibility of the author of the test driver
+to ensure that it implements all the above steps meaningfully and
+correctly; Automake isn't and can't be of any help here.
I'm wondering whether it's better to omit this sentence. It sounds like
a teacher talking to me; in programming, I am always expected to do
things correctly.
Yes, but often your tools offer diagnostic that helps to prevent some, or
many, mistakes. In case the tools can't or won't do that (like here),
extra warnings and be careful exhortations in the documentation are
IMHO warranted.
OK. How about s/sole // then? We might actually want to (or be able
to) provide help at some point in the future.
+@itemize
+@item
+concurrency through the use of @command{make}'s option @option{-j};
+@item
+per-test @file{.log} files, and generation of a summary @file{.log}
+file from them;
+@item
+@code{recheck} target and lazy reruns of tests;
+@item
+inter-test dependencies;
+@item
+support @code{check_*} variables (@code{check_PROGRAMS},
+@code{check_LIBRARIES}, ...);
+@item
+use of @code{VERBOSE} environment variable to get verbose output on
+testsuite failures;
+@item
+definition and honoring of @code{TESTS_ENVIRONMENT} and
+@code{AM_TESTS_ENVIRONMENT} variables;
+@item
+definition of generic and extension-specific @code{LOG_COMPILER} and
+@code{LOG_FLAGS} variables.
+@end itemize
+
@noindent?
You mean before the following lines? (I guess you do)
Yes.
+Custom testsuite drivers are declared by defining the make variables
+@code{LOG_DRIVER} or @code{@var{ext}_LOG_DRIVER} (where @var{ext} must
+be declared in @code{TEST_EXTENSIONS}). They must be defined to
+programs or scripts that will be used to drive the execution, logging,
+and outcome report of the tests with corresponding extensions (or of
+those with no registered extension in the case of @code{LOG_DRIVER}).
+Clearly, multiple distinct test drivers can be declared in the same
+@file{Makefile.am}. Note moreover that the @code{LOG_DRIVER} variables
+are @emph{not} a substitute for the @code{LOG_COMPILER} variables: the
+two sets of variables can, and often do, usefully and legitimately
+coexist.
+
+@c TODO: We should really be able to point to a clarifying example here!
Yep. As a reader, this leaves me confused. Why should there be two
separate script instances?
Actually, why should there (other than maybe for backward
compatibility)? I mean, not just in the documentation, but in the
Automake code as well? Can't we simplify this? Thanks.
Oh no! They are really meant to fullfill two orthogonal and complementary
needs. Let's see if an example can help to clarify the situation.
Assume you are testing a static code checker for C; in this case, you
might to write most of your test cases as C files with expected diagnostic
embedded in special comments, as in e.g.:
...
if (a =
