Re: [FYI] New public temporary branch, about documentation for custom test drivers
On Tuesday 28 June 2011, Stefano Lattarini wrote:
On Monday 27 June 2011, Ralf Wildenhues wrote:
* Stefano Lattarini wrote on Mon, Jun 27, 2011 at 01:11:51PM CEST:
+If the @code{AM_TESTS_ENVIRONMENT} variable is set, its contents
+@emph{must} be terminated by a semicolon.
Why is that necessarily so?
Otherwise the user couldn't safely use TESTS_ENVIRONMENT for overriding
AM_TESTS_ENVIRONMENT; for example, with foo=0 foo=1 COMMAND ..., some
shells (e.g., bash and dash) will export foo to `0' in the environment
of COMMAND, while other shells (e.g., Solaris /bin/sh) will export foo
to `1'. And BTW, it was you that pointed this out to me originally ;-)
See, I simply didn't think of TESTS_ENVIRONMENT. So, the documentation
should make it clear that variables actually need to be exported (not
just set); it can't hurt to say why the semicolon is useful: so that
the user can set TESTS_ENVIRONMENT to override variables further.
I like this, and I'll make the change. But notice that the documentation
about AM_TESTS_ENVIRONMENT we're referring to was already in master, so
that the right thing to do is to fix it there, merge master to the branch
'test-protocols', and rebase the documentation branch on that.
Here it is the patch I've applied to master.
From 3e334a272fa601bebb5896e25cfb63f34822a275 Mon Sep 17 00:00:00 2001
Message-Id:
3e334a272fa601bebb5896e25cfb63f34822a275.1309333773.git.stefano.lattar...@gmail.com
From: Stefano Lattarini stefano.lattar...@gmail.com
Date: Wed, 29 Jun 2011 09:44:14 +0200
Subject: [PATCH] docs: explain why AM_TESTS_ENVIRONMENT must be
semicolon-terminated
* doc/automake.texi (Simple Tests using parallel-tests): Ditto, and
related adjustments.
Suggestion by Ralf Wildenhues.
---
ChangeLog |7 +++
doc/automake.texi |8
2 files changed, 11 insertions(+), 4 deletions(-)
diff --git a/ChangeLog b/ChangeLog
index 7ce..64c2bbb 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,10 @@
+2011-06-29 Stefano Lattarini stefano.lattar...@gmail.com
+
+ docs: explain why AM_TESTS_ENVIRONMENT must be semicolon-terminated
+ * doc/automake.texi (Simple Tests using parallel-tests): Ditto, and
+ related adjustments.
+ Suggestion by Ralf Wildenhues.
+
2011-06-29 Stefano Lattarini stefano.lattar...@gmail.com
docs: fix unportable example of AM_TESTS_ENVIRONMENT usage
diff --git a/doc/automake.texi b/doc/automake.texi
index d60bfe4..da2e2b4 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -8740,10 +8740,10 @@ but should be reserved for the user.
The @code{AM_TESTS_ENVIRONMENT} variable can be used to run initialization
code and set environment variables for the tests' runs. The user can
still employ the @code{TESTS_ENVIRONMENT} variable to override settings
-from @code{AM_TESTS_ENVIRONMENT}. Note that, for implementation reasons,
-if the @code{AM_TESTS_ENVIRONMENT} variable is set, its contents
-@emph{must} be terminated by a semicolon. Here is an example of a
-slightly elaborate definition:
+from @code{AM_TESTS_ENVIRONMENT}; for that to work portably, however,
+the contents of a non-empty @code{AM_TESTS_ENVIRONMENT} @emph{must} be
+terminated by a semicolon. Here is an example of a slightly elaborate
+definition:
@example
AM_TESTS_ENVIRONMENT = \
--
1.7.2.3
Re: [FYI] New public temporary branch, about documentation for custom test drivers
On Tuesday 28 June 2011, Stefano Lattarini wrote:
Please go ahead after addressing my issues above. The questions I wrote
should still be discussed (and might need changing stuff again) but that
shouldn't hold up the patch.
OK, will push this evening or tomorrow.
I've pushed to 'test-protocols' now, and I've deleted the temporary branch
'GSoC/experimental/docs-custom-test-drivers'. Attached is the patch that
has been pushed, for reference.
Regards,
Stefano
From 1402741313cf6c2fe8611eb294f9a8111c994a32 Mon Sep 17 00:00:00 2001
Message-Id: 1402741313cf6c2fe8611eb294f9a8111c994a32.1309354565.git.stefano.lattar...@gmail.com
From: Stefano Lattarini stefano.lattar...@gmail.com
Date: Wed, 22 Jun 2011 21:59:34 +0200
Subject: [PATCH] docs: document custom test drivers and protocols
* doc/automake.texi (Simple Tests): Note that the TESTS_ENVIRONMENT
use suggested here is not portable to 'parallel-tests'.
(Simple Tests using parallel-tests): Document new restrictions on
the uses of TESTS_ENVIRONMENT and AM_TESTS_ENVIRONMENT.
(Custom Test Drivers): New section and node.
(Overview of Custom Test Drivers Support): New subsection.
(Declaring Custom Test Drivers in @file{Makefile.am}): Likewise.
(APIs for Custom Test Drivers): Likewise.
(Options): Update description of color-tests.
* lib/am/check ($(TEST_SUITE_LOG)): Remove comments that have been
moved into the manual.
(recheck, recheck-html): Minor adjustments to better conform to the
documentation (this should cause no semantic changes w.r.t. the
former behaviour); minor improvements and extensions to existing
comments.
* tests/test-driver-create-log-dir.test: New test.
* tests/test-driver-strip-vpath.test: Likewise.
* tests/test-driver-global-log.test: Likewise.
* tests/test-driver-recheck.test: Likewise.
* tests/Makefile.am (TESTS): Update.
---
ChangeLog | 24 ++
doc/automake.texi | 378 +++--
lib/Automake/tests/Makefile.in|2 +-
lib/am/check.am | 15 +-
tests/Makefile.am |4 +
tests/Makefile.in |6 +-
tests/test-driver-create-log-dir.test | 74 +++
tests/test-driver-global-log.test | 110 ++
tests/test-driver-recheck.test| 126 +++
tests/test-driver-strip-vpath.test| 96 +
10 files changed, 804 insertions(+), 31 deletions(-)
create mode 100755 tests/test-driver-create-log-dir.test
create mode 100755 tests/test-driver-global-log.test
create mode 100755 tests/test-driver-recheck.test
create mode 100755 tests/test-driver-strip-vpath.test
diff --git a/ChangeLog b/ChangeLog
index c99944a..6203f89 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,27 @@
+2011-06-29 Stefano Lattarini stefano.lattar...@gmail.com
+
+ docs: document custom test drivers and protocols
+ * doc/automake.texi (Simple Tests): Note that the TESTS_ENVIRONMENT
+ use suggested here is not portable to 'parallel-tests'.
+ (Simple Tests using parallel-tests): Document new restrictions on
+ the uses of TESTS_ENVIRONMENT and AM_TESTS_ENVIRONMENT.
+ (Custom Test Drivers): New section and node.
+ (Overview of Custom Test Drivers Support): New subsection.
+ (Declaring Custom Test Drivers in @file{Makefile.am}): Likewise.
+ (APIs for Custom Test Drivers): Likewise.
+ (Options): Update description of color-tests.
+ * lib/am/check ($(TEST_SUITE_LOG)): Remove comments that have been
+ moved into the manual.
+ (recheck, recheck-html): Minor adjustments to better conform to the
+ documentation (this should cause no semantic changes w.r.t. the
+ former behaviour); minor improvements and extensions to existing
+ comments.
+ * tests/test-driver-create-log-dir.test: New test.
+ * tests/test-driver-strip-vpath.test: Likewise.
+ * tests/test-driver-global-log.test: Likewise.
+ * tests/test-driver-recheck.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+
2011-06-29 Stefano Lattarini stefano.lattar...@gmail.com
docs: explain why AM_TESTS_ENVIRONMENT must be semicolon-terminated
diff --git a/doc/automake.texi b/doc/automake.texi
index 481ada4..a26f2a5 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -319,9 +319,23 @@ Support for test suites
* Simple Tests::Listing programs and scripts in @code{TESTS}
* Simple Tests using parallel-tests:: More powerful test driver
+* Custom Test Drivers:: Writing and using custom test drivers
* DejaGnu Tests:: Interfacing with the external testing framework
* Install Tests:: Running tests on installed packages
+Custom Test Drivers
+
+* Overview of Custom Test Drivers Support::
+* Declaring Custom Test Drivers::
+* API for Custom Test Drivers::
+
+API for Custom Test Drivers
+
+* Command-line arguments for test drivers::
+* Log files generation and test results recording::
+* Testsuite progress output::
+* HTML generation from testsuite logs::
+
Changing Automake's Behavior
[FYI] 'master' merged into 'test-protocols'
I've merged 'master' into 'test-protocols', and pushed. Regards, Stefano
[PATCH] parallel-tests: make parsing of test results safer (was: Re: [PATCH] {test-protocols} parallel-tests: fix bug in Automake's own testsuite)
Reference: http://lists.gnu.org/archive/html/automake-patches/2011-06/msg00199.html The new code for parsing the testsuite-generated `.log' files, as introduced in commit `v1.11-872-gc96b881', considers each `:test-result:' field anywhere in a `.log' file as a declaration of a test result, and accounts for it as such in the testsuite summary. Unfortunately this could easily cause spurious test failures being reported in the testsuite summary. This happened in practice with the Automake's own testsuite; for example: $ make check TESTS='check12-p.test'; echo exit: $? ... PASS: check12-p.test = 4 of 5 tests failed See tests/test-suite.log Please report to bug-autom...@gnu.org = make[2]: *** [test-suite.log] Error 1 make: *** [check-am] Error 2 exit: 2 This change introduces a new special `:test-result:' END, that, when seen, prevents the rest of the log file from being parsed. OK for the 'test-protocols' branch? Regards, Stefano From 310e80b275ed120af4999e0dad018b8a5732fd02 Mon Sep 17 00:00:00 2001 Message-Id: 310e80b275ed120af4999e0dad018b8a5732fd02.1309372874.git.stefano.lattar...@gmail.com From: Stefano Lattarini stefano.lattar...@gmail.com Date: Wed, 29 Jun 2011 16:02:52 +0200 Subject: [PATCH] parallel-tests: make parsing of test results safer The new code for parsing the testsuite-generated `.log' files, as introduced in commit `v1.11-872-gc96b881', considers each `:test-result:' field anywhere in a `.log' file as a declaration of a test result, and accounts for it as such in the testsuite summary. Unfortunately this could easily cause spurious test failures being reported in the testsuite summary. This happened in practice with the Automake's own testsuite; for example: $ make check TESTS='check12-p.test'; echo exit: $? ... PASS: check12-p.test = 4 of 5 tests failed See tests/test-suite.log Please report to bug-autom...@gnu.org = make[2]: *** [test-suite.log] Error 1 make: *** [check-am] Error 2 exit: 2 This change introduces a new special `:test-result:' END, that, when seen, prevents the rest of the log file from being parsed. For more information, refer to the thread: http://lists.gnu.org/archive/html/automake-patches/2011-06/msg00199.html * lib/am/check.am ($(TEST_SUITE_LOG)): Stop the parsing of a log file as soon as the special :test-result:END directive is seen. Related changes and enhancements. * lib/test-driver: Protect the rest of the log after the result lined with a :test-result:END directive. * tests/parallel-tests-no-spurious-summary.test: New test. * tests/test-driver-end-test-results.test: Likewise. * tests/Makefile.am (TESTS): Update. --- ChangeLog | 36 ++ doc/automake.texi | 21 +-- lib/Automake/tests/Makefile.in| 16 - lib/am/check.am | 31 +++- lib/test-driver |6 +- tests/Makefile.am |2 + tests/Makefile.in | 18 - tests/parallel-tests-no-spurious-summary.test | 66 ++ tests/test-driver-end-test-results.test | 91 + 9 files changed, 270 insertions(+), 17 deletions(-) create mode 100755 tests/parallel-tests-no-spurious-summary.test create mode 100755 tests/test-driver-end-test-results.test diff --git a/ChangeLog b/ChangeLog index 6203f89..ab7ba2f 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,5 +1,41 @@ 2011-06-29 Stefano Lattarini stefano.lattar...@gmail.com + parallel-tests: make parsing of test results safer + The new code for parsing the testsuite-generated `.log' files, + as introduced in commit `v1.11-872-gc96b881', considers each + `:test-result:' field anywhere in a `.log' file as a declaration + of a test result, and accounts for it as such in the testsuite + summary. Unfortunately this could easily cause spurious test + failures being reported in the testsuite summary. This happened + in practice with the Automake's own testsuite; for example: + $ make check TESTS='check12-p.test'; echo exit: $? + ... + PASS: check12-p.test + = + 4 of 5 tests failed + See tests/test-suite.log + Please report to bug-autom...@gnu.org + = + make[2]: *** [test-suite.log] Error 1 + make: *** [check-am] Error 2 + exit: 2 + This change introduces a new special `:test-result:' END, that, + when seen, prevents the rest of the log file from being parsed. + For more information, refer to the thread: + http://lists.gnu.org/archive/html/automake-patches/2011-06/msg00199.html + * lib/am/check.am ($(TEST_SUITE_LOG)): Stop the parsing of a log + file as soon as the special :test-result:END directive is seen. + Related
