I've applied the attached patch to the 'GSoC/experimental/test-results-work'
branch Note that this is not an FYI, since that branch is temporary, and can
thus be amended and modified. Reviews are welcome!
Thanks,
Stefano
From 2df72384ab57d165c7e1da0f88e513171a0035e8 Mon Sep 17 00:00:00 2001
Message-Id: <2df72384ab57d165c7e1da0f88e513171a0035e8.1309445709.git.stefano.lattar...@gmail.com>
From: Stefano Lattarini <stefano.lattar...@gmail.com>
Date: Thu, 30 Jun 2011 14:55:43 +0200
Subject: [PATCH] parallel-tests: new recognized test result 'ERROR'
* lib/am/check.am ($(TEST_SUITE_LOG)): Recognize a new test result
`ERROR'. Use it when encountering unreadable test logs (previously
a simple `FAIL' was used in this situations).
* lib/test-driver: Set the global test result to `ERROR' when the
test exit status is 99. When doing colorized output, color `ERROR'
results in magenta.
* doc/automake.texi (Log files generation and test results
recording): Update by also listing `ERROR' among the list of valid
`:test-results:' arguments.
* tests/trivial-test-driver: Update.
* tests/parallel-tests.test: Likewise.
* tests/parallel-tests-harderror.test: Likewise.
* tests/parallel-tests-no-spurious-summary.test: Likewise.
* tests/test-driver-global-log.test: Likewise.
* tests/test-driver-recheck.test: Likewise.
* tests/test-driver-custom-multitest-recheck.test: Likewise.
* tests/test-driver-custom-multitest-recheck2.test: Likewise.
* tests/test-driver-custom-multitest.test: Likewise.
* tests/test-driver-custom-no-html.test: Likewise.
* tests/test-driver-end-test-results.test: Likewise.
* tests/color.test: Likewise, and make stricter.
* tests/color2.test: Likewise, and improve syncing with color.test.
* tests/parallel-tests-exit-statuses.test: New test.
* tests/parallel-tests-console-output.test: New test.
* tests/Makefile.am (TESTS): Update.
---
ChangeLog | 29 ++++++
doc/automake.texi | 8 +-
lib/am/check.am | 8 +-
lib/test-driver | 5 +-
tests/Makefile.am | 2 +
tests/Makefile.in | 6 +-
tests/color.test | 48 ++++++----
tests/color2.test | 51 ++++++----
tests/parallel-tests-console-output.test | 104 +++++++++++++++++++++
tests/parallel-tests-exit-statuses.test | 108 ++++++++++++++++++++++
tests/parallel-tests-harderror.test | 6 +-
tests/parallel-tests-no-spurious-summary.test | 2 +
tests/parallel-tests.test | 9 +-
tests/test-driver-custom-multitest-recheck.test | 27 +++---
tests/test-driver-custom-multitest-recheck2.test | 56 ++++++++---
tests/test-driver-custom-multitest.test | 33 ++++---
tests/test-driver-custom-no-html.test | 2 +-
tests/test-driver-end-test-results.test | 1 +
tests/test-driver-global-log.test | 4 +-
tests/test-driver-recheck.test | 2 +-
tests/trivial-test-driver | 11 +-
21 files changed, 419 insertions(+), 103 deletions(-)
create mode 100755 tests/parallel-tests-console-output.test
create mode 100755 tests/parallel-tests-exit-statuses.test
diff --git a/ChangeLog b/ChangeLog
index ab7ba2f..22e7087 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,32 @@
+2011-06-30 Stefano Lattarini <stefano.lattar...@gmail.com>
+
+ parallel-tests: new recognized test result 'ERROR'
+ * lib/am/check.am ($(TEST_SUITE_LOG)): Recognize a new test result
+ `ERROR'. Use it when encountering unreadable test logs (previously
+ a simple `FAIL' was used in this situations).
+ * lib/test-driver: Set the global test result to `ERROR' when the
+ test exit status is 99. When doing colorized output, color `ERROR'
+ results in magenta.
+ * doc/automake.texi (Log files generation and test results
+ recording): Update by listing `ERROR' too among the list of valid
+ `:test-results:' arguments.
+ * tests/trivial-test-driver: Update.
+ * tests/parallel-tests.test: Likewise.
+ * tests/parallel-tests-harderror.test: Likewise.
+ * tests/parallel-tests-no-spurious-summary.test: Likewise.
+ * tests/test-driver-global-log.test: Likewise.
+ * tests/test-driver-recheck.test: Likewise.
+ * tests/test-driver-custom-multitest-recheck.test: Likewise.
+ * tests/test-driver-custom-multitest-recheck2.test: Likewise.
+ * tests/test-driver-custom-multitest.test: Likewise.
+ * tests/test-driver-custom-no-html.test: Likewise.
+ * tests/test-driver-end-test-results.test: Likewise.
+ * tests/color.test: Likewise, and make stricter.
+ * tests/color2.test: Likewise, and improve syncing with color.test.
+ * tests/parallel-tests-exit-statuses.test: New test.
+ * tests/parallel-tests-console-output.test: New test.
+ * tests/Makefile.am (TESTS): Update.
+
2011-06-29 Stefano Lattarini <stefano.lattar...@gmail.com>
parallel-tests: make parsing of test results safer
diff --git a/doc/automake.texi b/doc/automake.texi
index 1e251db..16804cd 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -9248,10 +9248,10 @@ leading whitespace will not be ignored.
@c Keep this in sync with lib/am/check-am:$(TEST_SUITE_LOG).
The only recognized test results are currently @code{PASS}, @code{XFAIL},
-@code{SKIP}, @code{FAIL} and @code{XPASS}. These results, when declared
-with @code{:test-result:}, can be optionally followed by text holding
-the name and/or a brief description of the corresponding test; the
-@option{parallel-tests} harness will ignore such extra text when
+@code{SKIP}, @code{FAIL}, @code{XPASS} and @code{ERROR}. These results,
+when declared with @code{:test-result:}, can be optionally followed by
+text holding the name and/or a brief description of the corresponding
+test; the @option{parallel-tests} harness will ignore such extra text when
generating @file{test-suite.log} and preparing the testsuite summary.
Also, @code{:test-result:} can be used with a special ``pseudo-result''
@code{END}, that will instruct the testsuite harness to stop scanning
diff --git a/lib/am/check.am b/lib/am/check.am
index f2528e0..76609b5 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -66,7 +66,7 @@ include inst-vars.am
## appended.
##
## In addition to the magic "exit 77 means SKIP" feature (which was
-## imported from automake), there is a magic "exit 99 means FAIL" feature
+## imported from automake), there is a magic "exit 99 means ERROR" feature
## which is useful if you need to issue a hard error no matter whether the
## test is XFAIL or not. You can disable this feature by setting the
## variable DISABLE_HARD_ERRORS to a nonempty value.
@@ -153,7 +153,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
## Readable test logs.
list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \
## Each unreadable test log counts as a failed test.
- results1=`for f in $$list; do test -r $$f || echo FAIL; done`; \
+ results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \
## Now we're going to extract the outcome of all the testcases from the
## test logs.
results2=''; \
@@ -187,6 +187,10 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
skip=`echo "$$results" | grep -c '^SKIP'`; \
xfail=`echo "$$results" | grep -c '^XFAIL'`; \
xpass=`echo "$$results" | grep -c '^XPASS'`; \
+ error=`echo "$$results" | grep -c '^ERROR'`; \
+## FIXME: for the moment, we count errors as failures, otherwise the code
+## that display the testsuite summary will become too much complicated.
+ fail=`expr $$fail + $$error`; \
failures=`expr $$fail + $$xpass`; \
all=`expr $$all - $$skip`; \
if test "$$all" -eq 1; then tests=test; All=; \
diff --git a/lib/test-driver b/lib/test-driver
index 145bbea..aa39c5b 100755
--- a/lib/test-driver
+++ b/lib/test-driver
@@ -85,9 +85,10 @@ if test $color_tests = yes; then
grn='[0;32m' # Green.
lgn='[1;32m' # Light green.
blu='[1;34m' # Blue.
+ mgn='[0;35m' # Magenta.
std='[m' # No color.
else
- red= grn= lgn= blu= std=
+ red= grn= lgn= blu= mgn= std=
fi
tmpfile=$logfile-t
@@ -109,7 +110,7 @@ case $estatus:$expect_failure in
0:yes) col=$red; res=XPASS;;
0:*) col=$grn; res=PASS ;;
77:*) col=$blu; res=SKIP ;;
- 99:*) col=$red; res=FAIL ;;
+ 99:*) col=$mgn; res=ERROR;;
*:yes) col=$lgn; res=XFAIL;;
*:*) col=$red; res=FAIL ;;
esac
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 9c0605d..64299b8 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -731,6 +731,8 @@ parallel-tests-log-override-recheck.test \
parallel-tests-empty-testlogs.test \
parallel-test-driver-install.test \
parallel-tests-no-spurious-summary.test \
+parallel-tests-exit-statuses.test \
+parallel-tests-console-output.test \
test-driver-end-test-results.test \
test-driver-custom-no-extra-driver.test \
test-driver-custom.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index dc03179..5ef82da 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -991,6 +991,8 @@ parallel-tests-log-override-recheck.test \
parallel-tests-empty-testlogs.test \
parallel-test-driver-install.test \
parallel-tests-no-spurious-summary.test \
+parallel-tests-exit-statuses.test \
+parallel-tests-console-output.test \
test-driver-end-test-results.test \
test-driver-custom-no-extra-driver.test \
test-driver-custom.test \
@@ -1400,7 +1402,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
nlinit=`echo 'nl="'; echo '"'`; eval "$$nlinit"; unset nlinit; \
list='$(TEST_LOGS)'; \
list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \
- results1=`for f in $$list; do test -r $$f || echo FAIL; done`; \
+ results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \
results2=''; \
exec 5<&0; \
for f in $$list2; do \
@@ -1421,6 +1423,8 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
skip=`echo "$$results" | grep -c '^SKIP'`; \
xfail=`echo "$$results" | grep -c '^XFAIL'`; \
xpass=`echo "$$results" | grep -c '^XPASS'`; \
+ error=`echo "$$results" | grep -c '^ERROR'`; \
+ fail=`expr $$fail + $$error`; \
failures=`expr $$fail + $$xpass`; \
all=`expr $$all - $$skip`; \
if test "$$all" -eq 1; then tests=test; All=; \
diff --git a/tests/color.test b/tests/color.test
index ecd4f94..bcaf17b 100755
--- a/tests/color.test
+++ b/tests/color.test
@@ -22,18 +22,24 @@
TERM=ansi
export TERM
-red='[0;31m'
-grn='[0;32m'
-lgn='[1;32m'
-blu='[1;34m'
-std='[m'
+esc=''
+red="$esc[0;31m"
+grn="$esc[0;32m"
+lgn="$esc[1;32m"
+blu="$esc[1;34m"
+mgn="$esc[0;35m"
+std="$esc[m"
+
+# Tho old serial testsuite driver doesn't distinguish between failures
+# and hard errors.
+test x"$parallel_tests" = x"yes" || mgn=$red
# Check that grep can parse nonprinting characters.
# BSD 'grep' works from a pipe, but not a seekable file.
# GNU or BSD 'grep -a' works on files, but is not portable.
-case `echo "$std" | grep .` in
+case `echo "$std" | $EGREP .` in
"$std") ;;
- *) echo "$me: grep can't parse nonprinting characters" >&2; Exit 77;;
+ *) echo "$me: egrep can't parse nonprinting characters" >&2; Exit 77;;
esac
cat >>configure.in <<END
@@ -43,7 +49,7 @@ END
cat >Makefile.am <<'END'
AUTOMAKE_OPTIONS = color-tests
TESTS = $(check_SCRIPTS)
-check_SCRIPTS = pass fail skip xpass xfail
+check_SCRIPTS = pass fail skip xpass xfail error
XFAIL_TESTS = xpass xfail
END
@@ -62,9 +68,14 @@ cat >skip <<END
exit 77
END
+cat >error <<END
+#! /bin/sh
+exit 99
+END
+
cp fail xfail
cp pass xpass
-chmod +x pass fail skip xpass xfail
+chmod +x pass fail skip xpass xfail error
$ACLOCAL
$AUTOCONF
@@ -75,21 +86,20 @@ $AUTOMAKE --add-missing
test_color ()
{
# Not a useless use of cat; see above comments about grep.
- cat stdout | grep ": pass" | $FGREP "$grn"
- cat stdout | grep ": fail" | $FGREP "$red"
- cat stdout | grep ": xfail" | $FGREP "$lgn"
- cat stdout | grep ": xpass" | $FGREP "$red"
- cat stdout | grep ": skip" | $FGREP "$blu"
+ cat stdout | $EGREP ": pass" | $FGREP "$grn"
+ cat stdout | $EGREP ": fail" | $FGREP "$red"
+ cat stdout | $EGREP ": xfail" | $FGREP "$lgn"
+ cat stdout | $EGREP ": xpass" | $FGREP "$red"
+ cat stdout | $EGREP ": skip" | $FGREP "$blu"
+ cat stdout | $EGREP ": error" | $FGREP "$mgn"
}
test_no_color ()
{
# Not a useless use of cat; see above comments about grep.
- cat stdout | grep ": pass" | $FGREP "$grn" && Exit 1
- cat stdout | grep ": fail" | $FGREP "$red" && Exit 1
- cat stdout | grep ": xfail" | $FGREP "$lgn" && Exit 1
- cat stdout | grep ": xpass" | $FGREP "$red" && Exit 1
- cat stdout | grep ": skip" | $FGREP "$blu" && Exit 1
+ cat stdout \
+ | $EGREP ": (pass|fail|xfail|xpass|skip|error)" \
+ | $FGREP "$esc" && Exit 1
:
}
diff --git a/tests/color2.test b/tests/color2.test
index 4f8862e..f8996ce 100755
--- a/tests/color2.test
+++ b/tests/color2.test
@@ -22,18 +22,24 @@
TERM=ansi
export TERM
-red='[0;31m'
-grn='[0;32m'
-lgn='[1;32m'
-blu='[1;34m'
-std='[m'
+esc=''
+red="$esc[0;31m"
+grn="$esc[0;32m"
+lgn="$esc[1;32m"
+blu="$esc[1;34m"
+mgn="$esc[0;35m"
+std="$esc[m"
+
+# Tho old serial testsuite driver doesn't distinguish between failures
+# and hard errors.
+test x"$parallel_tests" = x"yes" || mgn=$red
# Check that grep can parse nonprinting characters.
# BSD 'grep' works from a pipe, but not a seekable file.
# GNU or BSD 'grep -a' works on files, but is not portable.
-case `echo "$std" | grep .` in
+case `echo "$std" | $EGREP .` in
"$std") ;;
- *) echo "$me: grep can't parse nonprinting characters" >&2; Exit 77;;
+ *) echo "$me: egrep can't parse nonprinting characters" >&2; Exit 77;;
esac
# Check that we have a working expect program.
@@ -56,7 +62,7 @@ END
cat >Makefile.am <<'END'
AUTOMAKE_OPTIONS = color-tests
TESTS = $(check_SCRIPTS)
-check_SCRIPTS = pass fail skip xpass xfail
+check_SCRIPTS = pass fail skip xpass xfail error
XFAIL_TESTS = xpass xfail
END
@@ -75,33 +81,38 @@ cat >skip <<END
exit 77
END
+cat >error <<END
+#! /bin/sh
+exit 99
+END
+
cp fail xfail
cp pass xpass
-chmod +x pass fail skip xpass xfail
+chmod +x pass fail skip xpass xfail error
$ACLOCAL
-$AUTOMAKE -a
$AUTOCONF
+$AUTOMAKE --add-missing
+
./configure
test_color ()
{
# Not a useless use of cat; see above comments about grep.
- cat stdout | grep ": pass" | $FGREP "$grn"
- cat stdout | grep ": fail" | $FGREP "$red"
- cat stdout | grep ": xfail" | $FGREP "$lgn"
- cat stdout | grep ": xpass" | $FGREP "$red"
- cat stdout | grep ": skip" | $FGREP "$blu"
+ cat stdout | $EGREP ": pass" | $FGREP "$grn"
+ cat stdout | $EGREP ": fail" | $FGREP "$red"
+ cat stdout | $EGREP ": xfail" | $FGREP "$lgn"
+ cat stdout | $EGREP ": xpass" | $FGREP "$red"
+ cat stdout | $EGREP ": skip" | $FGREP "$blu"
+ cat stdout | $EGREP ": error" | $FGREP "$mgn"
}
test_no_color ()
{
# Not a useless use of cat; see above comments about grep.
- cat stdout | grep ": pass" | $FGREP "$grn" && Exit 1
- cat stdout | grep ": fail" | $FGREP "$red" && Exit 1
- cat stdout | grep ": xfail" | $FGREP "$lgn" && Exit 1
- cat stdout | grep ": xpass" | $FGREP "$red" && Exit 1
- cat stdout | grep ": skip" | $FGREP "$blu" && Exit 1
+ cat stdout \
+ | $EGREP ": (pass|fail|xfail|xpass|skip|error)" \
+ | $FGREP "$esc" && Exit 1
:
}
diff --git a/tests/parallel-tests-console-output.test b/tests/parallel-tests-console-output.test
new file mode 100755
index 0000000..5c59b72
--- /dev/null
+++ b/tests/parallel-tests-console-output.test
@@ -0,0 +1,104 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# parallel-tests: some checks on console output about testsuite
+# advancement.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+XFAIL_TESTS = sub/xpass.test xfail.test error.test
+TESTS = $(XFAIL_TESTS) fail.test pass.test a/b/skip.test sub/error2.test
+pass.log: fail.log
+error.log: pass.log
+sub/xpass.log: error.log
+sub/error2.log: xfail.log
+a/b/skip.log: sub/error2.log
+END
+
+cat > exp <<'END'
+FAIL: fail.test
+PASS: pass.test
+ERROR: error.test
+XPASS: sub/xpass.test
+XFAIL: xfail.test
+ERROR: sub/error2.test
+SKIP: a/b/skip.test
+END
+
+mkdir sub a a/b
+
+cat > pass.test << 'END'
+#!/bin/sh
+exit 0
+END
+cp pass.test sub/xpass.test
+
+cat > fail.test << 'END'
+#!/bin/sh
+exit 1
+END
+
+cat > xfail.test << 'END'
+#!/bin/sh
+# The sleep should ensure expected execution order of tests
+# even when make is run in parallel mode.
+# FIXME: quotes below required by maintainer-check
+sleep '10'
+exit 1
+END
+
+cat > error.test << 'END'
+#!/bin/sh
+exit 99
+END
+cp error.test sub/error2.test
+
+cat > a/b/skip.test << 'END'
+#!/bin/sh
+exit 77
+END
+
+chmod a+x pass.test fail.test xfail.test sub/xpass.test \
+ a/b/skip.test error.test sub/error2.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+for vpath in : false; do
+ if $vpath; then
+ mkdir build
+ cd build
+ srcdir=..
+ else
+ srcdir=.
+ fi
+ $srcdir/configure
+ $MAKE check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ LC_ALL=C grep '^[A-Z][A-Z]*:' stdout > got
+ cat got
+ diff $srcdir/exp got
+ cd $srcdir
+done
+
+:
diff --git a/tests/parallel-tests-exit-statuses.test b/tests/parallel-tests-exit-statuses.test
new file mode 100755
index 0000000..531526a
--- /dev/null
+++ b/tests/parallel-tests-exit-statuses.test
@@ -0,0 +1,108 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check parallel-tests features: normal and special exit statuses
+# in the test scripts.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+# $failure_statuses should be defined to the list of all integers between
+# 1 and 255 (inclusive), excluded 77 and 99.
+# Let's use `seq' if available, it's faster than the loop.
+failure_statuses=`seq 1 255 \
+ || { i=1; while test $i -le 255; do echo $i; i=\`expr $i + 1\`; done; }`
+failure_statuses=`
+ for i in $failure_statuses; do
+ test $i -eq 77 || test $i -eq 99 || echo $i
+ done | tr "$nl" ' '`
+# For debugging.
+echo "failure_statuses: $failure_statuses"
+# Sanity check.
+test `for st in $failure_statuses; do echo $st; done | wc -l` -eq 253 \
+ || fatal_ "initializing list of exit statuses for simple failures"
+
+cat > Makefile.am <<END
+LOG_COMPILER = ./do-exit
+fail_tests = $failure_statuses
+TESTS = 0 77 99 $failure_statuses
+\$(TESTS):
+END
+
+cat > do-exit <<'END'
+#!/bin/sh
+echo "$0: $1"
+case $1 in
+ [0-9]|[0-9][0-9]|[0-9][0-9][0-9]) st=$1;;
+ */[0-9]|*/[0-9][0-9]|*/[0-9][0-9][0-9]) st=`echo x"$1" | sed 's|.*/||'`;;
+ *) st=99;;
+esac
+exit $st
+END
+chmod a+x do-exit
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+{
+ echo PASS: 0
+ echo SKIP: 77
+ echo ERROR: 99
+ for st in $failure_statuses; do
+ echo "FAIL: $st"
+ done
+} | LC_ALL=C sort > exp-fail
+
+sed 's/^FAIL:/XFAIL:/' exp-fail | LC_ALL=C sort > exp-xfail-1
+sed '/^ERROR:/d' exp-xfail-1 > exp-xfail-2
+
+sort exp-fail
+sort exp-xfail-1
+sort exp-xfail-2
+
+./configure
+
+st=1
+$MAKE check >stdout && st=0
+cat stdout
+cat test-suite.log
+test $st -gt 0 || Exit 1
+LC_ALL=C grep '^[A-Z][A-Z]*:' stdout | LC_ALL=C sort > got-fail
+diff exp-fail got-fail
+
+st=1
+XFAIL_TESTS="$failure_statuses 99" $MAKE -e check >stdout && st=0
+cat stdout
+cat test-suite.log
+test $st -gt 0 || Exit 1
+LC_ALL=C grep '^[A-Z][A-Z]*:' stdout | LC_ALL=C sort > got-xfail-1
+diff exp-xfail-1 got-xfail-1
+
+st=0
+XFAIL_TESTS="$failure_statuses" TESTS="0 77 $failure_statuses" \
+ $MAKE -e check >stdout || st=$?
+cat stdout
+cat test-suite.log
+test $st -eq 0 || Exit 1
+LC_ALL=C grep '^[A-Z][A-Z]*:' stdout | LC_ALL=C sort > got-xfail-2
+diff exp-xfail-2 got-xfail-2
+
+:
diff --git a/tests/parallel-tests-harderror.test b/tests/parallel-tests-harderror.test
index 8680e29..f30fd52 100755
--- a/tests/parallel-tests-harderror.test
+++ b/tests/parallel-tests-harderror.test
@@ -69,13 +69,13 @@ DISABLE_HARD_ERRORS=x $MAKE check
$MAKE check DISABLE_HARD_ERRORS='' && Exit 1
cat test-suite.log
-grep '^FAIL: foo\.test .*exit.*99' test-suite.log
+grep '^ERROR: foo\.test .*exit.*99' test-suite.log
cd sub
# The `-e' is wanted here.
DISABLE_HARD_ERRORS='' $MAKE -e check && Exit 1
cat test-suite.log
-grep '^FAIL: bar\.test .*exit.*99' test-suite.log
+grep '^ERROR: bar\.test .*exit.*99' test-suite.log
cd ..
# Check the distributions.
@@ -90,7 +90,7 @@ $AUTOMAKE Makefile
./config.status Makefile
VERBOSE=yes $MAKE check && Exit 1
grep '^FAIL' test-suite.log && Exit 1
-grep '^FAIL: bar\.test .*exit.*99' sub/test-suite.log
+grep '^ERROR: bar\.test .*exit.*99' sub/test-suite.log
echo 'DISABLE_HARD_ERRORS = zardoz' >> sub/Makefile
VERBOSE=yes $MAKE check
diff --git a/tests/parallel-tests-no-spurious-summary.test b/tests/parallel-tests-no-spurious-summary.test
index 43c54a5..8b9547a 100755
--- a/tests/parallel-tests-no-spurious-summary.test
+++ b/tests/parallel-tests-no-spurious-summary.test
@@ -32,10 +32,12 @@ cat > foo.test <<'END'
#! /bin/sh
echo :test-result:XFAIL
echo :test-result: SKIP
+echo :test-result:ERROR
exit 0
END
cat > bar.test <<'END'
#! /bin/sh
+echo :test-result: ERROR
echo :test-result:FAIL
echo :test-result: XPASS
exit 0
diff --git a/tests/parallel-tests.test b/tests/parallel-tests.test
index f046c3f..95f78e7 100755
--- a/tests/parallel-tests.test
+++ b/tests/parallel-tests.test
@@ -68,10 +68,13 @@ $AUTOMAKE -a
$MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
-# There should be two errors: bar.test is a hard error.
-test `grep -c '^FAIL' stdout` -eq 2
+# There should be one failure and one hard error.
+test `grep -c '^FAIL:' stdout` -eq 1
+test `grep -c '^ERROR:' stdout` -eq 1
test -f mylog.log
-test `grep -c '^FAIL' mylog.log` -eq 2
+cat mylog.log
+test `grep -c '^FAIL:' mylog.log` -eq 1
+test `grep -c '^ERROR:' mylog.log` -eq 1
test -f baz.log
test -f bar.log
test -f foo.log
diff --git a/tests/test-driver-custom-multitest-recheck.test b/tests/test-driver-custom-multitest-recheck.test
index e0f010f..c178fa0 100755
--- a/tests/test-driver-custom-multitest-recheck.test
+++ b/tests/test-driver-custom-multitest-recheck.test
@@ -49,7 +49,7 @@ echo PASS:
if test -f b.ok; then
echo PASS:
else
- echo FAIL:
+ echo ERROR:
fi
: > b.run
END
@@ -97,14 +97,15 @@ do_recheck ()
do_count ()
{
- pass=ERR fail=ERR xpass=ERR xfail=ERR skip=ERR
+ pass=ERR fail=ERR xpass=ERR xfail=ERR skip=ERR error=ERR
eval "$@"
- $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP)' stdout || : # For debugging.
+ $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP|ERROR)' stdout || : # For debugging.
test `grep -c '^PASS:' stdout` -eq $pass
test `grep -c '^FAIL:' stdout` -eq $fail
test `grep -c '^XPASS:' stdout` -eq $xpass
test `grep -c '^XFAIL:' stdout` -eq $xfail
test `grep -c '^SKIP:' stdout` -eq $skip
+ test `grep -c '^ERROR:' stdout` -eq $error
}
for vpath in : false; do
@@ -129,7 +130,7 @@ for vpath in : false; do
test ! -r c.log
test ! -r d.run
test ! -r d.log
- do_count pass=0 fail=0 xpass=0 xfail=0 skip=0
+ do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
: Run the tests for the first time.
$MAKE check >stdout && { cat stdout; Exit 1; }
@@ -140,7 +141,7 @@ for vpath in : false; do
test -f b.run
test -f c.run
test -f d.run
- do_count pass=3 fail=3 xpass=1 xfail=1 skip=1
+ do_count pass=3 fail=2 xpass=1 xfail=1 skip=1 error=1
: Let us make b.test pass.
echo OK > b.ok
@@ -151,7 +152,7 @@ for vpath in : false; do
test -f b.run
test -f c.run
test -f d.run
- do_count pass=2 fail=2 xpass=1 xfail=1 skip=1
+ do_count pass=2 fail=2 xpass=1 xfail=1 skip=1 error=0
: Let us make the first part of c.test pass.
echo OK > c.pass
@@ -160,7 +161,7 @@ for vpath in : false; do
test ! -r b.run
test -f c.run
test -f d.run
- do_count pass=1 fail=1 xpass=1 xfail=1 skip=1
+ do_count pass=1 fail=1 xpass=1 xfail=1 skip=1 error=0
: Let us make also the second part of c.test pass.
echo KO > c.xfail
@@ -169,7 +170,7 @@ for vpath in : false; do
test ! -r b.run
test -f c.run
test -f d.run
- do_count pass=1 fail=1 xpass=0 xfail=2 skip=1
+ do_count pass=1 fail=1 xpass=0 xfail=2 skip=1 error=0
: Nothing changed, so only d.test should be run.
for i in 1 2; do
@@ -178,7 +179,7 @@ for vpath in : false; do
test ! -r b.run
test ! -r c.run
test -f d.run
- do_count pass=0 fail=1 xpass=0 xfail=0 skip=1
+ do_count pass=0 fail=1 xpass=0 xfail=0 skip=1 error=0
done
: Let us make d.test run more testcases, and experience _more_ failures.
@@ -190,14 +191,16 @@ for vpath in : false; do
echo XPASS: xp
echo FAIL: f 3
echo FAIL: f 4
+ echo ERROR: e 1
echo PASS: p 2
+ echo ERROR: e 2
END
do_recheck --fail
test ! -r a.run
test ! -r b.run
test ! -r c.run
test -f d.run
- do_count pass=2 fail=4 xpass=1 xfail=0 skip=2
+ do_count pass=2 fail=4 xpass=1 xfail=0 skip=2 error=2
: Let us finally make d.test pass.
echo : > d.extra
@@ -206,7 +209,7 @@ END
test ! -r b.run
test ! -r c.run
test -f d.run
- do_count pass=0 fail=0 xpass=0 xfail=0 skip=1
+ do_count pass=0 fail=0 xpass=0 xfail=0 skip=1 error=0
: All tests have been successful or skipped, nothing should be re-run.
do_recheck --pass
@@ -214,7 +217,7 @@ END
test ! -r b.run
test ! -r c.run
test ! -r d.run
- do_count pass=0 fail=0 xpass=0 xfail=0 skip=0
+ do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
cd $srcdir
diff --git a/tests/test-driver-custom-multitest-recheck2.test b/tests/test-driver-custom-multitest-recheck2.test
index a18bdd8..d2e979e 100755
--- a/tests/test-driver-custom-multitest-recheck2.test
+++ b/tests/test-driver-custom-multitest-recheck2.test
@@ -56,7 +56,13 @@ END
cat > c.test << 'END'
#! /bin/sh
-echo XPASS: xp
+if test -f c.err; then
+ echo ERROR: xxx
+elif test -f c.ok; then
+ echo PASS: ok
+else
+ echo XPASS: xp
+fi
: > c.run
END
@@ -68,14 +74,15 @@ $AUTOMAKE
do_count ()
{
- pass=ERR fail=ERR xpass=ERR xfail=ERR skip=ERR
+ pass=ERR fail=ERR xpass=ERR xfail=ERR skip=ERR error=ERR
eval "$@"
- $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP)' stdout || : # For debugging.
+ $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP|ERROR)' stdout || : # For debugging.
test `grep -c '^PASS:' stdout` -eq $pass
test `grep -c '^FAIL:' stdout` -eq $fail
test `grep -c '^XPASS:' stdout` -eq $xpass
test `grep -c '^XFAIL:' stdout` -eq $xfail
test `grep -c '^SKIP:' stdout` -eq $skip
+ test `grep -c '^ERROR:' stdout` -eq $error
}
for vpath in : false; do
@@ -96,7 +103,7 @@ for vpath in : false; do
test -f a.run
test -f b.run
test -f c.run
- do_count pass=2 fail=1 xpass=1 xfail=0 skip=1
+ do_count pass=2 fail=1 xpass=1 xfail=0 skip=1 error=0
rm -f *.run
@@ -104,7 +111,7 @@ for vpath in : false; do
for var in TESTS TEST_LOGS; do
env "$var=" $MAKE -e recheck >stdout || { cat stdout; Exit 1; }
cat stdout
- do_count pass=0 fail=0 xpass=0 xfail=0 skip=0
+ do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
test ! -r a.run
test ! -r b.run
test ! -r c.run
@@ -115,7 +122,7 @@ for vpath in : false; do
env TESTS=a.test $MAKE -e recheck >stdout \
|| { cat stdout; Exit 1; }
cat stdout
- do_count pass=0 fail=0 xpass=0 xfail=0 skip=0
+ do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
test ! -r a.run
test ! -r b.run
test ! -r c.run
@@ -128,7 +135,7 @@ for vpath in : false; do
test ! -r a.run
test -f b.run
test ! -r c.run
- do_count pass=0 fail=0 xpass=0 xfail=1 skip=1
+ do_count pass=0 fail=0 xpass=0 xfail=1 skip=1 error=0
rm -f *.run
@@ -136,35 +143,54 @@ for vpath in : false; do
TEST_LOGS=b.log $MAKE -e recheck >stdout \
|| { cat stdout; Exit 1; }
cat stdout
- do_count pass=0 fail=0 xpass=0 xfail=0 skip=0
+ do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
test ! -r a.run
test ! -r b.run
test ! -r c.run
TESTS='a.test b.test' $MAKE -e recheck >stdout \
|| { cat stdout; Exit 1; }
cat stdout
- do_count pass=0 fail=0 xpass=0 xfail=0 skip=0
+ do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
test ! -r a.run
test ! -r b.run
test ! -r c.run
- # An XPASS should count a failure.
+ : No need to re-run a.test anymore, but c.test should be rerun,
+ : as it contained an XPASS. And this time, make it fail with
+ : an hard error.
+ : > c.err
env TEST_LOGS='a.log c.log' $MAKE -e recheck >stdout \
&& { cat stdout; Exit 1; }
cat stdout
- do_count pass=0 fail=0 xpass=1 xfail=0 skip=0
+ do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=1
test ! -r a.run
test ! -r b.run
test -f c.run
- rm -f *.run
- env TESTS='c.test b.test' $MAKE -e recheck >stdout \
- && { cat stdout; Exit 1; }
+
+ rm -f *.run *.err
+
+ : c.test contained and hard error the last time, so it should be re-run.
+ : This time, make it pass
+ : > c.ok
+ env TESTS='c.test a.test' $MAKE -e recheck >stdout \
+ || { cat stdout; Exit 1; }
cat stdout
- do_count pass=0 fail=0 xpass=1 xfail=0 skip=0
+ do_count pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
test ! -r a.run
test ! -r b.run
test -f c.run
+ rm -f *.run *.err *.ok
+
+ : Nothing should be rerun anymore, as all tests have been evenatually
+ : succesfull.
+ $MAKE recheck >stdout || { cat stdout; Exit 1; }
+ cat stdout
+ do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
+ test ! -r a.run
+ test ! -r b.run
+ test ! -r c.run
+
cd $srcdir
done
diff --git a/tests/test-driver-custom-multitest.test b/tests/test-driver-custom-multitest.test
index a1c1010..2366b5b 100755
--- a/tests/test-driver-custom-multitest.test
+++ b/tests/test-driver-custom-multitest.test
@@ -40,7 +40,7 @@ TESTS = \
pass-fail.t \
pass4-skip.t \
pass3-skip2-xfail.t \
- pass-xpass-fail-xfail-skip.t
+ pass-xpass-fail-xfail-skip-error.t
END
expected_pass=10
@@ -48,6 +48,7 @@ expected_fail=5
expected_skip=4
expected_xfail=2
expected_xpass=1
+expected_error=1
cat > pass.t << 'END'
echo %% pass %%
@@ -99,13 +100,14 @@ mv || echo "XFAIL: mv cannot read users' mind yet"
exit 127
END
-cat > pass-xpass-fail-xfail-skip.t << 'END'
+cat > pass-xpass-fail-xfail-skip-error.t << 'END'
echo PASS:
echo FAIL:
echo XFAIL:
echo XPASS:
echo SKIP:
-echo %% pass-xpass-fail-xfail-skip %%
+echo ERROR:
+echo %% pass-xpass-fail-xfail-skip-error %%
END
chmod a+x *.t
@@ -139,25 +141,28 @@ for vpath in : false; do
cat pass-fail.log
cat pass4-skip.log
cat pass3-skip2-xfail.log
- cat pass-xpass-fail-xfail-skip.log
+ cat pass-xpass-fail-xfail-skip-error.log
# For debugging.
- $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP)' stdout
+ $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP|ERROR)' stdout
test `grep -c '^PASS:' stdout` -eq $expected_pass
test `grep -c '^FAIL:' stdout` -eq $expected_fail
test `grep -c '^XPASS:' stdout` -eq $expected_xpass
test `grep -c '^XFAIL:' stdout` -eq $expected_xfail
test `grep -c '^SKIP:' stdout` -eq $expected_skip
+ test `grep -c '^ERROR:' stdout` -eq $expected_error
- grep '^PASS: pass-xpass-fail-xfail-skip.t\, testcase 1' stdout
- grep '^FAIL: pass-xpass-fail-xfail-skip\.t, testcase 2' stdout
- grep '^XFAIL: pass-xpass-fail-xfail-skip\.t, testcase 3' stdout
- grep '^XPASS: pass-xpass-fail-xfail-skip\.t, testcase 4' stdout
- grep '^SKIP: pass-xpass-fail-xfail-skip\.t, testcase 5' stdout
+ tst=pass-xpass-fail-xfail-skip-error
+ grep "^PASS: $tst\.t, testcase 1" stdout
+ grep "^FAIL: $tst\.t, testcase 2" stdout
+ grep "^XFAIL: $tst\.t, testcase 3" stdout
+ grep "^XPASS: $tst\.t, testcase 4" stdout
+ grep "^SKIP: $tst\.t, testcase 5" stdout
+ grep "^ERROR: $tst\.t, testcase 6" stdout
# Check testsuite summary printed on console.
sed -e 's/[()]/ /g' -e 's/^/ /' stdout > t
- grep ' 6 of 18 ' t
+ grep ' 7 of 19 ' t
grep ' 1 unexpected pass' t
grep ' 4 test.* not run' t
@@ -170,7 +175,7 @@ for vpath in : false; do
grep '%% fail %%' test-suite.log
grep '%% fail2 %%' test-suite.log
grep '%% pass-fail %%' test-suite.log
- grep '%% pass-xpass-fail-xfail-skip %%' test-suite.log
+ grep '%% pass-xpass-fail-xfail-skip-error %%' test-suite.log
test `grep -c '%% ' test-suite.log` -eq 4
TESTS='pass.t pass3-skip2-xfail.t' $MAKE -e check >stdout \
@@ -178,11 +183,11 @@ for vpath in : false; do
cat test-suite.log
cat stdout
# For debugging.
- $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP)' stdout
+ $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP|ERROR)' stdout
test `grep -c '^PASS:' stdout` -eq 4
test `grep -c '^SKIP:' stdout` -eq 2
test `grep -c '^XFAIL:' stdout` -eq 1
- $EGREP '^(FAIL|XPASS)' stdout && Exit 1
+ $EGREP '^(FAIL|XPASS|ERROR)' stdout && Exit 1
cd $srcdir
diff --git a/tests/test-driver-custom-no-html.test b/tests/test-driver-custom-no-html.test
index 9aceae6..683659b 100755
--- a/tests/test-driver-custom-no-html.test
+++ b/tests/test-driver-custom-no-html.test
@@ -62,6 +62,6 @@ $FGREP 'dummy title' test-suite.log
# Sanity check: trying to produce HTML output should fail.
$MAKE check-html >output 2>&1 && { cat output; Exit 1; }
cat output
-$EGREP 'SEVERE|ERROR' output
+$FGREP SEVERE output
:
diff --git a/tests/test-driver-end-test-results.test b/tests/test-driver-end-test-results.test
index 97c1f29..03af51b 100755
--- a/tests/test-driver-end-test-results.test
+++ b/tests/test-driver-end-test-results.test
@@ -57,6 +57,7 @@ PASS: b.test
:test-result:PASS
:test-result:END
:test-result:FAIL
+:test-result:ERROR
END
cat > c.test <<END
diff --git a/tests/test-driver-global-log.test b/tests/test-driver-global-log.test
index cdaaff6..35b6fe1 100755
--- a/tests/test-driver-global-log.test
+++ b/tests/test-driver-global-log.test
@@ -44,12 +44,13 @@ END
echo SKIP: skip.test > skip.test
echo FAIL: fail.test > fail.test
echo XPASS: xpass.test > xpass.test
+echo ERROR: error.test > error.test
echo :test-result: PASS > fake-pass.test
echo "$tab $tab$tab" > empty.test
cat > Makefile.am << 'END'
TEST_LOG_DRIVER = ./dummy-driver
-TESTS = pass.test skip.test fail.test xfail.test xpass.test \
+TESTS = pass.test skip.test fail.test xfail.test xpass.test error.test \
fake-pass.test empty.test
END
@@ -82,6 +83,7 @@ cat test-suite.log
grep '^SKIP: skip\.test$' test-suite.log
grep '^FAIL: fail.test$' test-suite.log
grep '^XPASS: xpass.test$' test-suite.log
+grep '^ERROR: error.test$' test-suite.log
grep '^:test-result: PASS$' test-suite.log
grep "^$tab $tab$tab$" test-suite.log
$EGREP 'not seen' test-suite.log && Exit 1
diff --git a/tests/test-driver-recheck.test b/tests/test-driver-recheck.test
index 2ae75b9..f9482b1 100755
--- a/tests/test-driver-recheck.test
+++ b/tests/test-driver-recheck.test
@@ -54,7 +54,7 @@ END
rechecked="$rechecked BAD-$R-1 BAD-$R-2 BAD-$R-3 BAD-$R-4"
done
-for R in FAIL XPASS UNKNOWN; do
+for R in FAIL XPASS ERROR UNKNOWN; do
echo $R: > $R-1
echo $R:foo > $R-2
echo $R: bar baz > $R-3
diff --git a/tests/trivial-test-driver b/tests/trivial-test-driver
index f06eb1c..113e158 100644
--- a/tests/trivial-test-driver
+++ b/tests/trivial-test-driver
@@ -20,9 +20,10 @@
# test results per test script.
#
# The exit status of the wrapped script is ignored. Lines in its stdout
-# and stderr beginning with `PASS', `FAIL', `XFAIL', `XPASS' or `SKIP'
-# count as a test case result with the obviously-corresponding outcome.
-# Every other line is ignored for what concerns the testsuite outcome.
+# and stderr beginning with `PASS', `FAIL', `XFAIL', `XPASS', `SKIP' or
+# `ERROR' count as a test case result with the obviously-corresponding
+# outcome. Every other line is ignored for what concerns the testsuite
+# outcome.
#
# This script is used at least by the `driver-custom-multitest*.test'
# tests.
@@ -61,10 +62,10 @@ tmp_res=$log_file-res.tmp
: > $tmp_res
while read line; do
case $line in
- PASS:*|FAIL:*|XPASS:*|XFAIL:*|SKIP:*)
+ PASS:*|FAIL:*|XPASS:*|XFAIL:*|SKIP:*|ERROR:*)
i=`expr $i + 1`
result=`LC_ALL=C expr "$line" : '\([A-Z]*\):.*'`
- case $result in FAIL|XPASS) st=1;; esac
+ case $result in FAIL|XPASS|ERROR) st=1;; esac
# Output testcase result to console.
echo "$result: $test_name, testcase $i"
# Register testcase outcome for the log file.
--
1.7.2.3
