[ dropping bug-coreutils, this mail doesn't address that question ] * Eric Blake wrote on Mon, Jun 14, 2010 at 10:23:51PM CEST: > On 06/14/2010 01:45 PM, Ralf Wildenhues wrote: > > * Eric Blake wrote on Mon, Jun 14, 2010 at 07:22:31PM CEST: > >> Did you make sure that AT_COLOR_TESTS can be specified either side of > >> AT_INIT, or is there a fixed invocation order that authors must be aware > >> of? > > > > Right now it has to be specified before AT_INIT. If there's a diversion > > thinhy we can easily fix that with, great, otherwise I guess it would > > need documentation and order warning in the code. Help? > > It seems like it would merely be a matter of unconditionally emitting > at_color=no in an early diversion, and having AT_COLOR_TESTS emit > at_color=auto in an intermediate diversion, all before the option > parsing that appears in a later diversion. That way, if the user didn't > specify --color, they pick up the appropriate default. > > That also implies that anything else that checks at m4-time whether > AT_color is defined (such as your m4_ifdef for changing the --help > output text) must be m4_wrap'd to delay any output based on that > decision of AT_color until after the user's source file has been > completely read in. Then again, we already m4_wrap some of our --help > text in order to accurately tell the user how many AT_SETUPs we > encountered, so this may already be taken care of given the current > architecture of autotest.
I think I got that working now, incremental patch below and full one attached. It wasn't even necessary to output at_color initialization more than once. One problem I see with this change is that the code documents # - HELP_TUNING # Tuning help text. Additional tuning options can be appended as # self-contained cat'd here-docs as generated by AS_HELP_STRING. so when we now use m4_wrap, tuning help from the user will get added before our own, right? So this is at least a semantic change. FWIW, I haven't found instances of HELP_TUNING in an online code search. Add another diversion HELP_TUNING_BASE and let HELP_TUNING be for the user? Cheers, Ralf diff --git a/lib/autotest/general.m4 b/lib/autotest/general.m4 index c2fc369..23bf145 100644 --- a/lib/autotest/general.m4 +++ b/lib/autotest/general.m4 @@ -389,8 +389,6 @@ at_jobs=1 at_traceon=: at_trace_echo=: at_check_filter_trace=: -# Whether to enable colored test results. -at_color=m4_ifdef([AT_color], [AT_color], [no]) # Shall we keep the debug scripts? Must be `:' when the suite is # run by a debug script, so that the script doesn't remove itself. @@ -425,6 +423,8 @@ m4_define([_AT_FINISH], [m4_ifdef([AT_ingroup], [m4_fatal([missing AT_CLEANUP detected])])dnl m4_divert_text([DEFAULTS], [ +# Whether to enable colored test results. +at_color=m4_ifdef([AT_color], [AT_color], [no]) # List of the tested programs. at_tested='m4_ifdef([AT_tested], [m4_translit(m4_dquote(m4_defn([AT_tested])), [ ], m4_newline)])' @@ -721,7 +721,7 @@ Operation modes: -l, --list describes all the tests, or the selected TESTS _ATEOF m4_divert_pop([HELP_MODES])dnl -m4_divert_push([HELP_TUNING])dnl +m4_wrap([m4_divert_push([HELP_TUNING])dnl cat <<_ATEOF || at_write_fail=1 dnl extra quoting prevents emacs whitespace mode from putting tabs in output @@ -745,7 +745,7 @@ Execution tuning: [ default for debugging scripts] -x, --trace enable tests shell tracing _ATEOF -m4_divert_pop([HELP_TUNING])dnl +m4_divert_pop([HELP_TUNING])])dnl m4_divert_push([HELP_END])dnl cat <<_ATEOF || at_write_fail=1
commit f372e5ba28b1fe30ce4f1f51ef29910efbfa3510 Author: Ralf Wildenhues <[email protected]> Date: Tue Jun 15 21:19:14 2010 +0200 Autotest: enable colored test results. * lib/autotest/general.m4 (AT_INIT): Accept --color and --color=no|never|yes|auto|always. If desired, colorize test results and testsuite summary on standard output. [HELP_TUNING]: Wrap this help diversion until the end, when we know whether AT_COLOR_TESTS has been specified. (AT_COLOR_TESTS): New macro, set the default for color to auto. * doc/autoconf.texi (Writing Testsuites): Document it. (testsuite Invocation): Document --color* options. * tests/local.at: Call AT_COLOR_TESTS for Autoconf's testsuite. * tests/autotest.at (color test results): New test, mirroring color.test from Automake. * NEWS: Update. diff --git a/NEWS b/NEWS index f2290a1..0ff482f 100644 --- a/NEWS +++ b/NEWS @@ -36,6 +36,8 @@ GNU Autoconf NEWS - User visible changes. AT_ARG_OPTION has been changed in that the negative of a long option --OPTION is now --no-OPTION rather than --noOPTION. +** Autotest testsuites may optionally provide colored test results. + * Major changes in Autoconf 2.65 (2009-11-21) [stable] Released by Eric Blake, based on git versions 2.64.*. diff --git a/doc/autoconf.texi b/doc/autoconf.texi index 1acf7d0..1e4ed01 100644 --- a/doc/autoconf.texi +++ b/doc/autoconf.texi @@ -23476,6 +23476,12 @@ Writing Testsuites several times. @end defmac +...@defmac AT_COLOR_TESTS +...@atindex{color_tests} +Enable colored test results by default when the output is connected to +a terminal. +...@end defmac + Autotest test suites rely on @env{PATH} to find the tested program. This avoids the need to generate absolute names of the various tools, and makes it possible to test installed programs. Therefore, knowing which @@ -23858,6 +23864,13 @@ testsuite Invocation Force more verbosity in the detailed output of what is being done. This is the default for debugging scripts. +...@item --color +...@itemx --co...@r{[}=no@r{|}ne...@r{|}...@r{|}a...@r{|}alw...@r{]} +Enable colored test results. Test results are colored if the argument +...@samp{always} is given, or if standard output is connected to a terminal +and either the macro @code{AT_COLOR_TESTS} is used or the arguments +...@samp{no} or @samp{never} are not given. + @item --debug @itemx -d Do not remove the files after a test group was performed ---but they are diff --git a/lib/autotest/general.m4 b/lib/autotest/general.m4 index 0bc529f..23bf145 100644 --- a/lib/autotest/general.m4 +++ b/lib/autotest/general.m4 @@ -423,6 +423,8 @@ m4_define([_AT_FINISH], [m4_ifdef([AT_ingroup], [m4_fatal([missing AT_CLEANUP detected])])dnl m4_divert_text([DEFAULTS], [ +# Whether to enable colored test results. +at_color=m4_ifdef([AT_color], [AT_color], [no]) # List of the tested programs. at_tested='m4_ifdef([AT_tested], [m4_translit(m4_dquote(m4_defn([AT_tested])), [ ], m4_newline)])' @@ -492,6 +494,17 @@ do at_clean=: ;; + --color ) + at_color=auto + ;; + --color=* ) + case $at_optarg in + no | never) at_color=never ;; + yes | auto) at_color=auto ;; + always | *) at_color=always ;; + esac + ;; + --debug | -d ) at_debug_p=: ;; @@ -663,6 +676,17 @@ else # Sort the tests, removing duplicates. at_groups=`AS_ECHO(["$at_groups"]) | tr ' ' "$as_nl" | sort -nu` fi + +if test x"$at_color" = xalways \ + || { test x"$at_color" = xauto && test -t 1; }; then + at_red=`printf '\033@<:@0;31m'` + at_grn=`printf '\033@<:@0;32m'` + at_lgn=`printf '\033@<:@1;32m'` + at_blu=`printf '\033@<:@1;34m'` + at_std=`printf '\033@<:@m'` +else + at_red= at_grn= at_lgn= at_blu= at_std= +fi m4_divert_pop([PARSE_ARGS_END])dnl m4_divert_push([HELP])dnl @@ -697,13 +721,17 @@ Operation modes: -l, --list describes all the tests, or the selected TESTS _ATEOF m4_divert_pop([HELP_MODES])dnl -m4_divert_push([HELP_TUNING])dnl +m4_wrap([m4_divert_push([HELP_TUNING])dnl cat <<_ATEOF || at_write_fail=1 dnl extra quoting prevents emacs whitespace mode from putting tabs in output Execution tuning: -C, --directory=DIR [ change to directory DIR before starting] + --color[[=no|never|yes|auto|always]] +[ ]m4_ifdef([AT_color], + [disable colored test results, or enable even without terminal], + [enable colored test results on terminal, or always]) -j, --jobs[[=N]] [ Allow N jobs at once; infinite jobs with no arg (default 1)] -k, --keywords=KEYWORDS @@ -717,7 +745,7 @@ Execution tuning: [ default for debugging scripts] -x, --trace enable tests shell tracing _ATEOF -m4_divert_pop([HELP_TUNING])dnl +m4_divert_pop([HELP_TUNING])])dnl m4_divert_push([HELP_END])dnl cat <<_ATEOF || at_write_fail=1 @@ -1156,35 +1184,40 @@ _ATEOF at_msg="UNEXPECTED PASS" at_res=xpass at_errexit=$at_errexit_p + at_color=$at_red ;; no:0) at_msg="ok" at_res=pass at_errexit=false + at_color=$at_grn ;; *:77) at_msg='skipped ('`cat "$at_check_line_file"`')' at_res=skip at_errexit=false + at_color=$at_blu ;; no:* | *:99) at_msg='FAILED ('`cat "$at_check_line_file"`')' at_res=fail at_errexit=$at_errexit_p + at_color=$at_red ;; yes:*) at_msg='expected failure ('`cat "$at_check_line_file"`')' at_res=xfail at_errexit=false + at_color=$at_lgn ;; esac echo "$at_res" > "$at_job_dir/$at_res" # In parallel mode, output the summary line only afterwards. if test $at_jobs -ne 1 && test -n "$at_verbose"; then - AS_ECHO(["$at_desc_line $at_msg"]) + AS_ECHO(["$at_desc_line $at_color$at_msg$at_std"]) else # Make sure there is a separator even with long titles. - AS_ECHO([" $at_msg"]) + AS_ECHO([" $at_color$at_msg$at_std"]) fi at_log_msg="$at_group. $at_desc ($at_setup_line): $at_msg" case $at_status in @@ -1485,12 +1518,14 @@ if $at_errexit_p && test $at_unexpected_count != 0; then at_result="$at_result $at_were run, one failed" fi at_result="$at_result unexpectedly and inhibited subsequent tests." + at_color=$at_red else # Don't you just love exponential explosion of the number of cases? + at_color=$at_red case $at_xpass_count:$at_fail_count:$at_xfail_count in # So far, so good. - 0:0:0) at_result="$at_result $at_were successful." ;; - 0:0:*) at_result="$at_result behaved as expected." ;; + 0:0:0) at_result="$at_result $at_were successful." at_color=$at_grn ;; + 0:0:*) at_result="$at_result behaved as expected." at_color=$at_lgn ;; # Some unexpected failures 0:*:0) at_result="$at_result $at_were run, @@ -1538,10 +1573,10 @@ $at_skip_count tests were skipped." ;; esac if test $at_unexpected_count = 0; then - echo "$at_result" + echo "$at_color$at_result$at_std" echo "$at_result" >&AS_MESSAGE_LOG_FD else - echo "ERROR: $at_result" >&2 + echo "${at_color}ERROR: $at_result$at_std" >&2 echo "ERROR: $at_result" >&AS_MESSAGE_LOG_FD { echo @@ -1757,6 +1792,12 @@ m4_define([AT_COPYRIGHT], [m4_default([$2], [m4_newline])([$1])])])# AT_COPYRIGHT +# AT_COLOR_TESTS +# -------------- +# Enable colored test results if standard error is connected to a terminal. +m4_define([AT_COLOR_TESTS], +[m4_define([AT_color], [auto])]) + # AT_SETUP(DESCRIPTION) # --------------------- # Start a group of related tests, all to be executed in the same subshell. diff --git a/tests/autotest.at b/tests/autotest.at index e09e189..fd3787a 100644 --- a/tests/autotest.at +++ b/tests/autotest.at @@ -1505,6 +1505,93 @@ AT_CHECK([test -s sigpipe-stamp || test ! -f micro-suite.dir/7/micro-suite.log], AT_CLEANUP + +# --color +AT_CHECK_AT_TEST([colored test results], + [AT_CHECK([:]) + AT_CLEANUP + AT_SETUP([fail]) + AT_CHECK([exit 1]) + AT_CLEANUP + AT_SETUP([xpass]) + AT_XFAIL_IF([:]) + AT_CHECK([:]) + AT_CLEANUP + AT_SETUP([xfail]) + AT_XFAIL_IF([:]) + AT_CHECK([exit 1]) + AT_CLEANUP + AT_SETUP([skip]) + AT_CHECK([exit 77]) + AT_CLEANUP + AT_SETUP([hardfail]) + AT_XFAIL_IF([:]) + AT_CHECK([exit 99]) +], [], [], [], [], [], [ + +TERM=ansi +export TERM + +red=`printf '\033@<:@0;31m'` +grn=`printf '\033@<:@0;32m'` +lgn=`printf '\033@<:@1;32m'` +blu=`printf '\033@<:@1;34m'` +std=`printf '\033@<:@m'` + +# Check that grep can parse nonprinting characters. +# BSD 'grep' works from a pipe, but not a seekable file. +# GNU or BSD 'grep -a' works on files, but is not portable. +AT_CHECK([case `echo "$std" | grep .` in #'' restore font-lock + $std) :;; + *) Exit 77;; + esac], [], [ignore], [], + [echo "grep can't parse nonprinting characters" >&2]) + +if echo 'ab*c' | grep -F 'ab*c' >/dev/null 2>&1; then + FGREP="grep -F" +else + FGREP=fgrep +fi + +# No color. +AT_CHECK([$CONFIG_SHELL ./micro-suite], [1], [stdout], [stderr]) +for color in "$red" "$grn" "$lgn" "$blu"; do + AT_CHECK([cat stdout stderr | $FGREP "$color"], [1]) +done + +# Color of test group results. +AT_CHECK([$CONFIG_SHELL ./micro-suite --color=always], [1], [stdout], [stderr]) +AT_CHECK([cat stdout | grep " only " | $FGREP "$grn"], [], [ignore]) +AT_CHECK([cat stdout | grep " fail " | $FGREP "$red"], [], [ignore]) +AT_CHECK([cat stdout | grep " xfail " | $FGREP "$lgn"], [], [ignore]) +AT_CHECK([cat stdout | grep " xpass " | $FGREP "$red"], [], [ignore]) +AT_CHECK([cat stdout | grep " skip " | $FGREP "$blu"], [], [ignore]) +AT_CHECK([cat stdout | grep " hardfail " | $FGREP "$red"], [], [ignore]) +AT_CHECK([cat stderr | grep ERROR | $FGREP "$red"], [], [ignore]) + +# The summary is green if all tests were successful, light green if all +# behaved as expected, and red otherwise. +AT_CHECK([$CONFIG_SHELL ./micro-suite --color=always 1 -k skip], + [0], [stdout]) +AT_CHECK([cat stdout | grep 'test.*successful' | $FGREP "$grn"], + [], [ignore]) +AT_CHECK([$CONFIG_SHELL ./micro-suite --color=always 1 -k xfail -k skip], + [0], [stdout]) +AT_CHECK([cat stdout | grep 'as expected' | $FGREP "$lgn"], [], [ignore]) +AT_CHECK([$CONFIG_SHELL ./micro-suite --color=always -k fail], + [1], [ignore], [stderr]) +AT_CHECK([cat stderr | grep ERROR | $FGREP "$red"], [], [ignore]) +AT_CHECK([$CONFIG_SHELL ./micro-suite --color=always -k xpass], + [1], [ignore], [stderr]) +AT_CHECK([cat stderr | grep ERROR | $FGREP "$red"], [], [ignore]) +AT_CHECK([$CONFIG_SHELL ./micro-suite --color=always -k hardfail], + [1], [ignore], [stderr]) +AT_CHECK([cat stderr | grep ERROR | $FGREP "$red"], [], [ignore]) +# Reset color on verbose output. +printf %s\\n "$std" +], [1]) + + ## ------------------- ## ## srcdir propagation. ## ## ------------------- ## diff --git a/tests/local.at b/tests/local.at index a812c43..39360ef 100644 --- a/tests/local.at +++ b/tests/local.at @@ -25,6 +25,8 @@ m4_pattern_allow([^m4_(define|shift)$]) # Programs this package provides AT_TESTED([autom4te autoconf autoheader autoupdate autoreconf ifnames]) +# Enable colored test output. +AT_COLOR_TESTS ## ---------------- ## ## Utility macros. ##
