bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
tags 8846 + moreinfo thanks Bruno Haible wrote: Jim Meyering wrote: --- a/tests/init.sh +++ b/tests/init.sh @@ -74,7 +74,7 @@ Exit () { set +e; (exit $1); exit $1; } # the reason for skip/failure to console, rather than to the .log files. : ${stderr_fileno_=2} -warn_ () { echo $@ 1$stderr_fileno_; } +warn_ () { eval 'echo $@ 1'$stderr_fileno_; } fail_ () { warn_ $ME_: failed test: $@; Exit 1; } skip_ () { warn_ $ME_: skipped test: $@; Exit 77; } framework_failure_ () { warn_ $ME_: set-up failure: $@; Exit 99; } After applying this patch, here's the test-suite.log that I now get. Hi Bruno, This turned into a long thread. I think all issues raised here have been resolved. Can you confirm?
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
Hi Jim, I think all issues raised here have been resolved. Can you confirm? I think the du/inaccessible-cwd failure was not tackled. But anyway, feel free to close it, and let's test again when you have a pre-release of coreutils-8.13 that you want to get tested. Bruno -- In memoriam Szczepan Ścibior http://pl.wikipedia.org/wiki/Szczepan_Ścibior http://home.clara.net/clinchy/neeb5.htm
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
Bruno Haible wrote: Hi Jim, I think all issues raised here have been resolved. Can you confirm? I think the du/inaccessible-cwd failure was not tackled. But anyway, feel free to close it, and let's test again when you have a pre-release of coreutils-8.13 that you want to get tested. Thanks. I think that failure is inevitable on any system with such poor (save-cwd/restore-cwd--based) openat emulation. I.e., since it has neither the *at functions nor the /proc-based support to emulate them, the test must fail. If only to forestall further bug reports, it'd be good to arrange to skip that test on such systems. From 24a047d93db642b8e73122a6e6553a475f216572 Mon Sep 17 00:00:00 2001 From: Jim Meyering meyer...@redhat.com Date: Sun, 7 Aug 2011 22:09:42 +0200 Subject: [PATCH] tests: avoid lack-of-support du test failure on HP-UX 11.31 * tests/du/inaccessible-cwd: Skip this test on systems like HP-UX 11.31 that lack both the *at functions and the /proc/self/fd-based support we might have used to emulate them. Reported by Bruno Haible in http://debbugs.gnu.org/8846 --- tests/du/inaccessible-cwd |5 + 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/tests/du/inaccessible-cwd b/tests/du/inaccessible-cwd index 220ac7b..3dfe601 100755 --- a/tests/du/inaccessible-cwd +++ b/tests/du/inaccessible-cwd @@ -22,6 +22,11 @@ . ${srcdir=.}/init.sh; path_prepend_ ../src print_ver_ du + +# Skip this test if your system has neither the openat-style functions +# nor /proc/self/fd support with which to emulate them. +require_openat_support_ + skip_if_root_ cwd=`pwd` -- 1.7.4.4
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
Notice: this is a JFTR answer; for the final solution of the issue, see the better working solution at: http://lists.gnu.org/archive/html/bug-coreutils/2011-06/msg00080.html http://lists.gnu.org/archive/html/bug-autoconf/2011-06/msg00016.html On Tuesday 14 June 2011, Eric Blake wrote: On 06/13/2011 04:24 PM, Stefano Lattarini wrote: On Monday 13 June 2011, Eric Blake wrote: Not possible to portably sniff out closed fds; quoting the autoconf manual: Don't rely on duplicating a closed file descriptor to cause an error. With Solaris @command{/bin/sh}, when the redirection fails, the output goes to the original file descriptor. Do the shells with the close-on-exec issue also suffer of the issue with closed fds you've reported? If not, the following could be enough to solve our situation without having to change automake: Where are you proposing this hack? In init.sh (after fd 9 has already been closed if the calling shell used exec instead of some other form of redirection)? That is, why are we trying to fix init.sh to deal with a closed fd after the fact, when it seems like the better approach is to guarantee that fd is never closed on exec in the first place? Which necessarily implies altering the caller. if (exec 3-; exec 43) /dev/null 21; then # Cannot determine whether a file descriptor is closed, fall back # to inferior hack. if test 2 -ne $stderr_fileno_ test ! -t $stderr_fileno_; then eval exec $stderr_fileno_2 # Or is `stderr_fileno_=2' enough? fi Or is the whole point of this hack to init.sh to avoid the spurious warning about EBADF and to instead redirect errors to the (possibly-altered-by-make) fd 2 when fd 9 was lost, so that at least the warning messages appear in the logs even though we lost the chance to display them on the tty? Yes, this was the idea. But it is moot now that we have a better solution to be used in TESTS_ENVRIRONMENT. At any rate, in answer to your question: Solaris /bin/sh (the shell where failed redirections are silently ignored) does this: $ /bin/sh -c '(exec 3-; exec 43) /dev/null 21; echo $?' 0 $ cat k #!/bin/sh e=9; warn_ () { echo $@ 1$e; }; warn_ x $ /bin/sh k x $ /bin/sh k /dev/null $ /bin/sh k 92 x $ /bin/sh k 92 /dev/null x That is, if fd 9 is closed, then 1$e (aka 9) is a no-op on that shell, and output goes to the original fd 1; but if fd 9 is open, then output goes to fd 9. Whereas ksh and HP-UX sh are both vocal on failed redirections, for reliable detection: $ ksh -c '(exec 3-; exec 43) /dev/null 21; echo $?' 1 $ ksh k k[2]: 9: cannot open [Bad file number] $ ksh k /dev/null k[2]: 9: cannot open [Bad file number] $ ksh k 92 x $ ksh k 92 /dev/null x This would have been a problem :-/ Luckily, no more need to worry about it now :-) Regards, Stefano
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
[adding automake-patches on CC:] Reference to original thread(s), mostly duplicated: http://lists.gnu.org/archive/html/bug-coreutils/2011-06/msg00051.html http://lists.gnu.org/archive/html/bug-autoconf/2011-06/msg2.html Reference to last relevant message there: http://lists.gnu.org/archive/html/bug-autoconf/2011-06/msg00018.html http://lists.gnu.org/archive/html/bug-coreutils/2011-06/msg00082.html On Tuesday 14 June 2011, Eric Blake wrote: On 06/13/2011 04:29 PM, Stefano Lattarini wrote: If this work, then using a bare `2' *at the end of TESTS_ENVIRONMENT* and You meant a bare `92', Yes, sorry. but yes that does seem to be workable for what we want! *without a following semicolon* might give a portable workaround, as if I'm not mistaken POSIX mandates that redirections can be specified anywere on the command line, and are to be evaluated from left to right. Yes, all shells support these as equivalent: 92 sh k sh k 92 UPDATE: Yes, it seems to work. I'll add a testcase to the 'maint' branch in case you and Jim decide to go with this solution (and you can confirm that it really works). Cool! Definitely worth documenting in the automake manual, as owner of TESTS_ENVIRONMENT and as a client of init.sh functionality, as well as your proposed automake testcase addition to ensure we don't break it. I'll then push the attached patch to automake master soonish (by tomorrow or so) if there is no objection by then. Regards, Stefano From 9f80020a24110e599b7dbdc5699dc8dbf5af0bd2 Mon Sep 17 00:00:00 2001 Message-Id: 9f80020a24110e599b7dbdc5699dc8dbf5af0bd2.1308037344.git.stefano.lattar...@gmail.com From: Stefano Lattarini stefano.lattar...@gmail.com Date: Tue, 14 Jun 2011 09:41:14 +0200 Subject: [PATCH] tests: check portable fd redirection in TESTS_ENVIRONMENT * tests/tests-environment-fd-redirect.test: New test. * tests/Makefile.am (TESTS): Update. Motivated by coreutils bug#8846: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8846 See also following CC:ed thread on bug-autoconf list: http://lists.gnu.org/archive/html/bug-autoconf/2011-06/msg2.html --- ChangeLog| 10 tests/Makefile.am|1 + tests/Makefile.in|1 + tests/tests-environment-fd-redirect.test | 75 ++ 4 files changed, 87 insertions(+), 0 deletions(-) create mode 100755 tests/tests-environment-fd-redirect.test diff --git a/ChangeLog b/ChangeLog index aad41c7..e482cbd 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,13 @@ +2011-06-14 Stefano Lattarini stefano.lattar...@gmail.com + + tests: check portable fd redirection in TESTS_ENVIRONMENT + * tests/tests-environment-fd-redirect.test: New test. + * tests/Makefile.am (TESTS): Update. + Motivated by coreutils bug#8846: + http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8846 + See also following CC:ed thread on bug-autoconf list: + http://lists.gnu.org/archive/html/bug-autoconf/2011-06/msg2.html + 2011-06-08 Stefano Lattarini stefano.lattar...@gmail.com test defs: new function 'fatal_', for hard errors diff --git a/tests/Makefile.am b/tests/Makefile.am index e68f6d7..c0f39ce 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -949,6 +949,7 @@ tar2.test \ tar3.test \ target-cflags.test \ targetclash.test \ +tests-environment-fd-redirect.test \ txinfo.test \ txinfo2.test \ txinfo3.test \ diff --git a/tests/Makefile.in b/tests/Makefile.in index 7e5fd09..4c223fc 100644 --- a/tests/Makefile.in +++ b/tests/Makefile.in @@ -1216,6 +1216,7 @@ tar2.test \ tar3.test \ target-cflags.test \ targetclash.test \ +tests-environment-fd-redirect.test \ txinfo.test \ txinfo2.test \ txinfo3.test \ diff --git a/tests/tests-environment-fd-redirect.test b/tests/tests-environment-fd-redirect.test new file mode 100755 index 000..2a0afa9 --- /dev/null +++ b/tests/tests-environment-fd-redirect.test @@ -0,0 +1,75 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see http://www.gnu.org/licenses/. + +# Test for a behaviour of `TESTS_ENVIRONMENT' and `AM_TESTS_ENVIRONMENT' +# w.r.t. file descriptor redirections which, although undocumented, +# is nonetheless required by Gnulib's 'tests/init.sh' and by coreutils' +# testsuite. +# The checked behaviour is that we can portably do file descriptor +# redirections by
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
Here's what I expect to do for coreutils, along with an advice-update for gnulib's init.sh: From e948173c1c461aac9f1c490061b257f55e42608d Mon Sep 17 00:00:00 2001 From: Jim Meyering meyer...@redhat.com Date: Tue, 14 Jun 2011 09:59:14 +0200 Subject: [PATCH] tests: accommodate HP-UX and Solaris shells Running make check normally prints a diagnostic to the outermost stderr (usually a tty) to explain why a test is skipped. It did this by redirecting FD 9 to stderr (via exec 92) before invoking the shell script. Shell scripts write skip-explanation to FD 9 via init.sh's skip_ function. However, with Solaris 10's ksh and HP-UX, the effects of exec 92 are canceled upon fork-and-exec, so we would get a Bad file number diagnostic and no skip explanation on those systems. * tests/check.mk (TESTS_ENVIRONMENT): Redirect more portably, via $(SHELL) $$1 92, rather than the prior exec 92; $(SHELL) ... Actually, we use shell_or_perl_ 92, to make this effective also for the perl-based tests. * tests/init.sh (stderr_fileno_): Update the advice in comments. See http://thread.gmane.org/gmane.comp.gnu.coreutils.bugs/22488 for lots of discussion. Stefano Lattarini suggested the solution of putting 92 after the command. Reported by Bruno Haible. --- tests/check.mk |3 +-- tests/init.sh |4 ++-- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/tests/check.mk b/tests/check.mk index db7f067..9db96af 100644 --- a/tests/check.mk +++ b/tests/check.mk @@ -58,7 +58,6 @@ TESTS_ENVIRONMENT = \ test -d $$tmp__ test -w $$tmp__ || tmp__=.; \ . $(srcdir)/envvar-check;\ TMPDIR=$$tmp__; export TMPDIR; \ - exec 92; \ shell_or_perl_() { \ if grep '^\#!/usr/bin/perl' $$1 /dev/null; then \ if $(PERL) -e 'use warnings' /dev/null 21; then \ @@ -100,6 +99,6 @@ TESTS_ENVIRONMENT = \ REPLACE_GETCWD=$(REPLACE_GETCWD) \ ; test -d /usr/xpg4/bin PATH='/usr/xpg4/bin$(PATH_SEPARATOR)'$$PATH; \ PATH='$(abs_top_builddir)/src$(PATH_SEPARATOR)'$$PATH \ - ; shell_or_perl_ + ; shell_or_perl_ 92 VERBOSE = yes diff --git a/tests/init.sh b/tests/init.sh index 60d1bc1..a769d8b 100644 --- a/tests/init.sh +++ b/tests/init.sh @@ -68,8 +68,8 @@ Exit () { set +e; (exit $1); exit $1; } # Print warnings (e.g., about skipped and failed tests) to this file number. # Override by defining to say, 9, in init.cfg, and putting say, -# export ...ENVVAR_SETTINGS...; exec 92; $(SHELL) in the definition -# of TESTS_ENVIRONMENT in your tests/Makefile.am file. +# export ...ENVVAR_SETTINGS...; $(SHELL) $$1 92 +# in the definition of TESTS_ENVIRONMENT in your tests/Makefile.am file. # This is useful when using automake's parallel tests mode, to print # the reason for skip/failure to console, rather than to the .log files. : ${stderr_fileno_=2} -- 1.7.6.rc0.293.g40857
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
Jim Meyering wrote: Bruno Haible wrote: On HP-UX 11.31, built with cc, coreutils-8.12 gives 3 test suite failures: FAIL: misc/printf-surprise (exit: 1) FAIL: dd/nocache (exit: 1) FAIL: du/inaccessible-cwd (exit: 1) Find attached the log file. ... FAIL: misc/printf-surprise (exit: 1) ./init.sh: line 77: 1: Bad file number printf (GNU coreutils) 8.12 Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ... Hi Bruno Thank you for the testing and report. That bad file number error comes from this code in init.sh: : ${stderr_fileno_=2} warn_ () { echo $@ 1$stderr_fileno_; } Because of that, the log contains less information than usual. If you can easily apply this patch and rerun the failing tests, I'd appreciate it. It should avoid the error and let us see more details about what is failing. From 25e7bded3f2abff58540b0fcead2ba110d344bb0 Mon Sep 17 00:00:00 2001 From: Jim Meyering meyer...@redhat.com Date: Mon, 13 Jun 2011 12:07:14 +0200 Subject: [PATCH] init.sh: accommodate shells for which 1$stderr_fileno_ fails * tests/init.sh (warn_): Use eval to work around a bug in some shells, like those of Solaris 10 and HP-UX 11.11. That change did not help, so I've reverted it. That's the first patch. The second one syncs from gnulib's init.sh. From 83e424e7319e8ef2e3398aefbaade972770a6f97 Mon Sep 17 00:00:00 2001 From: Jim Meyering meyer...@redhat.com Date: Mon, 13 Jun 2011 18:20:14 +0200 Subject: [PATCH 1/2] maint: revert previous commit Revert init.sh: accommodate shells for which 1$stderr_fileno_ fails This reverts commit 6fb9aeedd1b858a61d5cbf7f15782adf29ff733a. That change did not solve the problem. For details, see http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8846#74 --- tests/init.sh |2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/tests/init.sh b/tests/init.sh index 25850af..14f2e26 100644 --- a/tests/init.sh +++ b/tests/init.sh @@ -74,7 +74,7 @@ Exit () { set +e; (exit $1); exit $1; } # the reason for skip/failure to console, rather than to the .log files. : ${stderr_fileno_=2} -warn_ () { eval 'echo $@ 1'$stderr_fileno_; } +warn_ () { echo $@ 1$stderr_fileno_; } fail_ () { warn_ $ME_: failed test: $@; Exit 1; } skip_ () { warn_ $ME_: skipped test: $@; Exit 77; } framework_failure_ () { warn_ $ME_: set-up failure: $@; Exit 99; } -- 1.7.6.rc0.293.g40857 From ef6e57b24b51f247119a83be6c81c6a8a30b87a4 Mon Sep 17 00:00:00 2001 From: Jim Meyering meyer...@redhat.com Date: Mon, 13 Jun 2011 12:49:10 +0200 Subject: [PATCH 2/2] init.sh: sync from gnulib * tests/init.sh: Sync recent changes from gnulib. --- tests/init.sh |8 ++-- 1 files changed, 6 insertions(+), 2 deletions(-) diff --git a/tests/init.sh b/tests/init.sh index 14f2e26..60d1bc1 100644 --- a/tests/init.sh +++ b/tests/init.sh @@ -77,6 +77,7 @@ Exit () { set +e; (exit $1); exit $1; } warn_ () { echo $@ 1$stderr_fileno_; } fail_ () { warn_ $ME_: failed test: $@; Exit 1; } skip_ () { warn_ $ME_: skipped test: $@; Exit 77; } +fatal_ () { warn_ $ME_: hard error: $@; Exit 99; } framework_failure_ () { warn_ $ME_: set-up failure: $@; Exit 99; } # Sanitize this shell to POSIX mode, if possible. @@ -167,7 +168,10 @@ else st_=$? # $re_shell_ works just fine. Use it. -test $st_ = 10 break +if test $st_ = 10; then + gl_set_x_corrupts_stderr_=false + break +fi # If this is our first marginally acceptable shell, remember it. if test $st_:$marginal_ = 9: ; then @@ -400,7 +404,7 @@ mktempd_ () { case $# in 2);; - *) fail_ Usage: $ME DIR TEMPLATE;; + *) fail_ Usage: mktempd_ DIR TEMPLATE;; esac destdir_=$1 -- 1.7.6.rc0.293.g40857
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
On Tuesday 14 June 2011, Jim Meyering wrote: Here's what I expect to do for coreutils, along with an advice-update for gnulib's init.sh: Thanks, I'll apply something similar to automake `tests/defs' soonish. I have a doubt below, tough (see near the end). From e948173c1c461aac9f1c490061b257f55e42608d Mon Sep 17 00:00:00 2001 From: Jim Meyering meyer...@redhat.com Date: Tue, 14 Jun 2011 09:59:14 +0200 Subject: [PATCH] tests: accommodate HP-UX and Solaris shells Running make check normally prints a diagnostic to the outermost stderr (usually a tty) to explain why a test is skipped. It did this by redirecting FD 9 to stderr (via exec 92) before invoking the shell script. Shell scripts write skip-explanation to FD 9 via init.sh's skip_ function. However, with Solaris 10's ksh and HP-UX, the effects of exec 92 are canceled upon fork-and-exec, BTW, this is true also for the default ksh on Debian GNU/Linux (but note that this is just a trivia fact, since on GNU/Linux basically everybody uses either bash or dash as the non-interactive shell of choice). so we would get a Bad file number diagnostic and no skip explanation on those systems. * tests/check.mk (TESTS_ENVIRONMENT): Redirect more portably, via $(SHELL) $$1 92, rather than the prior exec 92; $(SHELL) ... Actually, we use shell_or_perl_ 92, to make this effective also for the perl-based tests. * tests/init.sh (stderr_fileno_): Update the advice in comments. See http://thread.gmane.org/gmane.comp.gnu.coreutils.bugs/22488 for lots of discussion. Stefano Lattarini suggested the solution of putting 92 after the command. Reported by Bruno Haible. --- tests/check.mk |3 +-- tests/init.sh |4 ++-- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/tests/check.mk b/tests/check.mk index db7f067..9db96af 100644 --- a/tests/check.mk +++ b/tests/check.mk @@ -58,7 +58,6 @@ TESTS_ENVIRONMENT = \ test -d $$tmp__ test -w $$tmp__ || tmp__=.; \ . $(srcdir)/envvar-check; \ TMPDIR=$$tmp__; export TMPDIR; \ - exec 92; \ shell_or_perl_() { \ if grep '^\#!/usr/bin/perl' $$1 /dev/null; then \ if $(PERL) -e 'use warnings' /dev/null 21; then\ @@ -100,6 +99,6 @@ TESTS_ENVIRONMENT =\ REPLACE_GETCWD=$(REPLACE_GETCWD) \ ; test -d /usr/xpg4/bin PATH='/usr/xpg4/bin$(PATH_SEPARATOR)'$$PATH; \ PATH='$(abs_top_builddir)/src$(PATH_SEPARATOR)'$$PATH \ - ; shell_or_perl_ + ; shell_or_perl_ 92 VERBOSE = yes diff --git a/tests/init.sh b/tests/init.sh index 60d1bc1..a769d8b 100644 --- a/tests/init.sh +++ b/tests/init.sh @@ -68,8 +68,8 @@ Exit () { set +e; (exit $1); exit $1; } # Print warnings (e.g., about skipped and failed tests) to this file number. # Override by defining to say, 9, in init.cfg, and putting say, -# export ...ENVVAR_SETTINGS...; exec 92; $(SHELL) in the definition -# of TESTS_ENVIRONMENT in your tests/Makefile.am file. +# export ...ENVVAR_SETTINGS...; $(SHELL) $$1 92 What is this `$$1' here for? +# in the definition of TESTS_ENVIRONMENT in your tests/Makefile.am file. # This is useful when using automake's parallel tests mode, to print # the reason for skip/failure to console, rather than to the .log files. : ${stderr_fileno_=2} -- 1.7.6.rc0.293.g40857 Regards, Stefano
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
Stefano Lattarini wrote: On Tuesday 14 June 2011, Jim Meyering wrote: Here's what I expect to do for coreutils, along with an advice-update for gnulib's init.sh: Thanks, I'll apply something similar to automake `tests/defs' soonish. I have a doubt below, tough (see near the end). From e948173c1c461aac9f1c490061b257f55e42608d Mon Sep 17 00:00:00 2001 From: Jim Meyering meyer...@redhat.com Date: Tue, 14 Jun 2011 09:59:14 +0200 Subject: [PATCH] tests: accommodate HP-UX and Solaris shells Running make check normally prints a diagnostic to the outermost stderr (usually a tty) to explain why a test is skipped. It did this by redirecting FD 9 to stderr (via exec 92) before invoking the shell script. Shell scripts write skip-explanation to FD 9 via init.sh's skip_ function. However, with Solaris 10's ksh and HP-UX, the effects of exec 92 are canceled upon fork-and-exec, BTW, this is true also for the default ksh on Debian GNU/Linux (but note that this is just a trivia fact, since on GNU/Linux basically everybody uses either bash or dash as the non-interactive shell of choice). Thanks. so we would get a Bad file number diagnostic and no skip explanation on those systems. * tests/check.mk (TESTS_ENVIRONMENT): Redirect more portably, via $(SHELL) $$1 92, rather than the prior exec 92; $(SHELL) ... Actually, we use shell_or_perl_ 92, to make this effective also for the perl-based tests. * tests/init.sh (stderr_fileno_): Update the advice in comments. See http://thread.gmane.org/gmane.comp.gnu.coreutils.bugs/22488 for lots of discussion. Stefano Lattarini suggested the solution of putting 92 after the command. Reported by Bruno Haible. --- tests/check.mk |3 +-- tests/init.sh |4 ++-- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/tests/check.mk b/tests/check.mk index db7f067..9db96af 100644 --- a/tests/check.mk +++ b/tests/check.mk @@ -58,7 +58,6 @@ TESTS_ENVIRONMENT =\ test -d $$tmp__ test -w $$tmp__ || tmp__=.;\ . $(srcdir)/envvar-check; \ TMPDIR=$$tmp__; export TMPDIR;\ - exec 92;\ shell_or_perl_() {\ if grep '^\#!/usr/bin/perl' $$1 /dev/null; then \ if $(PERL) -e 'use warnings' /dev/null 21; then \ @@ -100,6 +99,6 @@ TESTS_ENVIRONMENT = \ REPLACE_GETCWD=$(REPLACE_GETCWD) \ ; test -d /usr/xpg4/bin PATH='/usr/xpg4/bin$(PATH_SEPARATOR)'$$PATH; \ PATH='$(abs_top_builddir)/src$(PATH_SEPARATOR)'$$PATH \ - ; shell_or_perl_ + ; shell_or_perl_ 92 VERBOSE = yes diff --git a/tests/init.sh b/tests/init.sh index 60d1bc1..a769d8b 100644 --- a/tests/init.sh +++ b/tests/init.sh @@ -68,8 +68,8 @@ Exit () { set +e; (exit $1); exit $1; } # Print warnings (e.g., about skipped and failed tests) to this file number. # Override by defining to say, 9, in init.cfg, and putting say, -# export ...ENVVAR_SETTINGS...; exec 92; $(SHELL) in the definition -# of TESTS_ENVIRONMENT in your tests/Makefile.am file. +# export ...ENVVAR_SETTINGS...; $(SHELL) $$1 92 What is this `$$1' here for? Good catch. It doesn't belong. I fixed the log, too. From 67811abfda04dafdde114a50556f0e1d2e9b6270 Mon Sep 17 00:00:00 2001 From: Jim Meyering meyer...@redhat.com Date: Tue, 14 Jun 2011 09:59:14 +0200 Subject: [PATCH] tests: accommodate HP-UX and Solaris shells Running make check normally prints a diagnostic to the outermost stderr (usually a tty) to explain why a test is skipped. It did this by redirecting FD 9 to stderr (via exec 92) before invoking the shell script. Shell scripts write skip-explanation to FD 9 via init.sh's skip_ function. However, with Solaris 10's ksh and HP-UX, the effects of exec 92 are canceled upon fork-and-exec, so we would get a Bad file number diagnostic and no skip explanation on those systems. * tests/check.mk (TESTS_ENVIRONMENT): Redirect more portably, via $(SHELL) 92, rather than the prior exec 92; $(SHELL) ... Actually, we use shell_or_perl_ 92, to make this effective also for the perl-based tests. * tests/init.sh (stderr_fileno_): Update the advice in comments. See http://thread.gmane.org/gmane.comp.gnu.coreutils.bugs/22488 for lots of discussion. Stefano Lattarini suggested the solution of putting 92 after the command. Reported by Bruno Haible. --- tests/check.mk |3 +-- tests/init.sh |4 ++-- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/tests/check.mk b/tests/check.mk index db7f067..9db96af 100644 --- a/tests/check.mk +++ b/tests/check.mk @@ -58,7 +58,6 @@ TESTS_ENVIRONMENT = \ test -d $$tmp__ test -w $$tmp__ || tmp__=.; \ . $(srcdir)/envvar-check;\ TMPDIR=$$tmp__; export TMPDIR; \ - exec 92; \ shell_or_perl_() {
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
On 13/06/11 10:15, Bruno Haible wrote: On HP-UX 11.31, built with cc, coreutils-8.12 gives 3 test suite failures: FAIL: misc/printf-surprise (exit: 1) FAIL: dd/nocache (exit: 1) FAIL: du/inaccessible-cwd (exit: 1) Find attached the log file. Re dd/nocache, I'm suspecting an incorrect match against the error message returned. It should skip the test for 'Operation not supported' Could you give the error returned for: dd if=/dev/zero of=ifile conv=fdatasync count=100 dd if=ifile iflag=nocache count=0 cheers, Pádraig.
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
Pádraig Brady wrote: Could you give the error returned for: dd if=/dev/zero of=ifile conv=fdatasync count=100 dd if=ifile iflag=nocache count=0 Here are the results: $ ./dd if=/dev/zero of=ifile conv=fdatasync count=100 100+0 records in 100+0 records out 51200 bytes (51 kB) copied, 0.14338 s, 357 kB/s $ ./dd if=ifile iflag=nocache count=0 ./dd: failed to discard cache for: `ifile': Not a typewriter 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.000375367 s, 0.0 kB/s Bruno -- In memoriam Anna Göldi http://en.wikipedia.org/wiki/Anna_Göldi
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
Bruno Haible wrote: On HP-UX 11.31, built with cc, coreutils-8.12 gives 3 test suite failures: FAIL: misc/printf-surprise (exit: 1) FAIL: dd/nocache (exit: 1) FAIL: du/inaccessible-cwd (exit: 1) Find attached the log file. ... FAIL: misc/printf-surprise (exit: 1) ./init.sh: line 77: 1: Bad file number printf (GNU coreutils) 8.12 Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ... Hi Bruno Thank you for the testing and report. That bad file number error comes from this code in init.sh: : ${stderr_fileno_=2} warn_ () { echo $@ 1$stderr_fileno_; } Because of that, the log contains less information than usual. If you can easily apply this patch and rerun the failing tests, I'd appreciate it. It should avoid the error and let us see more details about what is failing. From 25e7bded3f2abff58540b0fcead2ba110d344bb0 Mon Sep 17 00:00:00 2001 From: Jim Meyering meyer...@redhat.com Date: Mon, 13 Jun 2011 12:07:14 +0200 Subject: [PATCH] init.sh: accommodate shells for which 1$stderr_fileno_ fails * tests/init.sh (warn_): Use eval to work around a bug in some shells, like those of Solaris 10 and HP-UX 11.11. --- tests/init.sh |2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/tests/init.sh b/tests/init.sh index 14f2e26..7a701f6 100644 --- a/tests/init.sh +++ b/tests/init.sh @@ -74,7 +74,7 @@ Exit () { set +e; (exit $1); exit $1; } # the reason for skip/failure to console, rather than to the .log files. : ${stderr_fileno_=2} -warn_ () { echo $@ 1$stderr_fileno_; } +warn_ () { eval 'echo '$@' 1'$stderr_fileno_; } fail_ () { warn_ $ME_: failed test: $@; Exit 1; } skip_ () { warn_ $ME_: skipped test: $@; Exit 77; } framework_failure_ () { warn_ $ME_: set-up failure: $@; Exit 99; } -- 1.7.6.rc0.293.g40857
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
Hello Jim. Just a minor nit, since your fix will have to be ported to automake's `tests/defs' too ... On Monday 13 June 2011, Jim Meyering wrote: Bruno Haible wrote: On HP-UX 11.31, built with cc, coreutils-8.12 gives 3 test suite failures: FAIL: misc/printf-surprise (exit: 1) FAIL: dd/nocache (exit: 1) FAIL: du/inaccessible-cwd (exit: 1) Find attached the log file. ... FAIL: misc/printf-surprise (exit: 1) ./init.sh: line 77: 1: Bad file number printf (GNU coreutils) 8.12 Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ... Hi Bruno Thank you for the testing and report. That bad file number error comes from this code in init.sh: : ${stderr_fileno_=2} warn_ () { echo $@ 1$stderr_fileno_; } Because of that, the log contains less information than usual. If you can easily apply this patch and rerun the failing tests, I'd appreciate it. It should avoid the error and let us see more details about what is failing. From 25e7bded3f2abff58540b0fcead2ba110d344bb0 Mon Sep 17 00:00:00 2001 From: Jim Meyering meyer...@redhat.com Date: Mon, 13 Jun 2011 12:07:14 +0200 Subject: [PATCH] init.sh: accommodate shells for which 1$stderr_fileno_ fails * tests/init.sh (warn_): Use eval to work around a bug in some shells, like those of Solaris 10 and HP-UX 11.11. --- tests/init.sh |2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/tests/init.sh b/tests/init.sh index 14f2e26..7a701f6 100644 --- a/tests/init.sh +++ b/tests/init.sh @@ -74,7 +74,7 @@ Exit () { set +e; (exit $1); exit $1; } # the reason for skip/failure to console, rather than to the .log files. : ${stderr_fileno_=2} -warn_ () { echo $@ 1$stderr_fileno_; } +warn_ () { eval 'echo '$@' 1'$stderr_fileno_; } Wouldn't it be simpler, and potentially more correct, to use simply: warn_ () { eval 'echo $@ 1'$stderr_fileno_; } With your new implementation, I see this on Debian with bash and dash: $ stderr_fileno_=2; warn_ () { eval 'echo '$@' 1'$stderr_fileno_; } $ $ warn_ '$HOME' /home/stefano which is not intended behaviour IMHO. fail_ () { warn_ $ME_: failed test: $@; Exit 1; } skip_ () { warn_ $ME_: skipped test: $@; Exit 77; } framework_failure_ () { warn_ $ME_: set-up failure: $@; Exit 99; } -- 1.7.6.rc0.293.g40857 Regards, Stefano
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
Stefano Lattarini wrote: Hello Jim. Just a minor nit, since your fix will have to be ported to automake's `tests/defs' too ... ... diff --git a/tests/init.sh b/tests/init.sh index 14f2e26..7a701f6 100644 --- a/tests/init.sh +++ b/tests/init.sh @@ -74,7 +74,7 @@ Exit () { set +e; (exit $1); exit $1; } # the reason for skip/failure to console, rather than to the .log files. : ${stderr_fileno_=2} -warn_ () { echo $@ 1$stderr_fileno_; } +warn_ () { eval 'echo '$@' 1'$stderr_fileno_; } Wouldn't it be simpler, and potentially more correct, to use simply: warn_ () { eval 'echo $@ 1'$stderr_fileno_; } With your new implementation, I see this on Debian with bash and dash: $ stderr_fileno_=2; warn_ () { eval 'echo '$@' 1'$stderr_fileno_; } $ $ warn_ '$HOME' /home/stefano which is not intended behaviour IMHO. Hah! Thank you for the prompt fix. That's more than a minor nit ;-) I'm glad you caught my error.
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
Stefano Lattarini wrote: Hello Jim. ... Wouldn't it be simpler, and potentially more correct, to use simply: warn_ () { eval 'echo $@ 1'$stderr_fileno_; } With your new implementation, I see this on Debian with bash and dash: $ stderr_fileno_=2; warn_ () { eval 'echo '$@' 1'$stderr_fileno_; } $ $ warn_ '$HOME' /home/stefano which is not intended behaviour IMHO. Thanks again. Here's what I've just pushed. I'll do the same for gnulib shortly. From 6fb9aeedd1b858a61d5cbf7f15782adf29ff733a Mon Sep 17 00:00:00 2001 From: Jim Meyering meyer...@redhat.com Date: Mon, 13 Jun 2011 12:07:14 +0200 Subject: [PATCH] init.sh: accommodate shells for which 1$stderr_fileno_ fails * tests/init.sh (warn_): Use eval to work around a bug in some shells, like those of Solaris 10 and HP-UX 11.11. Improved by Stefano Lattarini. --- tests/init.sh |2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/tests/init.sh b/tests/init.sh index 14f2e26..25850af 100644 --- a/tests/init.sh +++ b/tests/init.sh @@ -74,7 +74,7 @@ Exit () { set +e; (exit $1); exit $1; } # the reason for skip/failure to console, rather than to the .log files. : ${stderr_fileno_=2} -warn_ () { echo $@ 1$stderr_fileno_; } +warn_ () { eval 'echo $@ 1'$stderr_fileno_; } fail_ () { warn_ $ME_: failed test: $@; Exit 1; } skip_ () { warn_ $ME_: skipped test: $@; Exit 77; } framework_failure_ () { warn_ $ME_: set-up failure: $@; Exit 99; } -- 1.7.6.rc0.293.g40857
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
On 13/06/11 10:55, Bruno Haible wrote: Pádraig Brady wrote: Could you give the error returned for: dd if=/dev/zero of=ifile conv=fdatasync count=100 dd if=ifile iflag=nocache count=0 Here are the results: $ ./dd if=/dev/zero of=ifile conv=fdatasync count=100 100+0 records in 100+0 records out 51200 bytes (51 kB) copied, 0.14338 s, 357 kB/s $ ./dd if=ifile iflag=nocache count=0 ./dd: failed to discard cache for: `ifile': Not a typewriter Weird. I've relaxed the test with the following. cheers, Pádraig. --- a/tests/dd/nocache +++ b/tests/dd/nocache @@ -36,12 +36,12 @@ done # Advise to drop cache for whole file if ! dd if=ifile iflag=nocache count=0 2err; then - if grep -F 'Operation not supported' err /dev/null; then - warn_ 'skipping part; this file system lacks support for posix_fadvise()' - skip=1 - else -fail=1 - fi + # We could check for 'Operation not supported' in err here, + # but that was seen to be brittle. HPUX returns ENOTTY for example. + # So assume that if this basic operation fails, it's due to lack + # of support by the system. + warn_ 'skipping part; this file system lacks support for posix_fadvise()' + skip=1 fi if test $skip != 1; then
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
On Monday 13 June 2011, Jim Meyering wrote: Stefano Lattarini wrote: Hello Jim. ... Wouldn't it be simpler, and potentially more correct, to use simply: warn_ () { eval 'echo $@ 1'$stderr_fileno_; } With your new implementation, I see this on Debian with bash and dash: $ stderr_fileno_=2; warn_ () { eval 'echo '$@' 1'$stderr_fileno_; } $ $ warn_ '$HOME' /home/stefano which is not intended behaviour IMHO. Thanks again. Here's what I've just pushed. I'll do the same for gnulib shortly. From 6fb9aeedd1b858a61d5cbf7f15782adf29ff733a Mon Sep 17 00:00:00 2001 From: Jim Meyering meyer...@redhat.com Date: Mon, 13 Jun 2011 12:07:14 +0200 Subject: [PATCH] init.sh: accommodate shells for which 1$stderr_fileno_ fails * tests/init.sh (warn_): Use eval to work around a bug in some shells, like those of Solaris 10 and HP-UX 11.11. Improved by Stefano Lattarini. --- tests/init.sh |2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/tests/init.sh b/tests/init.sh index 14f2e26..25850af 100644 --- a/tests/init.sh +++ b/tests/init.sh @@ -74,7 +74,7 @@ Exit () { set +e; (exit $1); exit $1; } # the reason for skip/failure to console, rather than to the .log files. : ${stderr_fileno_=2} -warn_ () { echo $@ 1$stderr_fileno_; } +warn_ () { eval 'echo $@ 1'$stderr_fileno_; } fail_ () { warn_ $ME_: failed test: $@; Exit 1; } skip_ () { warn_ $ME_: skipped test: $@; Exit 77; } framework_failure_ () { warn_ $ME_: set-up failure: $@; Exit 99; } -- 1.7.6.rc0.293.g40857 Thanks. I'll port this to automake tests/defs soonish; I assume you are ok to be credited as lead author in ChangeLog and git commit, right? Regards, Stefano
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
Bruno Haible wrote: Jim Meyering wrote: --- a/tests/init.sh +++ b/tests/init.sh @@ -74,7 +74,7 @@ Exit () { set +e; (exit $1); exit $1; } # the reason for skip/failure to console, rather than to the .log files. : ${stderr_fileno_=2} -warn_ () { echo $@ 1$stderr_fileno_; } +warn_ () { eval 'echo $@ 1'$stderr_fileno_; } fail_ () { warn_ $ME_: failed test: $@; Exit 1; } skip_ () { warn_ $ME_: skipped test: $@; Exit 77; } framework_failure_ () { warn_ $ME_: set-up failure: $@; Exit 99; } After applying this patch, here's the test-suite.log that I now get. Thank you. However, it looks like using eval didn't help at all. What does this do on that system? sh -c 'e=2; warn_ () { echo $@ 1$e; }; warn_ x' This is what happens via tests/check.mk: printf '#!/bin/sh\ne=9; warn_ () { echo $@ 1$e; }; warn_ x\n' k sh -c 'exec 92; /bin/sh k' Does that evoke a warning for you? If so, does hard-coding the 9 help? printf '#!/bin/sh\ne=9; warn_ () { echo $@ 19; }; warn_ x\n' k sh -c 'exec 92; /bin/sh k'
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
On 06/13/2011 04:11 AM, Jim Meyering wrote: Thank you for the testing and report. That bad file number error comes from this code in init.sh: : ${stderr_fileno_=2} warn_ () { echo $@ 1$stderr_fileno_; } Because of that, the log contains less information than usual. I'm having problems summarizing this shell bug into a succinct description that would be worth including in the autoconf manual. What exactly is the bug here? That is, my attempt to provoke the problem on HP-UX 11.11 with this snippet is not failing with a bad file number: $ /bin/sh -c 'n=3; f(){ echo $@ 1$n; }; exec 32; f hi' hi -- Eric Blake ebl...@redhat.com+1-801-349-2682 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
Eric Blake wrote: On 06/13/2011 04:11 AM, Jim Meyering wrote: Thank you for the testing and report. That bad file number error comes from this code in init.sh: : ${stderr_fileno_=2} warn_ () { echo $@ 1$stderr_fileno_; } Because of that, the log contains less information than usual. I'm having problems summarizing this shell bug into a succinct description that would be worth including in the autoconf manual. What exactly is the bug here? That is, my attempt to provoke the problem on HP-UX 11.11 with this snippet is not failing with a bad file number: $ /bin/sh -c 'n=3; f(){ echo $@ 1$n; }; exec 32; f hi' hi Per Bruno's report, running make check on HP-UX 11.31 resulted in one Bad file number diagnostic per use of init.sh. Perhaps it's a problem in 11.31 that is not in 11.1.
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
Eric Blake wrote: On 06/13/2011 07:36 AM, Jim Meyering wrote: However, it looks like using eval didn't help at all. What does this do on that system? sh -c 'e=2; warn_ () { echo $@ 1$e; }; warn_ x' $ sh -c 'e=2; warn_ () { echo $@ 1$e; }; warn_ x' x This is what happens via tests/check.mk: printf '#!/bin/sh\ne=9; warn_ () { echo $@ 1$e; }; warn_ x\n' k sh -c 'exec 92; /bin/sh k' Does that evoke a warning for you? Yes: % sh -c 'exec 92; /bin/sh k' k: 9: Generated or received a file descriptor number that is not valid. But I think the reason that it invokes a problem is not because of the 1$e construct, but because HP-UX opens secondary file descriptors as cloexec or otherwise closing them at some point, so the '/bin/sh k' child process is not inheriting fd 9 from the parent process. Ah ha! That sounds like a syscall that needs to be gnulib-replaced in bash/zsh. Of course, first we'd need to gnulib-enable a shell. Does anyone know the HP-UX counterpart to Linux' strace in order to see what syscalls are in use by the HP-UX shell, and why/where fd 9 is getting closed? http://bhami.com/rosetta.html suggests the following: caliper fprof caliper ktrace trace (freeware) tusc (11+, freeware) If so, does hard-coding the 9 help? No. Thanks for investigating.
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
Stefano Lattarini wrote: On Monday 13 June 2011, Jim Meyering wrote: Stefano Lattarini wrote: Hello Jim. ... Wouldn't it be simpler, and potentially more correct, to use simply: warn_ () { eval 'echo $@ 1'$stderr_fileno_; } With your new implementation, I see this on Debian with bash and dash: $ stderr_fileno_=2; warn_ () { eval 'echo '$@' 1'$stderr_fileno_; } $ $ warn_ '$HOME' /home/stefano which is not intended behaviour IMHO. Thanks again. Here's what I've just pushed. I'll do the same for gnulib shortly. From 6fb9aeedd1b858a61d5cbf7f15782adf29ff733a Mon Sep 17 00:00:00 2001 From: Jim Meyering meyer...@redhat.com Date: Mon, 13 Jun 2011 12:07:14 +0200 Subject: [PATCH] init.sh: accommodate shells for which 1$stderr_fileno_ fails * tests/init.sh (warn_): Use eval to work around a bug in some shells, like those of Solaris 10 and HP-UX 11.11. Improved by Stefano Lattarini. --- tests/init.sh |2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/tests/init.sh b/tests/init.sh index 14f2e26..25850af 100644 --- a/tests/init.sh +++ b/tests/init.sh @@ -74,7 +74,7 @@ Exit () { set +e; (exit $1); exit $1; } # the reason for skip/failure to console, rather than to the .log files. : ${stderr_fileno_=2} -warn_ () { echo $@ 1$stderr_fileno_; } +warn_ () { eval 'echo $@ 1'$stderr_fileno_; } fail_ () { warn_ $ME_: failed test: $@; Exit 1; } skip_ () { warn_ $ME_: skipped test: $@; Exit 77; } framework_failure_ () { warn_ $ME_: set-up failure: $@; Exit 99; } -- 1.7.6.rc0.293.g40857 Thanks. I'll port this to automake tests/defs soonish; I assume you are ok to be credited as lead author in ChangeLog and git commit, right? Looks like that change is not useful after all. I'll revert it in coreutils.
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
On 06/13/2011 09:37 AM, Jim Meyering wrote: But I think the reason that it invokes a problem is not because of the 1$e construct, but because HP-UX opens secondary file descriptors as cloexec or otherwise closing them at some point, so the '/bin/sh k' child process is not inheriting fd 9 from the parent process. Ah ha! That sounds like a syscall that needs to be gnulib-replaced in bash/zsh. Of course, first we'd need to gnulib-enable a shell. Yep, I've confirmed that both HP-UX and Solaris 10 ksh use CLOEXEC fds for intermediate fd values created by exec; that POSIX permits (but fortunately does not require) this behavior, and that autoconf already documents the pitfall of expecting fds 2 to be preserved into child processes: http://www.gnu.org/software/autoconf/manual/autoconf.html#File-Descriptors Don't rely on open file descriptors being open in child processes. In ksh, file descriptors above 2 which are opened using ‘exec nfile’ are closed by a subsequent ‘exec’ (such as that involved in the fork-and-exec which runs a program or script). Thus, using sh, we have: $ cat ./descrips #!/bin/sh - echo hello 5 $ exec 5t $ ./descrips $ cat t hello $ But using ksh: $ exec 5t $ ./descrips hello $ cat t $ On Solaris 10, truss shows this for 'exec 92': dup(2) = 3 close(9)Err#9 EBADF fcntl(3, F_DUPFD, 0x0009) = 9 close(3)= 0 fcntl(9, F_SETFD, 0x0001) = 0 tusc (11+, freeware) Likewise on HP-UX 11.31, tusc shows: dup(2) ... = 3 close(9) . ERR#9 EBADF fcntl(3, F_DUPFD, 9) . = 9 close(3) . = 0 fcntl(9, F_SETFD, 1) . = 0 About the only thing I can think of is to pass the details about the desired redirection as a command-line or env-var setting, to be done by the script itself, rather than relying on the parent shell (the one in make) being able to do a redirection that persists into the child (the one sourcing init.sh). -- Eric Blake ebl...@redhat.com+1-801-349-2682 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
Bruno Haible wrote: Jim Meyering wrote: --- a/tests/init.sh +++ b/tests/init.sh @@ -317,6 +317,11 @@ path_prepend_ () setup_ () { + # If we're redirecting a file descriptor larger than 2, say via automake's + # TESTS_ENVIRONMENT, that redirected FD is closed-on-exec on some systems + # (at least Solaris 10 and HP-UX 11.x), so redirect it here again. + test $stderr_fileno_ = 2 || eval exec $stderr_fileno_2 + if test $VERBOSE = yes; then # Test whether set -x may cause the selected shell to corrupt an # application's stderr. Many do, including zsh-4.3.10 and the /bin/sh With this change, I get the attached log on HP-UX 11.31. == GNU coreutils 8.12: tests/test-suite.log == 3 of 365 tests failed. (88 tests were not run). ... Hi Bruno, Thanks for persevering. That confirms the init.sh change solves one problem. However, HP's /bin/sh has the bug that set -x corrupts stderr, and in that case, init.sh decides not to honor VERBOSE=yes because that led inevitably to numerous false positives due to the bug. The easiest way to work around that is to use some other shell. If you have bash, you might want to run this (but see below): sh=/usr/local/bin/bash make CONFIG_SHELL=$sh CONFIG_SHELL=$sh VERBOSE=yes \ check -C tests TESTS='misc/printf-surprise du/inaccessible-cwd' FAIL: dd/nocache (exit: 1) ... Pádraig's patch should have resolved this. FAIL: du/inaccessible-cwd (exit: 1) This failure may be expected on HP-UX, assuming it lacks the *at functions and its getcwd doesn't have the feature that Linux and others do of working even in an inaccessible directory. That leaves this one: FAIL: misc/printf-surprise (exit: 1) Which may be failing because the ulimit of 10MiB is too low: head -c 10 fifo out ( ulimit -v 1; env $prog %2000f 0 2err-msg fifo ) exit=$? Can you try it manually? E.g., run these commands and report the output? mkfifo fifo head -c 10 fifo out ( ulimit -v 1; env printf %2000f 0 2err-msg fifo ) exit=$? n_out=$(wc -c out) echo $n_out::$exit cat err-msg
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
Hi Jim. Probably you're totally going to hate me today, but ... On Monday 13 June 2011, Jim Meyering wrote: From d987cf87de5e7e597e295914c536bd332c24cc63 Mon Sep 17 00:00:00 2001 From: Jim Meyering meyer...@redhat.com Date: Mon, 13 Jun 2011 18:54:53 +0200 Subject: [PATCH] init.sh: redirect FD 9 to stderr again, for Solaris 10 and HP-UX * tests/init.sh (setup_): When $stderr_fileno_ is not 2, redirect it. Prior to this change, we would redirect before the shell fork-and-exec performed via automake's TESTS_ENVIRONMENT, but that redirection was ineffective on Solaris 10 and HP-UX 11.31, due to the fact that those systems set the CLOEXEC bit on FDs larger than 2. Thus our redirection of FD 9 would not survive the fork-and-exec of running each test script. --- tests/init.sh |5 + 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/tests/init.sh b/tests/init.sh index 60d1bc1..d101643 100644 --- a/tests/init.sh +++ b/tests/init.sh @@ -317,6 +317,11 @@ path_prepend_ () setup_ () { + # If we're redirecting a file descriptor larger than 2, say via automake's + # TESTS_ENVIRONMENT, that redirected FD is closed-on-exec on some systems + # (at least Solaris 10 and HP-UX 11.x), so redirect it here again. + test $stderr_fileno_ = 2 || eval exec $stderr_fileno_2 + ... isn't this equivalent to just using 2 unconditionally in 'warn_()'? IMHO, the right fix is to to modify the code in TESTS_ENVIRONMENT to avoid the definition of $stderr_fileno_ the shell performs closed-on-exec; e.g., TESTS_ENVIRONMENT = ...; \ if test x`(exec 91 sh -c 'echo foo 9' /dev/null 21)` = x'foo'; then stderr_fileno_=9; export stderr_fileno_; else unset stderr_fileno_ || : fi If we know that bash and zsh are well behaved, we can even avoid a couple of forks (Cygwin users won't hate us too much then): TESTS_ENVIRONMENT = ...; \ if test -n $${ZSH_VERSION}$${BASH_VERSION} || \ test x`(exec 91 sh -c 'echo foo 9' /dev/null 21)` = x'foo' then stderr_fileno_=9; export stderr_fileno_; else unset stderr_fileno_ || : fi A better fix would be to do the redirect $stderr_fileno_2 in tess/init.sh iff $stderr_fileno_ is closed, but how can that be portably determined without printing trash on the user screen (and for *each* test)? if test $VERBOSE = yes; then # Test whether set -x may cause the selected shell to corrupt an # application's stderr. Many do, including zsh-4.3.10 and the /bin/sh -- 1.7.6.rc0.293.g40857 Regards, Stefano
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
On 06/13/2011 02:32 PM, Stefano Lattarini wrote: +++ b/tests/init.sh @@ -317,6 +317,11 @@ path_prepend_ () setup_ () { + # If we're redirecting a file descriptor larger than 2, say via automake's + # TESTS_ENVIRONMENT, that redirected FD is closed-on-exec on some systems + # (at least Solaris 10 and HP-UX 11.x), so redirect it here again. + test $stderr_fileno_ = 2 || eval exec $stderr_fileno_2 + ... isn't this equivalent to just using 2 unconditionally in 'warn_()'? Not quite, in coreutils' current usage. Coreutils sets stderr_fileno_=9 in init.cfg, not TESTS_ENVIRONMENT; which means that stderr can be changed after a call to setup_ but before a call to warn_. Thus, we want to output to $stderr_fileno_ (which is 9, and is always the original stderr), even during a test where stderr has been redirected somewhere else. IMHO, the right fix is to to modify the code in TESTS_ENVIRONMENT to avoid the definition of $stderr_fileno_ the shell performs closed-on-exec; e.g., TESTS_ENVIRONMENT = ...; \ if test x`(exec 91 sh -c 'echo foo 9' /dev/null 21)` = x'foo'; then stderr_fileno_=9; export stderr_fileno_; else unset stderr_fileno_ || : fi But setting things during TESTS_ENVIRONMENT does seem a bit more robust. Or rather, I think that the best action is to set stderr_fileno_ alongside the code that does the redirection; if it is TESTS_ENVIRONMENT that does the exec 92, then TESTS_ENVIRONMENT does seem like the saner place (rather than init.cfg) for setting stderr_fileno_. If we know that bash and zsh are well behaved, we can even avoid a couple of forks (Cygwin users won't hate us too much then): TESTS_ENVIRONMENT = ...; \ if test -n $${ZSH_VERSION}$${BASH_VERSION} || \ test x`(exec 91 sh -c 'echo foo 9' /dev/null 21)` = x'foo' then stderr_fileno_=9; export stderr_fileno_; else unset stderr_fileno_ || : fi A better fix would be to do the redirect $stderr_fileno_2 in tess/init.sh iff $stderr_fileno_ is closed, but how can that be portably determined without printing trash on the user screen (and for *each* test)? Not possible to portably sniff out closed fds; quoting the autoconf manual: Don't rely on duplicating a closed file descriptor to cause an error. With Solaris @command{/bin/sh}, when the redirection fails, the output goes to the original file descriptor. -- Eric Blake ebl...@redhat.com+1-801-349-2682 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
On Monday 13 June 2011, Stefano Lattarini wrote: Hi Jim. Probably you're totally going to hate me today, but ... On Monday 13 June 2011, Jim Meyering wrote: From d987cf87de5e7e597e295914c536bd332c24cc63 Mon Sep 17 00:00:00 2001 From: Jim Meyering meyer...@redhat.com Date: Mon, 13 Jun 2011 18:54:53 +0200 Subject: [PATCH] init.sh: redirect FD 9 to stderr again, for Solaris 10 and HP-UX * tests/init.sh (setup_): When $stderr_fileno_ is not 2, redirect it. Prior to this change, we would redirect before the shell fork-and-exec performed via automake's TESTS_ENVIRONMENT, but that redirection was ineffective on Solaris 10 and HP-UX 11.31, due to the fact that those systems set the CLOEXEC bit on FDs larger than 2. Thus our redirection of FD 9 would not survive the fork-and-exec of running each test script. --- tests/init.sh |5 + 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/tests/init.sh b/tests/init.sh index 60d1bc1..d101643 100644 --- a/tests/init.sh +++ b/tests/init.sh @@ -317,6 +317,11 @@ path_prepend_ () setup_ () { + # If we're redirecting a file descriptor larger than 2, say via automake's + # TESTS_ENVIRONMENT, that redirected FD is closed-on-exec on some systems + # (at least Solaris 10 and HP-UX 11.x), so redirect it here again. + test $stderr_fileno_ = 2 || eval exec $stderr_fileno_2 + ... isn't this equivalent to just using 2 unconditionally in 'warn_()'? IMHO, the right fix is to to modify the code in TESTS_ENVIRONMENT to avoid the definition of $stderr_fileno_ the shell performs closed-on-exec; e.g., TESTS_ENVIRONMENT = ...; \ if test x`(exec 91 sh -c 'echo foo 9' /dev/null 21)` = x'foo'; then stderr_fileno_=9; export stderr_fileno_; else unset stderr_fileno_ || : fi If we know that bash and zsh are well behaved, we can even avoid a couple of forks (Cygwin users won't hate us too much then): TESTS_ENVIRONMENT = ...; \ if test -n $${ZSH_VERSION}$${BASH_VERSION} || \ test x`(exec 91 sh -c 'echo foo 9' /dev/null 21)` = x'foo' then stderr_fileno_=9; export stderr_fileno_; else unset stderr_fileno_ || : fi A better fix would be to do the redirect $stderr_fileno_2 in tess/init.sh iff $stderr_fileno_ is closed, but how can that be portably determined without printing trash on the user screen (and for *each* test)? But this last observaton makes me think. The only purpose of $stderr_fileno_ is to allow the test to print diagnostic on the user's tty, instead of burying it in the test logs; at this point, we might do the redirection only if the fd 2 is a tty, so that we will know that, in `tests/init.sh', either: [1] $stderr_fileno_ refers to a tty, even after the automake parallel-tests driver has made its own redirections; or: [2] $stderr_fileno_ is simply the file descriptor 2, which is expected to be open for writing in any remotely sane setup. Then we can test, from within tests/init.sh, whether $stderr_fileno_ has been closed or not by doing: test 2 -eq $stderr_fileno_ || test -t $stderr_fileno_ and if this is not the case, we eval exec $stderr_fileno_ 2 and live happily. All without extra forks or overly complex `TESTS_ENVIRONMENT' definitions. WDYT? Regards, Stefano
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
On Monday 13 June 2011, Stefano Lattarini wrote: On Monday 13 June 2011, Stefano Lattarini wrote: Hi Jim. Probably you're totally going to hate me today, but ... On Monday 13 June 2011, Jim Meyering wrote: From d987cf87de5e7e597e295914c536bd332c24cc63 Mon Sep 17 00:00:00 2001 From: Jim Meyering meyer...@redhat.com Date: Mon, 13 Jun 2011 18:54:53 +0200 Subject: [PATCH] init.sh: redirect FD 9 to stderr again, for Solaris 10 and HP-UX * tests/init.sh (setup_): When $stderr_fileno_ is not 2, redirect it. Prior to this change, we would redirect before the shell fork-and-exec performed via automake's TESTS_ENVIRONMENT, but that redirection was ineffective on Solaris 10 and HP-UX 11.31, due to the fact that those systems set the CLOEXEC bit on FDs larger than 2. Thus our redirection of FD 9 would not survive the fork-and-exec of running each test script. --- tests/init.sh |5 + 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/tests/init.sh b/tests/init.sh index 60d1bc1..d101643 100644 --- a/tests/init.sh +++ b/tests/init.sh @@ -317,6 +317,11 @@ path_prepend_ () setup_ () { + # If we're redirecting a file descriptor larger than 2, say via automake's + # TESTS_ENVIRONMENT, that redirected FD is closed-on-exec on some systems + # (at least Solaris 10 and HP-UX 11.x), so redirect it here again. + test $stderr_fileno_ = 2 || eval exec $stderr_fileno_2 + ... isn't this equivalent to just using 2 unconditionally in 'warn_()'? IMHO, the right fix is to to modify the code in TESTS_ENVIRONMENT to avoid the definition of $stderr_fileno_ the shell performs closed-on-exec; e.g., TESTS_ENVIRONMENT = ...; \ if test x`(exec 91 sh -c 'echo foo 9' /dev/null 21)` = x'foo'; then stderr_fileno_=9; export stderr_fileno_; else unset stderr_fileno_ || : fi If we know that bash and zsh are well behaved, we can even avoid a couple of forks (Cygwin users won't hate us too much then): TESTS_ENVIRONMENT = ...; \ if test -n $${ZSH_VERSION}$${BASH_VERSION} || \ test x`(exec 91 sh -c 'echo foo 9' /dev/null 21)` = x'foo' then stderr_fileno_=9; export stderr_fileno_; else unset stderr_fileno_ || : fi A better fix would be to do the redirect $stderr_fileno_2 in tests/init.sh iff $stderr_fileno_ is closed, but how can that be portably determined without printing trash on the user screen (and for *each* test)? But this last observaton makes me think. The only purpose of $stderr_fileno_ is to allow the test to print diagnostic on the user's tty, instead of burying it in the test logs; at this point, we might do the redirection only if the fd 2 is a tty, so that we will know that, in `tests/init.sh', either: [1] $stderr_fileno_ refers to a tty, even after the automake parallel-tests driver has made its own redirections; or: [2] $stderr_fileno_ is simply the file descriptor 2, which is expected to be open for writing in any remotely sane setup. Then we can test, from within tests/init.sh, whether $stderr_fileno_ has been closed or not by doing: test 2 -eq $stderr_fileno_ || test -t $stderr_fileno_ and if this is not the case, we eval exec $stderr_fileno_ 2 and live happily. All without extra forks or overly complex `TESTS_ENVIRONMENT' definitions. Only that BSD make and Solaris dmake, when running in concurrent mode, redirect stdout/stderr to temporary files or to named or anonymous pipes (for output serialization); so that my proposed idiom wouldn't work with them. Sigh. Regards, Stefano
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
On 06/13/2011 03:39 PM, Stefano Lattarini wrote: [3] The $(SHELL), which here we assume is ksh with close-on-exec, does a fork+exec to execute the test script; thus, when that script begins its execution, it has the fd 9 closed (d'oh!, but expected now); Oddly enough, it looks like it is _only_ 'exec 92' that suffers from close-on-exec semantics; using '(blah) 92' leaves fd 9 inheritable. So maybe the real fix is to do the redirection to a subshell or grouping {} rather than via exec, at which point even ksh and HP-UX sh will allow the original stderr to be inherited as fd 9 to the child script process: $ printf '#!/bin/sh\nexec 9\n' k $ sh k k[2]: 9: Generated or received a file descriptor number that is not valid. $ sh k 92 $ sh -c 'exec 92; /bin/sh k' k[2]: 9: Generated or received a file descriptor number that is not valid. $ sh -c 'sh k 92' $ sh -c '(sh k) 92' $ sh -c '{ sh k;} 92' $ but I don't know if that will require some changes in automake's parallel-test driver setup code. And Autoconf should probably document that the close-on-exec semantics of some shells applies only to the exec builtin, since they do not happen to other redirections. -- Eric Blake ebl...@redhat.com+1-801-349-2682 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
On Monday 13 June 2011, Eric Blake wrote: Not possible to portably sniff out closed fds; quoting the autoconf manual: Don't rely on duplicating a closed file descriptor to cause an error. With Solaris @command{/bin/sh}, when the redirection fails, the output goes to the original file descriptor. Do the shells with the close-on-exec issue also suffer of the issue with closed fds you've reported? If not, the following could be enough to solve our situation without having to change automake: if (exec 3-; exec 43) /dev/null 21; then # Cannot determine whether a file descriptor is closed, fall back # to inferior hack. if test 2 -ne $stderr_fileno_ test ! -t $stderr_fileno_; then eval exec $stderr_fileno_2 # Or is `stderr_fileno_=2' enough? fi else if (exec 39) /dev/null 21; then : else eval exec $stderr_fileno_2 # Or is `stderr_fileno_=2' enough? fi fi Regards, Stefano
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
On Tuesday 14 June 2011, Eric Blake wrote: On 06/13/2011 03:39 PM, Stefano Lattarini wrote: [3] The $(SHELL), which here we assume is ksh with close-on-exec, does a fork+exec to execute the test script; thus, when that script begins its execution, it has the fd 9 closed (d'oh!, but expected now); Oddly enough, it looks like it is _only_ 'exec 92' that suffers from close-on-exec semantics; using '(blah) 92' leaves fd 9 inheritable. So maybe the real fix is to do the redirection to a subshell or grouping {} rather than via exec, at which point even ksh and HP-UX sh will allow the original stderr to be inherited as fd 9 to the child script process: $ printf '#!/bin/sh\nexec 9\n' k $ sh k k[2]: 9: Generated or received a file descriptor number that is not valid. $ sh k 92 $ sh -c 'exec 92; /bin/sh k' k[2]: 9: Generated or received a file descriptor number that is not valid. $ sh -c 'sh k 92' If this work, then using a bare `2' *at the end of TESTS_ENVIRONMENT* and *without a following semicolon* might give a portable workaround, as if I'm not mistaken POSIX mandates that redirections can be specified anywere on the command line, and are to be evaluated from left to right. UPDATE: Yes, it seems to work. I'll add a testcase to the 'maint' branch in case you and Jim decide to go with this solution (and you can confirm that it really works). $ sh -c '(sh k) 92' $ sh -c '{ sh k;} 92' $ but I don't know if that will require some changes in automake's parallel-test driver setup code. And Autoconf should probably document that the close-on-exec semantics of some shells applies only to the exec builtin, since they do not happen to other redirections. Regards, Stefano
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
On 06/13/2011 04:24 PM, Stefano Lattarini wrote: On Monday 13 June 2011, Eric Blake wrote: Not possible to portably sniff out closed fds; quoting the autoconf manual: Don't rely on duplicating a closed file descriptor to cause an error. With Solaris @command{/bin/sh}, when the redirection fails, the output goes to the original file descriptor. Do the shells with the close-on-exec issue also suffer of the issue with closed fds you've reported? If not, the following could be enough to solve our situation without having to change automake: Where are you proposing this hack? In init.sh (after fd 9 has already been closed if the calling shell used exec instead of some other form of redirection)? That is, why are we trying to fix init.sh to deal with a closed fd after the fact, when it seems like the better approach is to guarantee that fd is never closed on exec in the first place? Which necessarily implies altering the caller. if (exec 3-; exec 43) /dev/null 21; then # Cannot determine whether a file descriptor is closed, fall back # to inferior hack. if test 2 -ne $stderr_fileno_ test ! -t $stderr_fileno_; then eval exec $stderr_fileno_2 # Or is `stderr_fileno_=2' enough? fi Or is the whole point of this hack to init.sh to avoid the spurious warning about EBADF and to instead redirect errors to the (possibly-altered-by-make) fd 2 when fd 9 was lost, so that at least the warning messages appear in the logs even though we lost the chance to display them on the tty? At any rate, in answer to your question: Solaris /bin/sh (the shell where failed redirections are silently ignored) does this: $ /bin/sh -c '(exec 3-; exec 43) /dev/null 21; echo $?' 0 $ cat k #!/bin/sh e=9; warn_ () { echo $@ 1$e; }; warn_ x $ /bin/sh k x $ /bin/sh k /dev/null $ /bin/sh k 92 x $ /bin/sh k 92 /dev/null x That is, if fd 9 is closed, then 1$e (aka 9) is a no-op on that shell, and output goes to the original fd 1; but if fd 9 is open, then output goes to fd 9. Whereas ksh and HP-UX sh are both vocal on failed redirections, for reliable detection: $ ksh -c '(exec 3-; exec 43) /dev/null 21; echo $?' 1 $ ksh k k[2]: 9: cannot open [Bad file number] $ ksh k /dev/null k[2]: 9: cannot open [Bad file number] $ ksh k 92 x $ ksh k 92 /dev/null x -- Eric Blake ebl...@redhat.com+1-801-349-2682 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature
bug#8846: coreutils-8.12 on HP-UX 11.31: 3 of 365 tests failed
On 06/13/2011 04:29 PM, Stefano Lattarini wrote: If this work, then using a bare `2' *at the end of TESTS_ENVIRONMENT* and You meant a bare `92', but yes that does seem to be workable for what we want! *without a following semicolon* might give a portable workaround, as if I'm not mistaken POSIX mandates that redirections can be specified anywere on the command line, and are to be evaluated from left to right. Yes, all shells support these as equivalent: 92 sh k sh k 92 UPDATE: Yes, it seems to work. I'll add a testcase to the 'maint' branch in case you and Jim decide to go with this solution (and you can confirm that it really works). Cool! Definitely worth documenting in the automake manual, as owner of TESTS_ENVIRONMENT and as a client of init.sh functionality, as well as your proposed automake testcase addition to ensure we don't break it. -- Eric Blake ebl...@redhat.com+1-801-349-2682 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature