In order for the Automake testsuite to be able to use sub-second delays to control whether certain files are considered newer than others, five(!) separate pieces of software all need to cooperate: automake itself, autoconf’s internal “autom4te” utility, the Perl interpreter and its libraries, the sleep(1) shell utility, and finally the filesystem hosting the build directory. The existing tests for this are a combination of inadequate and incorrect. This patch, in conjunction with a patch just committed to Autoconf trunk
https://git.savannah.gnu.org/cgit/autoconf.git/commit/?id=39d96e6fff7ceae63b823872602caf4d255a38c8 should make everything much more robust, as follows: - _AM_FILESYSTEM_TIMESTAMP_RESOLUTION is completely rewritten. It no longer looks for autom4te at all, because this macro is invoked unconditionally from AM_INIT, so *every* project that uses Automake would get this test that’s only relevant to Automake’s own testsuite. Also, it tries sleeping for as little as one millisecond (smaller delays consistently get rounded up to 1ms on my computer and I expect that’s universal), it should accurately detect FAT’s two-second resolution now, and it should not be tripped up anymore by running at precisely the moment that will make a 0.1s sleep cross a 1s boundary (this may sound unlikely but it used to cause a couple of test failures *every time* I ran the automake testsuite on a network filesystem that only supported 1s resolution). - In support of the above, the test for working ls -t moved from AM_SANITY_CHECK to _AM_FILESYSTEM_TIMESTAMP_RESOLUTION. This allowed me to simplify the test for $srcdir/configure being older than a freshly created file. - If automake is capable of reading high-resolution file modification timestamps from the operating system, it prints “Features: subsecond-mtime” as the second line of --version output. (We can’t just assume this works for sufficiently new automake, because it depends on whether the Perl interpreter provides this capability, and that’s not a simple question of which version of Perl you have either.) - The Autoconf patch mentioned above adds the same annotation to the output of autom4te --version. - Finally, t/ax/test-defs.in looks for the “Features: subsecond-mtime” string from both automake and autom4te and resets the sleep time to one second if it’s not there. There might be a better place to put this, somewhere it’ll execute every time the *overall testsuite* is invoked rather than once for each test, but I couldn’t find one. Tested on x86-64-linux with development automake and development autoconf. Previous discussion: - https://lists.gnu.org/archive/html/automake/2023-03/msg00000.html - https://lists.gnu.org/archive/html/automake/2023-04/msg00002.html - https://lists.gnu.org/archive/html/automake/2023-12/msg00005.html - https://debbugs.gnu.org/cgi/bugreport.cgi?bug=64756 ChangeLog: * m4/sanity.m4 (_AM_FILESYSTEM_TIMESTAMP_RESOLUTION): Rewrite for greater reliability. Don’t probe autom4te at all here. Check for working “ls -t” here. (AM_SANITY_CHECK): Do not cache the result. Do not check for working “ls -t” check here. Disentangle control flow in the loop probing the relative ages of build and source directory. * lib/Automake/FileUtils.pm: Sync from autoconf. * bin/automake.in (version): Include “Features: subsecond-mtime” in the output if $Automake::FileUtils::subsecond_mtime is true. * configure.ac: Rename the substitution variable MODIFICATION_DELAY to MTIME_RESOLUTION. * t/ax/test-defs.in: Require both $AUTOMAKE and $AUTOM4TE to report support for high-resolution timestamps before setting $sleep to delay for less than one second. --- bin/automake.in | 17 ++-- configure.ac | 9 +- lib/Automake/FileUtils.pm | 71 +++++++++++----- m4/sanity.m4 | 171 ++++++++++++++++++++------------------ t/ax/test-defs.in | 26 ++++-- 5 files changed, 180 insertions(+), 114 deletions(-) diff --git a/bin/automake.in b/bin/automake.in index c0ffc1986..820d0d541 100644 --- a/bin/automake.in +++ b/bin/automake.in @@ -8271,16 +8271,19 @@ General help using GNU software: <https://www.gnu.org/gethelp/>. sub version () { - print <<EOF; -automake (GNU $PACKAGE) $VERSION -Copyright (C) $RELEASE_YEAR Free Software Foundation, Inc. -License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl-2.0.html> + print "automake (GNU $PACKAGE) $VERSION\n"; + print "Features: subsecond-mtime\n" + if $Automake::FileUtils::subsecond_mtime; + print "\nCopyright (C) $RELEASE_YEAR Free Software Foundation, Inc."; + print ' +License GPLv2+: GNU GPL version 2 or later + <https://gnu.org/licenses/gpl-2.0.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. -Written by Tom Tromey <tromey\@redhat.com> - and Alexandre Duret-Lutz <adl\@gnu.org>. -EOF +Written by Tom Tromey <tro...@redhat.com> + and Alexandre Duret-Lutz <a...@gnu.org>. +'; # --version always returns 0 per GNU standards. exit 0; } diff --git a/configure.ac b/configure.ac index 5cda80a18..33aec7788 100644 --- a/configure.ac +++ b/configure.ac @@ -180,9 +180,6 @@ result=no test "x$am_cv_prog_ln" = xln && result=yes AC_MSG_RESULT([$result]) -MODIFICATION_DELAY=$am_cv_filesystem_timestamp_resolution -AC_SUBST([MODIFICATION_DELAY]) - ## ------------------------------------------- ## ## Test for things needed by the test suite. ## ## ------------------------------------------- ## @@ -190,6 +187,12 @@ AC_SUBST([MODIFICATION_DELAY]) AC_PROG_EGREP AC_PROG_FGREP +# The test suite needs to know the resolution of timestamps +# supported by the filesystem hosting the build. The value +# will be acceptable to 'sleep' on this system. +MTIME_RESOLUTION=$am_cv_filesystem_timestamp_resolution +AC_SUBST([MTIME_RESOLUTION]) + dnl FIXME: could we extract this in a simpler way through autoconf dnl FIXME: idioms or internals? AC_DEFUN( diff --git a/lib/Automake/FileUtils.pm b/lib/Automake/FileUtils.pm index 8d0b36802..553d5c6dc 100644 --- a/lib/Automake/FileUtils.pm +++ b/lib/Automake/FileUtils.pm @@ -38,24 +38,45 @@ use 5.006; use strict; use warnings FATAL => 'all'; -use Exporter; +BEGIN +{ + require Exporter; + our @ISA = qw (Exporter); + our @EXPORT = qw (&contents + &find_file &mtime + &update_file + &xsystem &xsystem_hint &xqx + &dir_has_case_matching_file &reset_dir_cache + &set_dir_cache_file); +} + +# Use sub-second resolution file timestamps if available, carry on +# with one-second resolution timestamps if Time::HiRes is not available. +# +# Unfortunately, even if Time::HiRes is available, we don't get +# timestamps to the full precision recorded by the operating system, +# because Time::HiRes converts timestamps to floating-point, and the +# rounding error is hundreds of nanoseconds for circa-2023 timestamps +# in IEEE double precision. But this is the best we can do without +# dropping down to C. +# +# $subsecond_mtime is not exported, but is intended for external +# consumption, as $Automake::FileUtils::subsecond_mtime. +BEGIN +{ + our $subsecond_mtime = 0; + eval + { + require Time::HiRes; + import Time::HiRes qw(stat); + $subsecond_mtime = 1; + } +} + use IO::File; - -# use sub-second resolution timestamps if available, -# carry on with one-second resolution timestamps if that is all we have -BEGIN { eval { require Time::HiRes; import Time::HiRes qw(stat) } } - use Automake::Channels; use Automake::ChannelDefs; -our @ISA = qw (Exporter); -our @EXPORT = qw (&contents - &find_file &mtime - &update_file - &xsystem &xsystem_hint &xqx - &dir_has_case_matching_file &reset_dir_cache - &set_dir_cache_file); - =over 4 =item C<find_file ($file_name, @include)> @@ -122,11 +143,6 @@ sub mtime ($) $atime,$mtime,$ctime,$blksize,$blocks) = stat ($file) or fatal "cannot stat $file: $!"; - # Unfortunately Time::HiRes converts timestamps to floating-point, and the - # rounding error can be hundreds of nanoseconds for circa-2023 timestamps. - # Perhaps some day Perl will support accurate file timestamps. - # For now, do the best we can without going outside Perl. - return $mtime; } @@ -394,3 +410,20 @@ sub set_dir_cache_file ($$) =cut 1; # for require + +### Setup "GNU" style for perl-mode and cperl-mode. +## Local Variables: +## perl-indent-level: 2 +## perl-continued-statement-offset: 2 +## perl-continued-brace-offset: 0 +## perl-brace-offset: 0 +## perl-brace-imaginary-offset: 0 +## perl-label-offset: -2 +## cperl-indent-level: 2 +## cperl-brace-offset: 0 +## cperl-continued-brace-offset: 0 +## cperl-label-offset: -2 +## cperl-extra-newline-before-brace: t +## cperl-merge-trailing-else: nil +## cperl-continued-statement-offset: 2 +## End: diff --git a/m4/sanity.m4 b/m4/sanity.m4 index 8887a0c98..93c4854b9 100644 --- a/m4/sanity.m4 +++ b/m4/sanity.m4 @@ -15,66 +15,87 @@ AS_IF([sleep 0.001 2>/dev/null], [am_cv_sleep_fractional_seconds=true], [am_cv_s # _AM_FILESYSTEM_TIMESTAMP_RESOLUTION # ----------------------------------- -# Determine the filesystem timestamp resolution. Modern systems are -# nanosecond capable, but historical systems could have millisecond, -# second, or even 2-second resolution. +# Determine the filesystem's resolution for file modification +# timestamps. The coarsest we know of is FAT, with a resolution +# of only two seconds, even with the most recent "exFAT" extensions. +# The finest (e.g. ext4 with large inodes, XFS, ZFS) is one +# nanosecond, matching clock_gettime. However, it is probably not +# possible to delay execution of a shell script for less than one +# millisecond, due to process creation overhead and scheduling +# granularity, so we don't check for anything finer than that. AC_DEFUN([_AM_FILESYSTEM_TIMESTAMP_RESOLUTION], [dnl AC_REQUIRE([_AM_SLEEP_FRACTIONAL_SECONDS]) -# -# Check if Autom4te uses Time::HiRes. If not, we cannot use fractional sleep, -# because this sanity test and automated tests will be unreliable due to -# Autom4te's caching of results and comparing timestamps. -# More info: long thread around -# https://lists.gnu.org/archive/html/automake/2023-04/msg00002.html -# and https://debbugs.gnu.org/cgi/bugreport.cgi?bug=64756. -# By the way, we cannot use Perl to see if %INC{q[Time/HiRes.pm]} is -# defined, because Time::HiRes might get pulled in from other system -# modules even when not used directly. (An idea suggested in that thread.) -AC_PATH_PROG([AUTOM4TE], [autom4te]) -if test x"$autom4te_perllibdir" = x; then - autom4te_perllibdir=`sed -n \ - '/autom4te_perllibdir/{s/^.*|| //;s/;$//;s/^.//;s/.$//;p;q}' <$AUTOM4TE` -fi -if grep HiRes "$autom4te_perllibdir"/Autom4te/FileUtils.pm >/dev/null; then - : -else - am_cv_sleep_fractional_seconds=false +AC_CACHE_CHECK([the filesystem timestamp resolution], am_cv_filesystem_timestamp_resolution, [dnl +# Default to the worst case. +am_cv_filesystem_timestamp_resolution=2 + +# Only try to go finer than 1s if sleep can do it. +am_try_resolutions=1 +if $am_cv_sleep_fractional_seconds; then + am_try_resolutions="0.001 0.01 0.1 $am_try_resolutions" fi -AC_CACHE_CHECK([the filesystem timestamp resolution], am_cv_filesystem_timestamp_resolution, [dnl -# Use names that lexically sort older-first when the timestamps are equal. -rm -f conftest.file.a conftest.file.b -: > conftest.file.a -AS_IF([$am_cv_sleep_fractional_seconds], [dnl - am_try_sleep=0.1 am_try_loops=20 -], [dnl - am_try_sleep=1 am_try_loops=2 -]) -am_try=0 -while :; do - AS_VAR_ARITH([am_try], [$am_try + 1]) - echo "timestamp $am_try" > conftest.file.b - set X `ls -t conftest.file.a conftest.file.b` - if test "$[2]" = conftest.file.b || test $am_try -eq $am_try_loops; then +# In order to catch current-generation FAT out, we must *modify* files +# that already exist; the *creation* timestamp is finer. Use names +# that make ls -t sort them differently when they have equal +# timestamps than when they have distinct timestamps, keeping +# in mind that ls -t prints the *newest* file first. +rm -f conftest.ts? +: > conftest.ts1 +: > conftest.ts2 +: > conftest.ts3 + +# Make sure ls -t actually works. Do 'set' in a subshell so we don't +# clobber the current shell's arguments. +if ( + set X `[ls -t conftest.ts[12]]` && + { + test "$[*]" != "X conftest.ts1 conftest.ts2" || + test "$[*]" != "X conftest.ts2 conftest.ts1"; + } +); then :; else + # If neither matched, then we have a broken ls. This can happen + # if, for instance, CONFIG_SHELL is bash and it inherits a + # broken ls alias from the environment. This has actually + # happened. Such a system could not be considered "sane". + _AS_ECHO_UNQUOTED( + ["Bad output from ls -t: \"`[ls -t conftest.ts[12]]`\""], + [AS_MESSAGE_LOG_FD]) + AC_MSG_FAILURE([ls -t produces unexpected output. +Make sure there is not a broken alias in your environment.]) +fi + +for am_try_res in $am_try_resolutions; do + # Any one fine-grained sleep might happen to cross the boundary + # between two values of a coarser actual resolution, but if we do + # two fine-grained sleeps in a row, at least one of them will fall + # entirely within a coarse interval. + echo alpha > conftest.ts1 + sleep $am_try_res + echo beta > conftest.ts2 + sleep $am_try_res + echo gamma > conftest.ts3 + + # We assume that 'ls -t' will make use of high-resolution + # timestamps if the operating system supports them at all. + if (set X `ls -t conftest.ts?` && + test "$[]2" = conftest.ts3 && + test "$[]3" = conftest.ts2 && + test "$[]4" = conftest.ts1); then + am_cv_filesystem_timestamp_resolution=$am_try_res break fi - sleep $am_try_sleep done -rm -f conftest.file.a conftest.file.b -am_cv_filesystem_timestamp_resolution=$am_try -AS_IF([$am_cv_sleep_fractional_seconds], [dnl - AS_VAR_ARITH([am_cv_filesystem_timestamp_resolution], [$am_try / 10]) - AS_VAR_ARITH([am_fraction], [$am_try % 10]) - AS_VAR_APPEND([am_cv_filesystem_timestamp_resolution], [.$am_fraction]) -]) +rm -f conftest.ts? ])]) # AM_SANITY_CHECK # --------------- AC_DEFUN([AM_SANITY_CHECK], [AC_REQUIRE([_AM_FILESYSTEM_TIMESTAMP_RESOLUTION]) -rm -f conftest.file -AC_CACHE_CHECK([whether build environment is sane], am_cv_build_env_is_sane, [dnl +# This check should not be cached, as it may vary across builds of +# different projects. +AC_MSG_CHECKING([whether build environment is sane]) # Reject unsafe characters in $srcdir or the absolute working directory # name. Accept space and tab only in the latter. am_lf=' @@ -93,41 +114,33 @@ esac # symlink; some systems play weird games with the mod time of symlinks # (eg FreeBSD returns the mod time of the symlink's containing # directory). -if ( - am_has_slept=no - for am_try in 1 2; do - echo "timestamp, slept: $am_has_slept" > conftest.file - set X `ls -Lt "$srcdir/configure" conftest.file 2> /dev/null` - if test "$[*]" = "X"; then - # -L didn't work. - set X `ls -t "$srcdir/configure" conftest.file` - fi - if test "$[*]" != "X $srcdir/configure conftest.file" \ - && test "$[*]" != "X conftest.file $srcdir/configure"; then +am_build_env_is_sane=no +am_has_slept=no +rm -f conftest.file +for am_try in 1 2; do + echo "timestamp, slept: $am_has_slept" > conftest.file + if ( + set X `ls -Lt "$srcdir/configure" conftest.file 2> /dev/null` + if test "$[*]" = "X"; then + # -L didn't work. + set X `ls -t "$srcdir/configure" conftest.file` + fi + test "$[2]" = conftest.file + ); then + am_build_env_is_sane=yes + break + fi + # Just in case. + sleep $am_cv_filesystem_timestamp_resolution + am_has_slept=yes +done - # If neither matched, then we have a broken ls. This can happen - # if, for instance, CONFIG_SHELL is bash and it inherits a - # broken ls alias from the environment. This has actually - # happened. Such a system could not be considered "sane". - AC_MSG_ERROR([ls -t appears to fail. Make sure there is not a broken - alias in your environment]) - fi - if test "$[2]" = conftest.file || test $am_try -eq 2; then - break - fi - # Just in case. - sleep $am_cv_filesystem_timestamp_resolution - am_has_slept=yes - done - test "$[2]" = conftest.file - ) -then - am_cv_build_env_is_sane=yes -else - AC_MSG_ERROR([newly created file is older than distributed files! +AC_MSG_RESULT([$am_build_env_is_sane]) +if test $am_build_env_is_sane = no; then + AC_MSG_ERROR([newly created file is older than distributed files! Check your system clock]) fi -]) + # If we didn't sleep, we still need to ensure time stamps of config.status and # generated files are strictly newer. am_sleep_pid= diff --git a/t/ax/test-defs.in b/t/ax/test-defs.in index e09a387cd..696403ee0 100644 --- a/t/ax/test-defs.in +++ b/t/ax/test-defs.in @@ -177,12 +177,26 @@ GNU_GCJFLAGS=${AM_TESTSUITE_GNU_GCJFLAGS-${GNU_GCJFLAGS-'@GNU_GCJFLAGS@'}} # this variable. TEX=${AM_TESTSUITE_TEX-'@TEX@'} -# The amount we should wait after modifying files depends on the platform. -# For instance, Windows '95, '98 and ME have 2-second granularity -# and can be up to 3 seconds in the future w.r.t. the system clock. -# The creative quoting is to avoid spuriously triggering a failure in -# the maintainer checks, -sleep='sleep ''@MODIFICATION_DELAY@' +# The time we should wait after modifying files depends on the platform, and +# also the capabilities of the 'automake' and 'autom4te' programs, which in +# turn depend on the capabilities of the Perl interpreter. configure has +# detected the platform's resolution for file modification times; we still +# need to check automake and autom4te. +MTIME_RESOLUTION='@MTIME_RESOLUTION@' +case $MTIME_RESOLUTION in + 1 | 2) ;; # Can assume these values are usable as is. + *) + if $AUTOMAKE --version 2>&1 | + grep 'Features:.*subsecond-mtime' > /dev/null 2>&1 && + $AUTOM4TE --version 2>&1 | + grep 'Features:.*subsecond-mtime' > /dev/null 2>&1; then + : + else + MTIME_RESOLUTION=1 + fi + ;; +esac +sleep="sleep $MTIME_RESOLUTION" # An old timestamp that can be given to a file, in "touch -t" format. # The time stamp should be portable to all file systems of interest. -- 2.41.0
