Re: preparing for a new release

2020-11-15 Thread Jim Meyering 
On Sun, Nov 15, 2020 at 6:52 PM Karl Berry  wrote:
>
> Hi Jim - (and thanks for the test runs and logs, Gavin)
>
>   make -k check TESTS='t/txinfo-no-clutter' \
> AM_TESTSUITE_MAKE="make -j$(( 2*$(nproc) + 1 ))"
>
> I know little about programming for parallel (auto)make, but I'm
> guessing that with the above invocation, the make commands inside the
> test are also being run in parallel?
>
> As far as I can see, this test is simply not set up to support that. It
> essentially runs, for every format being built by texi2dvi:
>   make FMT && make clean
>
> Thus it seems to me that one make clean could be cleaning the .dvi while
> the next make ps is trying to use it.
>
> And/or, maybe dependencies are missing. all-local is depending on
> "ps pdf dvi html", so I suppose they'll get spawned in parallel.  And the
> only dependency I'm seeing is the suffix rule, .dvi.ps, so maybe the
> purportedly intermediate .dvi is getting removed by one of the parallel
> makes?  So I tried adding
>
> .PRECIOUS: %.dvi
>
> to both Makefile.am's, and the fancy make command above then succeeded.
> Does that get around the problem for you?  Doesn't exactly seem like the
> best fix, though, since ordinarily one would want the intermediate file
> to be deleted. Maybe it would be cleaner to just force serialization of
> the makes inside this test? Is that possible?
>
> Further:
>
> 1) I am not smart enough to tell what's actually happening from the log
> since there's no indication of which make is doing what (and I can't
> even imagine how that could be indicated). How does one ever debug these
> things for sure?
>
> 2) I had not seen AM_TESTSUITE_MAKE before. I was running tests simply
> with make -j12 TARGET, and there the test does not fail -- not
> surprisingly, I guess, since then (I think) the makes inside the test
> are not being run in parallel.
>
> 3) I can't see why this is anything new. That test has not changed since
> 2012. The mentioned bug
> (https://debbugs.gnu.org/cgi/bugreport.cgi?bug=11146) describes the
> commit at that time as improving the test wrt parallelism.  Thus I
> surmise no one has ever tried with a parallel AM_TESTSUITE_MAKE before now.
>
> 4) I feel surprised that this parallelism does not cause trouble with
> other tests that involve suffix rules, like yacc or lex. I don't see
> .PRECIOUS being used in any other test, aside from precious.sh. But
> maybe no other test is doing quite what txinfo-no-clutter is.

Thanks for investigating. I've just fixed that with the attached (add
.NOTPARALLEL):


0001-tests-avoid-missing-.dvi-failure-with-parallel-tests.patch
Description: Binary data

Re: preparing for a new release

2020-11-15 Thread Karl Berry 
Hi Jim - (and thanks for the test runs and logs, Gavin)

  make -k check TESTS='t/txinfo-no-clutter' \
AM_TESTSUITE_MAKE="make -j$(( 2*$(nproc) + 1 ))"

I know little about programming for parallel (auto)make, but I'm
guessing that with the above invocation, the make commands inside the
test are also being run in parallel?

As far as I can see, this test is simply not set up to support that. It
essentially runs, for every format being built by texi2dvi:
  make FMT && make clean 

Thus it seems to me that one make clean could be cleaning the .dvi while
the next make ps is trying to use it. 

And/or, maybe dependencies are missing. all-local is depending on
"ps pdf dvi html", so I suppose they'll get spawned in parallel.  And the
only dependency I'm seeing is the suffix rule, .dvi.ps, so maybe the
purportedly intermediate .dvi is getting removed by one of the parallel
makes?  So I tried adding

.PRECIOUS: %.dvi

to both Makefile.am's, and the fancy make command above then succeeded.
Does that get around the problem for you?  Doesn't exactly seem like the
best fix, though, since ordinarily one would want the intermediate file
to be deleted. Maybe it would be cleaner to just force serialization of
the makes inside this test? Is that possible?

Further:

1) I am not smart enough to tell what's actually happening from the log
since there's no indication of which make is doing what (and I can't
even imagine how that could be indicated). How does one ever debug these
things for sure?

2) I had not seen AM_TESTSUITE_MAKE before. I was running tests simply
with make -j12 TARGET, and there the test does not fail -- not
surprisingly, I guess, since then (I think) the makes inside the test
are not being run in parallel.

3) I can't see why this is anything new. That test has not changed since
2012. The mentioned bug
(https://debbugs.gnu.org/cgi/bugreport.cgi?bug=11146) describes the
commit at that time as improving the test wrt parallelism.  Thus I
surmise no one has ever tried with a parallel AM_TESTSUITE_MAKE before now.

4) I feel surprised that this parallelism does not cause trouble with
other tests that involve suffix rules, like yacc or lex. I don't see
.PRECIOUS being used in any other test, aside from precious.sh. But
maybe no other test is doing quite what txinfo-no-clutter is.

Thanks,
Karl

Re: preparing for a new release

2020-11-15 Thread Gavin Smith 
On Sun, Nov 15, 2020 at 09:45:44AM -0800, Jim Meyering wrote:
> I've fixed a few of these failures, but here is still at least one
> that I have not fixed: t/txinfo-no-clutter. In this case, running it
> with no parallelism and by itself passes:
>   make -k check TESTS='t/txinfo-no-clutter'
> Yet running it with parallel make fails (I'm using make-4.3.90 built
> from git, in case it matters):
>   make -k check TESTS='t/txinfo-no-clutter' AM_TESTSUITE_MAKE="make
> -j$(( 2*$(nproc) + 1 ))"
> 
> Not sure I'll work more on this today, so sharing: The attached log
> shows that this fails when building that test's final "all" rule,
> often because a .dvi file could not be found:
> 
> ) localization,mv: cannot stat 'bar.dvi': No such file or directory

My suspicion is that it is interference between the dvi and ps targets.
When I remove the ps targets with the change at the bottom of this email,
the test passes every time.  The ps file is made from the dvi file so
perhaps the ps rules are deleting the dvi file while the dvi rules are
still using it, or vice versa.

diff --git a/t/txinfo-no-clutter.sh b/t/txinfo-no-clutter.sh
index b311f0348..4701462d2 100644
--- a/t/txinfo-no-clutter.sh
+++ b/t/txinfo-no-clutter.sh
@@ -26,7 +26,7 @@ AC_OUTPUT
 END
 
 cat > Makefile.am << 'END'
-all-local: ps pdf dvi html # For "make distcheck".
+all-local: pdf dvi html # For "make distcheck".
 info_TEXINFOS = foo.texi doc/bar.texi baz.texi
 SUBDIRS = sub
 END
@@ -34,7 +34,7 @@ END
 mkdir sub doc
 
 cat > sub/Makefile.am << 'END'
-all-local: ps pdf dvi html # For "make distcheck".
+all-local: pdf dvi html # For "make distcheck".
 info_TEXINFOS = baz.texi
 END
 
@@ -90,7 +90,7 @@ $AUTOCONF
 
 # Try one by one, to ensure later targets don't involuntarily
 # clean up potential cruft left by earlier ones.
-for fmt in info pdf ps dvi html all; do
+for fmt in info pdf dvi html all; do
   $MAKE $fmt
   # For debugging.
   ls -l . doc sub
@@ -103,7 +103,7 @@ for fmt in info pdf ps dvi html all; do
   test -e sub/baz.html
   ;;
 all)
-  for x in info pdf ps dvi; do
+  for x in info pdf dvi; do
 test -f foo.$x
 test -f doc/bar.$x
 test -f baz.$x
@@ -127,10 +127,9 @@ for fmt in info pdf ps dvi html all; do
   case $fmt in
 pdf) extension_rx="(texi|pdf|t2p)";;
 dvi) extension_rx="(texi|dvi|t2d)";;
- ps) extension_rx="(texi|ps|dvi|t2d)";;
info) extension_rx="(texi|info)";;
html) extension_rx="(texi|html)";;
-all) extension_rx="(texi|html|info|pdf|ps|dvi|t2[pd])";;
+all) extension_rx="(texi|html|info|pdf|dvi|t2[pd])";;
   *) fatal_ "unreachable code reached";;
   esac
   $EGREP -v "^$basename_rx\.$extension_rx$" lst && exit 1

Re: preparing for a new release

2020-11-15 Thread Gavin Smith 
On Sun, Nov 15, 2020 at 09:45:44AM -0800, Jim Meyering wrote:
> Karl has made quite a few improvements, so I'm preparing to make a new
> release. But first, I wanted to make the tests pass reliably also when
> running them in parallel (the only way I run them, because otherwise,
> they take way too long). I run a command like this:
> 
> make check AM_TESTSUITE_MAKE="make -j$(( 2*$(nproc) + 1 ))"
> 
> That evoked a surprising number of test failures. Note that these are
> nondeterministic and depend on the amount of parallelism. All tests
> may pass consistently on a system with 2 cores, yet some will fail
> every time when nproc prints at least 6.
> 
> I've fixed a few of these failures, but here is still at least one
> that I have not fixed: t/txinfo-no-clutter. In this case, running it
> with no parallelism and by itself passes:
>   make -k check TESTS='t/txinfo-no-clutter'
> Yet running it with parallel make fails (I'm using make-4.3.90 built
> from git, in case it matters):
>   make -k check TESTS='t/txinfo-no-clutter' AM_TESTSUITE_MAKE="make
> -j$(( 2*$(nproc) + 1 ))"
> 
> Not sure I'll work more on this today, so sharing: The attached log
> shows that this fails when building that test's final "all" rule,
> often because a .dvi file could not be found:
> 
> ) localization,mv: cannot stat 'bar.dvi': No such file or directory
> 
> I've attached the four commits I'm about to push as well as that
> individual testsuite.log, in case someone wants to help accelerate the
> process.

I got the same with the recent version in git: ran OK with

make -k check TESTS='t/txinfo-no-clutter'

but broke with


make -k check TESTS='t/txinfo-no-clutter' AM_TESTSUITE_MAKE="make -j$(( 
2*$(nproc) + 1 ))"

My make version is GNU Make 4.1.  I'm attaching testsuite.log.





==
   GNU Automake 1.16b: ./test-suite.log
==

# TOTAL: 1
# PASS:  0
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: t/txinfo-no-clutter
=

txinfo-no-clutter: running makeinfo --version
texi2any (GNU texinfo) 6.7dev

Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
txinfo-no-clutter: running texi2dvi --version
texi2dvi (GNU Texinfo 6.7dev)

Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
txinfo-no-clutter: running dvips --version
This is dvips(k) 5.997 Copyright 2017 Radical Eye Software
kpathsea version 6.2.3
Copyright 2017 Radical Eye Software.
There is NO warranty.  You may redistribute this software
under the terms of the GNU General Public License
and the Dvips copyright.
For more information about these matters, see the files
named COPYING and dvips.h.
Primary author of Dvips: T. Rokicki.
Running from installcheck: no
Test Protocol: none
PATH = 
/home/g/src/automake-GIT/automake/bin:/home/g/src/automake-GIT/automake/t/ax:/home/g/go/bin:/home/g/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/go/bin:/home/g/.emacs.d/bin/doom
++ pwd
/home/g/src/automake-GIT/automake/t/txinfo-no-clutter.dir
+ cat
+ cat
+ mkdir sub doc
+ cat
+ cat
+ cat
+ cat
+ cp baz.texi sub
+ aclocal-1.16 -Werror
+ automake-1.16 --foreign -Werror -Wall --add-missing
Makefile.am:2: installing './mdate-sh'
Makefile.am:2: installing './texinfo.tex'
sub/Makefile.am:2: installing 'sub/texinfo.tex'
+ autoconf
+ ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a race-free mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating sub/Makefile
+ for fmt in info pdf ps dvi html all
+ make -j5 info
Making info in sub
make[1]: Entering directory 
'/home/g/src/automake-GIT/automake/t/txinfo-no-clutter.dir/sub'
restore=: && backupdir=".am$$" && \
am__cwd=`pwd` && CDPATH="${ZSH_VERSION+.}:" && cd . && \
rm -rf $backupdir && mkdir $backupdir && \
if (/bin/bash 
'/home/g/src/automake-GIT/automake/t/txinfo-no-clutter.dir/missing' makeinfo 
--version) >/dev/null 2>&1; then \
  for f in baz.info baz.info-[0-9] baz.info-[0-9][0-9] baz.i[0-9] 
baz.i[0-9

preparing for a new release

2020-11-15 Thread Jim Meyering 
Karl has made quite a few improvements, so I'm preparing to make a new
release. But first, I wanted to make the tests pass reliably also when
running them in parallel (the only way I run them, because otherwise,
they take way too long). I run a command like this:

make check AM_TESTSUITE_MAKE="make -j$(( 2*$(nproc) + 1 ))"

That evoked a surprising number of test failures. Note that these are
nondeterministic and depend on the amount of parallelism. All tests
may pass consistently on a system with 2 cores, yet some will fail
every time when nproc prints at least 6.

I've fixed a few of these failures, but here is still at least one
that I have not fixed: t/txinfo-no-clutter. In this case, running it
with no parallelism and by itself passes:
  make -k check TESTS='t/txinfo-no-clutter'
Yet running it with parallel make fails (I'm using make-4.3.90 built
from git, in case it matters):
  make -k check TESTS='t/txinfo-no-clutter' AM_TESTSUITE_MAKE="make
-j$(( 2*$(nproc) + 1 ))"

Not sure I'll work more on this today, so sharing: The attached log
shows that this fails when building that test's final "all" rule,
often because a .dvi file could not be found:

) localization,mv: cannot stat 'bar.dvi': No such file or directory

I've attached the four commits I'm about to push as well as that
individual testsuite.log, in case someone wants to help accelerate the
process.


0001-tests-avoid-failures-due-to-missing-ar-lib.patch
Description: Binary data


0002-tests-accommodate-an-ac_aux_dir-of-.-or.patch
Description: Binary data


0003-doc-fix-quoting-in-suggested-parallel-test-invocatio.patch
Description: Binary data


0004-tests-protect-against-parallel-false-failure.patch
Description: Binary data


test-suite.log
Description: Binary data