On Sat, Jul 2, 2022, at 5:27 PM, Tom Lane wrote:
> I thought I'd respond to the question raised at [1] about which
> versions of m4 have this problem.
...
> I could build most versions on NetBSD 9.2/amd64,
...
> m4-1.4.7 shows largely the same set of failures as with 1.4.6
...
> With m4-1.4.8, there are just two failing tests
This is super helpful. I had been assuming it was a bad Apple patch ultimately
at fault, but I can reproduce more-or-less the same set of failures with m4
1.4.[67] on my Linux workstation, in enough detail to investigate.
It turns out that there are three underlying issues. One is a buggy regexp in
Autoconf's testsuite that does not do what the comments say it does, causing
any test that inspects m4 error messages to fail spuriously due to harmless
variation in the wording between m4 versions. I checked in commit 5d8a59e3 to
fix this.
The other two issues are genuine problems with m4 1.4.6 and 1.4.7:
1) This command
printf '%s\n' \
'changequote(<,>)define(<T>,<>)dnl' \
'define(<F>,<T(<traced>)>)dnl' \
'm4wrap(<F>)dnl' \
| m4 --debug=aflq --trace=T
prints
m4trace:stdin:3: -1- T(<traced>)
with m4 >= 1.4.8, but
m4trace: -1- T(<traced>)
with 1.4.6 and 1.4.7. The 'autom4te' wrapper script expects all 'm4trace:'
lines to start with a file name and a line number, so it fails to parse the
output. This is the root cause of roughly half the testsuite failures.
2) This command
printf '%s\n' \
'changequote(<,>)define(<T>,<__line__>)dnl' \
'T(<' \
'abc' \
'>)' \
| m4
prints '2' with m4 >= 1.4.8 but '4' with 1.4.6 and 1.4.7. That breaks the
other half of the failing tests, which are all looking at autoconf diagnostic
output and care about what line number is associated with a problem.
If it was just (2) I might have considered fixing it in the testsuite, but (1)
is serious enough that I do think desupporting 1.4.6 and 1.4.7 is justified.
The NEWS file for m4 1.4.8 mentions both of these as deliberate changes:
# * Warnings and errors issued during macro expansion are now consistently
# reported at the line where the macro name was detected, rather than
# where the close parenthesis resides. Text wrapped by `m4wrap' now
# remembers the location that was in effect when m4wrap was invoked,
# rather than changing to line 0 and the empty string for a file.
I have checked in commit 5d8a59e3 to make configure reject 1.4.6 and 1.4.7, and
verified that, on my computer, I get 100% test successes with 1.4.8 and 1.4.19
(again, this is Linux) after both patches are applied. I also verified that
configure does fail when 1.4.6 is the only version of m4 available.
The content of the patches is attached to this message.
> I soon found that GNU m4 is an incredibly fragile, system-dependent
> mess.
Yeah, with all the good will in the world, I don't think quadratic strstr() or
lack of support for fseeko() are things that an application should be trying to
work around. Get the C library fixed instead!
...
> 1.4.18 does build, and we're back to
>
> Subject: [GNU Autoconf 2.71] testsuite: 260 269 failed
I _would_ like to see the testsuite.log files from this run, specifically. NB
The test numbers change every time someone adds a new test, so it's not super
helpful to know that "260 and 269" are failing consistently, with no further
information.
I'll follow up on the AC_HEADER_STDBOOL failures on macOS in a separate message
(and possibly not for a few days).
> My recommendation would actually be to take out the check in m4.m4
> that purports to reject buggy strstr behavior --- if that's looking
> for something real, you couldn't tell it from the test suite.
It's a very real bug: the `index` builtin fails to find substrings that are
actually present. `index` (renamed `m4_index` for namespace cleanliness) is
used in dozens of places in autoconf's code, so if the testsuite passes with
the bug present, it's only by dumb luck.
zw
From fec8a33d87bbe9f024a31a18642618716b7ad7c9 Mon Sep 17 00:00:00 2001
From: Zack Weinberg <za...@panix.com>
Date: Wed, 6 Jul 2022 16:11:43 -0400
Subject: [PATCH 2/2] Require GNU M4 1.4.8 or later.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
The former minimum version was 1.4.6. 1.4.6 and 1.4.7 do not track
the original location of text fed to âm4wrapâ, which breaks autom4teâs
ability to trace macros invoked from _AC_FINALIZE.
Right now, the only _user_ visible effect of this is that autoconf
running on M4 1.4.6 or 1.4.7 will emit an internal error, instead of
the intended warning message, when it processes a configure.ac that
neglects to invoke AC_INIT or AC_OUTPUT. Perhaps more importantly,
it causes a bunch of scary-sounding failures in our own testsuite,
which deliberately doesnât use AC_OUTPUT sometimes.
M4 1.4.6 and 1.4.7 also have bugs in location tracking of macro
invocations spread over multiple lines, which, guess what, causes
even more testsuite failures.
1.4.8 came out in 2006. As a practical matter, this change to our
requirements means that people using macOS *to run autoconf* (not just
to run generated configure scripts) cannot use the system-provided m4
anymore. {Free,Net,Open}BSD already donât ship GNU M4 as their system
m4, so users of those OSes should not be affected.
---
NEWS | 14 ++++++++++++++
m4/m4.m4 | 14 +++++++++++---
2 files changed, 25 insertions(+), 3 deletions(-)
diff --git a/NEWS b/NEWS
index 4e41345d..91164390 100644
--- a/NEWS
+++ b/NEWS
@@ -7,6 +7,20 @@ GNU Autoconf NEWS - User visible changes.
*** Autoconf now requires perl 5.8 (2002) or later.
Generated 'configure' scripts continue to run without perl.
+*** Autoconf now requires GNU M4 1.4.8 (2006) or later.
+ Generated 'configure' scripts continue to run without M4.
+
+ Use of GNU M4 1.4.16 or later is recommended, as all earlier versions
+ are known to have had serious bugs in the text-processing builtins
+ on some, but not all, operating systems. Autoconfâs own configure
+ script will attempt to find a version of M4 that is not affected by
+ these bugs.
+
+ Note: Autoconf 2.70 and 2.71 include code that malfunctions with
+ M4 1.4.6 or 1.4.7. However, the only effect of the malfunction is
+ that you will get a confusing error message if you run autoconf on
+ a configure.ac that neglects to use AC_INIT or AC_OUTPUT.
+
** New features
** Obsolete features and new warnings
diff --git a/m4/m4.m4 b/m4/m4.m4
index 0fedaca8..66e7341f 100644
--- a/m4/m4.m4
+++ b/m4/m4.m4
@@ -10,7 +10,7 @@
# AC_PROG_GNU_M4
# --------------
-# Check for GNU M4, at least 1.4.6 (all earlier versions had bugs in
+# Check for GNU M4, at least 1.4.8 (all earlier versions had bugs in
# trace support and regexp support):
# https://lists.gnu.org/archive/html/bug-gnu-utils/2006-11/msg00096.html
# https://lists.gnu.org/archive/html/bug-autoconf/2009-07/msg00023.html
@@ -18,7 +18,7 @@
# is supported, and AC_SUBST M4_DEBUGFILE accordingly.
# Also avoid versions of m4 that trigger strstr bugs.
AC_DEFUN([AC_PROG_GNU_M4],
- [AC_ARG_VAR([M4], [Location of GNU M4 1.4.6 or later. Defaults to the first
+ [AC_ARG_VAR([M4], [Location of GNU M4 1.4.8 or later. Defaults to the first
program of 'm4', 'gm4', or 'gnum4' on PATH that meets Autoconf needs.])
AC_CACHE_CHECK([for GNU M4 that supports accurate traces], [ac_cv_path_M4],
[rm -f conftest.m4f
@@ -36,13 +36,21 @@ AC_PATH_PROGS_FEATURE_CHECK([M4], [m4 gm4 gnum4],
ac_snippet=${ac_snippet}${as_nl}if'else(in''dex(dnl
;:11-:12-:12-:12-:12-:12-:12-:12-:12.:12.:12.:12.:12.:12.:12.:12.:12-,dnl
:12-:12-:12-:12-:12-:12-:12-:12-),-1,,strstr-bug2)'
+ # Root out M4 1.4.6 and 1.4.7, which do not implement --debug=aflq
+ # correctly for macros invoked from m4wrap.
+ ac_snip2=change'quote(<,>)def''ine(<T>,<>)d'nl
+ ac_snip2=${ac_snip2}${as_nl}def'ine(<F>,<T(<traced>)>)d'nl
+ ac_snip2=${ac_snip2}${as_nl}m4'wrap(<F>)d'nl
test -z "`$ac_path_M4 -F conftest.m4f </dev/null 2>&1`" \
&& test -z "`AS_ECHO([$ac_snippet]) | $ac_path_M4 --trace=mac 2>&1`" \
&& test -f conftest.m4f \
+ && test x"`AS_ECHO([$ac_snip2]) | \
+ $ac_path_M4 --trace=T --debug=aflq 2>&1`" = \
+ x'm4trace:stdin:3: -1- T(<traced>)' \
&& ac_cv_path_M4=$ac_path_M4 ac_path_M4_found=:
rm -f conftest.m4f],
[AC_MSG_ERROR([no acceptable m4 could be found in \$PATH.
-GNU M4 1.4.6 or later is required; 1.4.16 or newer is recommended.
+GNU M4 1.4.8 or later is required; 1.4.16 or newer is recommended.
GNU M4 1.4.15 uses a buggy replacement strstr on some systems.
Glibc 2.9 - 2.12 and GNU M4 1.4.11 - 1.4.15 have another strstr bug.])])])
M4=$ac_cv_path_M4
--
2.36.1
From 5d8a59e337712be6b32b77e61fcc269093a80c66 Mon Sep 17 00:00:00 2001
From: Zack Weinberg <za...@panix.com>
Date: Wed, 6 Jul 2022 16:05:42 -0400
Subject: [PATCH 1/2] testsuite: Handle `balanced ASCII quotes' correctly in m4
errors.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
AT_CHECK_M4 was _documented_ to convert
m4: cannot open `foo': No such file or directory
to
m4: cannot open 'foo': No such file or directory
but the regexp that was supposed to do this didnât work.
Fix the regexp, and remove a workaround for the bug in one test
(yes, we had code in our own testsuite to work around bugs in our
own testsuite!)
* tests/local.at (AT_CHECK_M4): Fix regex for âcannot openâ messages.
* tests/tools.at (autom4te cache): Remove workaround for buggy regex
in AT_CHECK_M4.
---
tests/local.at | 12 ++++++------
tests/tools.at | 12 +-----------
2 files changed, 7 insertions(+), 17 deletions(-)
diff --git a/tests/local.at b/tests/local.at
index 4c2c9903..3f348929 100644
--- a/tests/local.at
+++ b/tests/local.at
@@ -158,14 +158,14 @@ m4_define([AT_CHECK_M4],
m4_case([$4], [], [], [ignore], [],
[AT_CHECK([[mv stderr stderr-raw &&
sed 's/^[^:]*m4[-.ex0-9]*: *\([^:]*:\) *\([0-9][0-9]*: \)/m4:\1\2/
- s/^\([^:]*:\) *\([0-9][0-9]*:\)[^:]*m4[-.ex0-9]*: /m4:\1\2 /
- s/: C\(annot open \)\([^`'\'':]*\):/: c\1'\''\2'\'':/
- s/: include:\( cannot open\)/:\1/
- s/^autom4te: [^ ]*m4[.ex]* /autom4te: m4 /
- s/^autom4te: error: [^ ]*m4[.ex]* /autom4te: error: m4 /
+ s/^\([^:]*:\) *\([0-9][0-9]*:\)[^:]*m4[-.ex0-9]*: /m4:\1\2 /
+ s/: include: [cC]\(annot open\)/: c\1/
+ s/: [cC]\(annot open \)[`'\'']*\([^'\'':]*\)'\''*:/: c\1'\''\2'\'':/
+ s/^autom4te: [^ ]*m4[.ex]* /autom4te: m4 /
+ s/^autom4te: error: [^ ]*m4[.ex]* /autom4te: error: m4 /
s!^.*/\([^/][^/]*\)\.m4: *[0-9][0-9]*: *!\1.m4: !
s!^.*/\([^/][^/]*\)\.m4: *[0-9][0-9]*: *[0-9][0-9]*: *!\1.m4: !
- s/ (E[A-Z]*)$//
+ s/ (E[A-Z]*)$//
' stderr-raw >&2]], [0], [], [$4])])
])
diff --git a/tests/tools.at b/tests/tools.at
index ce7efc33..bd225e04 100644
--- a/tests/tools.at
+++ b/tests/tools.at
@@ -98,20 +98,10 @@ AT_CHECK_M4SUGAR
# We moved a file: it should fail
mkdir sub
mv foo sub
-case `$M4 no/such/file 2>&1` in
- *\`no/such/file*)
- AT_CHECK_M4SUGAR([], [1], [],
-[m4:script.4s:1: cannot open `foo': No such file or directory
-autom4te: error: m4 failed with exit status: 1
-])
- ;;
- *)
- AT_CHECK_M4SUGAR([], [1], [],
+AT_CHECK_M4SUGAR([], [1], [],
[m4:script.4s:1: cannot open 'foo': No such file or directory
autom4te: error: m4 failed with exit status: 1
])
- ;;
-esac
# But if we change the main file, then we should no longer complain of
# missing files.
--
2.36.1
