Re: Reproducible tarball problem due to aclocal -I ordering
Simon Josefsson via Gnulib discussion list writes: >>> What do you think about this change? >>> >>> - AUTORECONFFLAGS="--verbose --install --force -I $m4_base $ACLOCAL_FLAGS" >>> + AUTORECONFFLAGS="--verbose --install --force $ACLOCAL_FLAGS" >> >> It sounds like 'autoreconf' does a better job at extracting the list of >> m4 files from configure.ac that 'bootstrap'. gnulib-tool also has code >> for doing this (see gnulib-tool.sh lines 7322..7342). >> So: If the extra '-I $m4_base' is not needed, then omit it. But if it is >> needed, I would be in favour of adding all necessary -I options here, not >> just one of them. > > Great. I'll patch away this particular '-I' out of libidn2's > bootstrap-funclib.sh and see if it fixes the problem and doesn't break > anything else for libidn2. Later on we can try it with other packages > too, and maybe change things in gnulib too. I've used this patch in releases of libidn2 and libidn, and now I ran into the same problem when making GNU SASL tarballs reproducible too. So I think the patch is the right solution. I've pushed the attached patch so we can see if it breaks any other project. I can only think that this could cause problems if the gnulib m4_base is missing from configure.ac AC_CONFIG_MACRO_DIRS, but that is a fatal error anyway. Although with these subtle things you never really know for sure, it seems, at least not until you try. /Simon From 54fc57c23dcd833819a7adbdfcc3bd1c805103a8 Mon Sep 17 00:00:00 2001 From: Simon Josefsson Date: Thu, 27 Mar 2025 20:01:28 +0100 Subject: [PATCH] bootstrap: Improve "make dist" tarball reproducability. * top/bootstrap-funclib.sh (autogen): Don't pass unnecessary -I's when invoking autoreconf. The order of M4 include directories is reflected in Makefile.in content. Our GNUmakefile _version rule invoke autoreconf without any extra -I, and this difference in how autoreconf is invoked results in different tarballs for "make dist" vs "make release" which is undesirable. --- ChangeLog| 10 ++ top/bootstrap-funclib.sh | 4 ++-- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/ChangeLog b/ChangeLog index a6b62b9f44..53fa2eb158 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,13 @@ +2025-03-27 Simon Josefsson + + bootstrap: Improve "make dist" tarball reproducability. + * top/bootstrap-funclib.sh (autogen): Don't pass unnecessary -I's + when invoking autoreconf. The order of M4 include directories is + reflected in Makefile.in content. Our GNUmakefile _version rule + invoke autoreconf without any extra -I, and this difference in how + autoreconf is invoked results in different tarballs for "make + dist" vs "make release" which is undesirable. + 2025-03-27 Simon Josefsson maintainer-makefile: Check spelling using 'codespell'. diff --git a/top/bootstrap-funclib.sh b/top/bootstrap-funclib.sh index 47e015bd24..81884cc8b3 100644 --- a/top/bootstrap-funclib.sh +++ b/top/bootstrap-funclib.sh @@ -1,6 +1,6 @@ # A library of shell functions for autopull.sh, autogen.sh, and bootstrap. -scriptlibversion=2025-02-16.12; # UTC +scriptlibversion=2025-03-27.18; # UTC # Copyright (C) 2003-2025 Free Software Foundation, Inc. # @@ -1300,7 +1300,7 @@ autogen() # Invoke autoreconf with --force --install to ensure upgrades of tools # such as ylwrap. - AUTORECONFFLAGS="--verbose --install --force -I $m4_base $ACLOCAL_FLAGS" + AUTORECONFFLAGS="--verbose --install --force $ACLOCAL_FLAGS" AUTORECONFFLAGS="$AUTORECONFFLAGS --no-recursive" # Tell autoreconf not to invoke autopoint or libtoolize; they were run above. -- 2.49.0 signature.asc Description: PGP signature
Re: Reproducible tarball problem due to aclocal -I ordering
Bruno Haible via Gnulib discussion list writes: > Simon Josefsson wrote: >> If the diff is unreadable, the summary is that it just moves around >> things. >> ... >> The problem happens because 'make release' invoke (via GNUmakefile >> '_version' target) autoreconf to re-generate files for the new version >> number. I believe this step is working fine, and uses the "right" m4 >> directory path as specified by configure.ac. It is bootstrap that is >> using a different m4 path ordering, which I suspect is "incorrect". >> These two different usages is causing the problem. > > One may view it differently: 'bootstrap' (or autogen.sh in other packages) > is the official way to build the generated files. 'autoreconf' is not > generally supported by GNU packages: in some packages it may work, in > other packages it does not work (because autogen.sh often contains > more commands, or because of macros like pkg.m4 that autoreconf does > not know how to pull it). Right, I think there are several ways to view this. It is gnulib that is invoking 'autoreconf' here on 'make release' and I think that usage is fine and should continue to work (or we should fix gnulib not to call autoreconf from the GNUmakefile '_version' rule). What may be missing is to understand the end-user experience: to trust release tarballs, presumably a user would want to be able to reproduce them from sources in the git repository for the release tag. A simple way is for a user to do roughly: git clone ... cd project git checkout v1.2.3 ./bootstrap ./configure make dist and assuming they have the same set of build dependencies that were documented in the release notes, you should end up with a bit-by-bit identical tarball. I don't think we want to require that users run 'make release' here. For a very long time, 'make dist' has been the documented way to prepare tarballs, and I don't see a strong reason to break this now. Also 'make release' does many other things beyond preparing the tarballs (PGP signature, announcement text, runs 'git commit' etc), which are not useful for the user in this situation. Btw, I think you earlier suggested that there should be some machine-readable format for describing build dependencies. I've realized that we are quite close to this now: The Guix commit id that I included in the last inetutils+libtasn1 releases is sufficient to establish a build environment with all the right pre-dependencies of the right version, and Guix guarantee that these ought to be forever available via 'guix time-machine'. This approach is Guix-specific, and it is currently missing the list of necessary package names (which can be inferred/guessed easily, and the names are stable over time for the same git id), but at least it is a step in the right direction. Accomplishing anything near this level of long-term reproducability with Debian/Fedora/ArchLinux/etc-based distributions are not possible as far as I know, and I don't see any way to implement that either. Generalizing beyond a specific distribution is difficult, but the >> But there shouldn't be any requirement that the gnulib m4 directory is >> the first, should there? > > There is no such requirement. Good! Thanks for agreeing on that, I wasn't certain of this assumption. I hadn't expect the aclocal -I directory ordering to lead to different tarballs just because the files are sorted in different ways in aclocal.m4 and Makefile.in's. What is your understanding of aclocal here, would it be possible for 'aclocal' to sort all *.m4 files it was using before storing them into aclocal.m4? Then -I sort order would not matter any more. I suspect this may break things though (just like changing order of -I's to gcc may break things). >> What do you think about this change? >> >> - AUTORECONFFLAGS="--verbose --install --force -I $m4_base $ACLOCAL_FLAGS" >> + AUTORECONFFLAGS="--verbose --install --force $ACLOCAL_FLAGS" > > It sounds like 'autoreconf' does a better job at extracting the list of > m4 files from configure.ac that 'bootstrap'. gnulib-tool also has code > for doing this (see gnulib-tool.sh lines 7322..7342). > So: If the extra '-I $m4_base' is not needed, then omit it. But if it is > needed, I would be in favour of adding all necessary -I options here, not > just one of them. Great. I'll patch away this particular '-I' out of libidn2's bootstrap-funclib.sh and see if it fixes the problem and doesn't break anything else for libidn2. Later on we can try it with other packages too, and maybe change things in gnulib too. /Simon signature.asc Description: PGP signature
Re: Reproducible tarball problem due to aclocal -I ordering
Simon Josefsson wrote: > If the diff is unreadable, the summary is that it just moves around > things. > ... > The problem happens because 'make release' invoke (via GNUmakefile > '_version' target) autoreconf to re-generate files for the new version > number. I believe this step is working fine, and uses the "right" m4 > directory path as specified by configure.ac. It is bootstrap that is > using a different m4 path ordering, which I suspect is "incorrect". > These two different usages is causing the problem. One may view it differently: 'bootstrap' (or autogen.sh in other packages) is the official way to build the generated files. 'autoreconf' is not generally supported by GNU packages: in some packages it may work, in other packages it does not work (because autogen.sh often contains more commands, or because of macros like pkg.m4 that autoreconf does not know how to pull it). > But there shouldn't be any requirement that the gnulib m4 directory is > the first, should there? There is no such requirement. > What do you think about this change? > > - AUTORECONFFLAGS="--verbose --install --force -I $m4_base $ACLOCAL_FLAGS" > + AUTORECONFFLAGS="--verbose --install --force $ACLOCAL_FLAGS" It sounds like 'autoreconf' does a better job at extracting the list of m4 files from configure.ac that 'bootstrap'. gnulib-tool also has code for doing this (see gnulib-tool.sh lines 7322..7342). So: If the extra '-I $m4_base' is not needed, then omit it. But if it is needed, I would be in favour of adding all necessary -I options here, not just one of them. Bruno
