Re: split bootstrap in two phases

[Simon Josefsson via Gnulib discussion list]

Bruno Haible  writes:

> Hi all 'bootstrap' users,
>
> Over the last few years, it has become more and more clear that bootstrap
> does two things:
>   (1) Fetch auxiliary files that are not in the git checkout.
>   This is the part that requires network access and that has
>   supply-chain concerns.
>   (2) Generate files such as configure, config.h, Makefile.in etc.
>   This includes running gnulib-tool.
>
> Recent discussion in gnu-prog-discuss has shown that making the separation
> into two phases (1) and (2) explicit will have several benefits:
>
>   * For reproducible builds, it is necessary to be execute the second
> phase without the first one.
>
>   * For people who have local modifications in dependency packages
> (e.g. in gnulib), or for distros who want to apply patches to .m4 files,
> it is important to have these phases separated.
>
>   * The second phase is a way for people to regenerate files in a
> supported way. So far, too many users have been running 'autoreconf -fvi',
> which many GNU packages don't support.

To replace the use of 'autoreconf -fvi', doesn't (at least) the
autogen.sh script needs to be EXTRA_DIST'ed?  How should this be
achieved?  Probably some end-user documentation snippet about this
should be available.

It would be nice if Debian would then use ./autogen.sh instead of
autoreconf -fvi to re-generated all generated files from the tarball.
But the first step is to ship it.

/Simon


signature.asc
Description: PGP signature

Re: split bootstrap in two phases, GNU libidn

[Simon Josefsson via Gnulib discussion list]

Bruno Haible  writes:

> Hi dear GNU libidn developers (Simon et al.),
..
> These adjustments will be needed the next time you use the '--bootstrap-sync'
> option.

Thank you Bruno -- should be fixed in libidn git now.

/Simon


signature.asc
Description: PGP signature

Re: split bootstrap in two phases

[Paul Eggert]

On 7/30/22 22:47, Bruno Haible wrote:

Does the new documentation ([1], case 2, sub-case (C), as well as case 3)
cover Emacs adequately?


Yes, Emacs uses case (3), mostly I think because nobody but me wanted to 
worry about integrating with Gnulib.

Re: split bootstrap in two phases

[Bruno Haible]

Paul Eggert wrote:
> For what it's worth Emacs has its own autogen.sh. But it doesn't use 
> 'bootstrap' so I expect it won't collide.

Does the new documentation ([1], case 2, sub-case (C), as well as case 3)
cover Emacs adequately?
We can't go into all details here, because there is significant variation
between what the various package do. But are the main statements correct
now?

[1] https://www.gnu.org/software/gnulib/manual/html_node/VCS-Issues.html

Re: split bootstrap in two phases

[Paul Eggert]

On 7/30/22 16:43, Bruno Haible wrote:

There is no actual collision, because your tool is 'autogen' [1],
and this is 'autogen.sh'.


For what it's worth Emacs has its own autogen.sh. But it doesn't use 
'bootstrap' so I expect it won't collide.

Re: split bootstrap in two phases

[Bruce Korb]

On 7/30/22 16:43, Bruno Haible wrote:

I offered to help Miguel de Icaza with his Gnome project,
he didn't want the help, but did adopt my project's name.


Oh, so the similarity between the names is not completely fortuitous.

But meanwhile the name has spread; it is now even used in the Linux
kernel [2]. I guess it's too late to turn back the clock. "This train
has departed", we say in German.


It's left the station in English, too. I'm just thinking we don't need 
an extra rail car running around the tracks. :)

Re: split bootstrap in two phases

[Bruno Haible]

Hi Bruce,

> >> The second phase is a script 'autogen.sh'.
> 
> I'd like to object to that name. :)

There is no actual collision, because your tool is 'autogen' [1],
and this is 'autogen.sh'. Also, since a significant percentage of the
users don't have '.' in $PATH, the instructions say to run './autogen.sh';
hence there is no conflict with any program in /usr/bin.

But I admit there may be some confusion.

> I offered to help Miguel de Icaza with his Gnome project,
> he didn't want the help, but did adopt my project's name.

Oh, so the similarity between the names is not completely fortuitous.

But meanwhile the name has spread; it is now even used in the Linux
kernel [2]. I guess it's too late to turn back the clock. "This train
has departed", we say in German.

> I've even gotten some folks to ask me for help with their
> bootstrapping scripts because they were named, "autogen.sh".
> So, please, not that. :)

You could see it as a business opportunity :)

Bruno

[1] https://www.gnu.org/software/autogen/
[2] 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/usb/usbip/autogen.sh?h=v5.18

Re: split bootstrap in two phases

[Bruce Korb]

On 7/30/22 13:45, Simon Josefsson via Gnulib discussion list wrote:

Wonderful, thank you!  I have yet to digest everything, so I'll fall
back on stylistic comments:


The first phase is a script 'autopull.sh'.
The second phase is a script 'autogen.sh'.


I'd like to object to that name. :)
I offered to help Miguel de Icaza with his Gnome project,
he didn't want the help, but did adopt my project's name.
I've even gotten some folks to ask me for help with their
bootstrapping scripts because they were named, "autogen.sh".
So, please, not that. :)

Re: split bootstrap in two phases - .sh suffix

[Bruno Haible]

Simon Josefsson wrote:
> > The first phase is a script 'autopull.sh'.
> > The second phase is a script 'autogen.sh'.
> ...
> > The names of these scripts end with '.sh' in order to make them easily
> > editable in editors that recognize the file type from the suffix.
> 
> What do you think about dropping the .sh suffix?  As far as I know, few
> if any tools that users/developers are expected to invoke have a .sh
> extension.  To me, these scripts are similar to ./configure or
> ./bootstrap and we haven't used the .sh suffix for those.

That was my initial thought too: If we want a standard interface for
invocation only, then we don't need a suffix.

But:
  * The majority of packages will probably (like libunistring, gettext,
libiconv, libsigsegv, gperf, etc.) continue to use hand-written
autopull and autogen scripts. Simply because a 10-lines hand-written
script is easier to start with than yet another configuration
file whose syntax and semantics has to be understood.

  * Such scripts share the same limitations as configure: They can call
only the coreutils, sed, grep, and git. Therefore it's unlikely that
they will be compiled programs or python or perl scripts.

  * By adding the suffix '.sh' we're inviting the developers to see what's
in it, rather than to consider them as a black box. And to create
their own from scratch.

> I know autogen.sh is a common idiom, but it does different things in
> different projects

If the concept of the two phases, that you defined, is convincing, the
other packages will converge to this semantics over time.

Bruno

Re: split bootstrap in two phases - why two scripts

[Bruno Haible]

Simon Josefsson wrote:
> Thinking about all this, do we really need two new scripts?  There
> is inflation in all these tools and documentation files.  Could the
> separation be done via './bootstrap --pull' and './bootstrap --generate'
> with the default './bootstrap' be to do both?

It's two different scripts because the developer runs them at different
time points:

  * When starting to work on a package, after a period of inactivity, the
developer will run
  $ git pull && ./autopull.sh
Or when switching to a different branch:
  $ git checkout branch-x-y && ./autopull.sh

  * After making changes to the source code (including the source code
of the submodules), the developer will run
  $ ./autogen.sh

  * After making changes to the source code (excluding the submodules and
the bootstrap.conf), the developer will run
  $ ./autogen.sh --skip-gnulib

Those packages that support this option document it like this:
  --skip-gnulib   Avoid fetching files from Gnulib.
  This option is useful
  - when you are working from a released tarball 
(possibly
with modifications), or
  - as a speedup, if the set of gnulib modules did not
change since the last time you ran this script.

For tools that are convenient to developers, long options are a handicap.
(Do you like typing '../libtool --mode=execute gdb program' ?)

The second reason for having two scripts is that the developer can change
the source code of the submodules in between.

Take, as an analogy, our usual build recipe:

  ./configure && make && make check

Is every package shipping a 'build' script

   #!/bin/sh
   ./configure "$@" && ${MAKE-make} && ${MAKE-make} check

?
No.

Do we have a 'build' script that can be called as
   ./build --skip-configure
   ./build --skip-install
   ./build --only-make
?
No again. Why? Because

  - The developers like to do some things between the various phases
(e.g. remove -O2 options in the selected generated Makefiles).

  - There are many different ways a developer can work; for example,
one developer may want every output to be stored in a log file;
another developer wants to compare the output to the output of the
previous run; and so on. It's easier for the developers to use the
individual bricks (configure, make, etc.) rather than a script that
does all at once. (I'm *not* talking about Visual Studio or Xcode
addicts.)

These considerations also apply to autopull.sh and autogen.sh.

Bruno

Re: split bootstrap in two phases

[Simon Josefsson via Gnulib discussion list]

Bruno Haible  writes:

> Recent discussion in gnu-prog-discuss has shown that making the separation
> into two phases (1) and (2) explicit will have several benefits:

Wonderful, thank you!  I have yet to digest everything, so I'll fall
back on stylistic comments:

> The first phase is a script 'autopull.sh'.
> The second phase is a script 'autogen.sh'.
...
> The names of these scripts end with '.sh' in order to make them easily
> editable in editors that recognize the file type from the suffix.

What do you think about dropping the .sh suffix?  As far as I know, few
if any tools that users/developers are expected to invoke have a .sh
extension.  To me, these scripts are similar to ./configure or
./bootstrap and we haven't used the .sh suffix for those.

I know autogen.sh is a common idiom, but it does different things in
different projects, so keeping the same name may not really be
important.  That may even be a reason to chose a different name than
autogen.sh for our new well-defined and documented purpose of the script
you propose.

Hmm.  Thinking about all this, do we really need two new scripts?  There
is inflation in all these tools and documentation files.  Could the
separation be done via './bootstrap --pull' and './bootstrap --generate'
with the default './bootstrap' be to do both?  While internally the
implementation needs a significant change, externally and
documentation-wise it is not a large change.  Making sure that
./bootstrap --generate never ever pull things from the Internet is
critical, and is more complex than if that part was separated.

I've yet to catch up on the gnu-prog-discuss followups, but hope to get
to it in the upcoming week..

/Simon


signature.asc
Description: PGP signature

Re: split bootstrap in two phases, GNU tar

[Bruno Haible]

Hi Paul,

> Unfortunately that didn't work for me when I tried it with GNU Tar, 
> which led to some of my confusion.
> 
> Here's how I can reproduce the problem, starting with a fresh checkout 
> of the latest GNU Tar (commit 5c4f8cadbd362497dc3c3136566589557ce00f1b). 
> The following commands replace Tar's 'bootstrap' (which is 
> top/bootstrap) with build-aux/bootstrap, removes the files that are 
> supposed to be fetched automatically, and then runs ./bootstrap which 
> fails because the files are not fetched automatically.
> 
>$ cp $HOME/src/gnu/gnulib/build-aux/bootstrap .
>$ rm autogen.sh autopull.sh bootstrap-funclib.sh
>$ ./bootstrap
>grep: paxutils/gnulib.modules: No such file or directory
>./bootstrap: Bootstrapping from checked-out tar sources...
>./bootstrap: line 793: ./autopull.sh: No such file or directory
>./bootstrap: autopull.sh failed.

Oh, I see. From this recipe, I guess the first phase of the automatic
sync succeeded, but the second phase didn't (because your bootstrap.conf
was not yet adjusted). As a result, the intermediate 'bootstrap' without
its companion files was left on the disk. And that is what caused the
further errors and need for manual intervention.

This patch fixes it.


2022-07-25  Bruno Haible  

bootstrap: Make the automatic sync more resilient.
Reported by Paul Eggert in
.
* top/bootstrap (bootstrap_sync): Set to true when bootstrap-funclib.sh
is not present.
* top/gen-bootstrap.sed: Insert a couple of comment lines.
* build-aux/bootstrap: Regenerated using "make build-aux/bootstrap".

diff --git a/top/bootstrap b/top/bootstrap
index b7216d20fe..e9d8ad82aa 100755
--- a/top/bootstrap
+++ b/top/bootstrap
@@ -153,14 +153,7 @@ do
   --bootstrap-sync)
 bootstrap_sync=true;;
   --no-bootstrap-sync)
-if test -f "$medir"/bootstrap-funclib.sh; then
-  bootstrap_sync=false
-else
-  # We have only completed the first phase of an upgrade from a bootstrap
-  # version < 2022-07-24. Need to do the second phase now.
-  bootstrap_sync=true
-fi
-;;
+bootstrap_sync=false;;
   --no-git)
 use_git=false;;
   *)
@@ -179,6 +172,12 @@ fi
 
 check_build_prerequisites $use_git
 
+if ! test -f "$medir"/bootstrap-funclib.sh; then
+  # We have only completed the first phase of an upgrade from a bootstrap
+  # version < 2022-07-24. Need to do the second phase now.
+  bootstrap_sync=true
+fi
+
 if $bootstrap_sync; then
   prepare_GNULIB_SRCDIR
   upgrade_bootstrap
diff --git a/top/gen-bootstrap.sed b/top/gen-bootstrap.sed
index 2851db5eaf..c182abfa50 100644
--- a/top/gen-bootstrap.sed
+++ b/top/gen-bootstrap.sed
@@ -1,3 +1,13 @@
+1{
+a\
+# DO NOT EDIT! GENERATED AUTOMATICALLY!
+a\
+# This script is only a trampoline that fetches the companion scripts
+a\
+# (bootstrap-funclib.sh, autopull.sh, autogen.sh).
+a\
+
+}
 /^[.] "[$]medir"[/]bootstrap-funclib.sh/{
   s/^.*$//
   r top/bootstrap-funclib.sh

Re: split bootstrap in two phases, GNU tar

[Paul Eggert]

On 7/24/22 16:18, Bruno Haible wrote:


For (a), everything works already. The old 'bootstrap' script has
hardwired code to fetch gnulib/build-aux/bootstrap. This gets executed,
and this intermediate, not really functional 'bootstrap' script gets
copied into the package and gets executed. It notices that the 3
companion files are still missing, copies the 4 files into the package,
and re-execs the (now fully functional) 'bootstrap'. In other words,
the only purpose of the gnulib/build-aux/bootstrap is a trampoline
to fetch the other files.


Unfortunately that didn't work for me when I tried it with GNU Tar, 
which led to some of my confusion.


Here's how I can reproduce the problem, starting with a fresh checkout 
of the latest GNU Tar (commit 5c4f8cadbd362497dc3c3136566589557ce00f1b). 
The following commands replace Tar's 'bootstrap' (which is 
top/bootstrap) with build-aux/bootstrap, removes the files that are 
supposed to be fetched automatically, and then runs ./bootstrap which 
fails because the files are not fetched automatically.


  $ cp $HOME/src/gnu/gnulib/build-aux/bootstrap .
  $ rm autogen.sh autopull.sh bootstrap-funclib.sh
  $ ./bootstrap
  grep: paxutils/gnulib.modules: No such file or directory
  ./bootstrap: Bootstrapping from checked-out tar sources...
  ./bootstrap: line 793: ./autopull.sh: No such file or directory
  ./bootstrap: autopull.sh failed.

(The "grep:" diagnostic is an annoyance that I haven't bothered to get 
to the bottom of; it is output even if I omit the "rm" command which 
causes the bootstrap to succeed.)


Perhaps I'm doing something wrong in tar/bootstrap.conf? It wasn't 
entirely clear to me which actions should go into which hooks.

Re: split bootstrap in two phases, GNU tar

[Bruno Haible]

Replying to https://lists.gnu.org/archive/html/bug-gnulib/2022-07/msg00058.html 
:

Hi Paul,

> I got confused by the two Gnulib files build-aux/bootstrap and top/bootstrap. 
> Although top/bootstrap says "The canonical version of this script is 
> maintained as build-aux/bootstrap in gnulib", it's really in top/bootstrap.

Oops, indeed. Fixing the comments through the patch below.

> More important, build-aux/bootstrap exists separately from top/bootstrap only 
> to support packages that don't have bootstrap-funclib.sh ... fix 
> top/gen-bootstrap.sed so that it inlines autopull.sh and autogen.sh as well 
> as inlining bootstrap-funclib.sh

What you see as a problem is already the solution to the problem.

Namely, there are two ways for a maintainer to upgrade to a newer 'bootstrap'.
(a) Run './bootstrap --bootstrap-sync', or just './bootstrap' if the
configuration contains the assignment 'bootstrap_sync=true'. Then,
commit the local modifications to the package's git repository.
(b) Copy the 4 files from gnulib/top/ manually, and commit them in the
package's git repository.

For (a), everything works already. The old 'bootstrap' script has
hardwired code to fetch gnulib/build-aux/bootstrap. This gets executed,
and this intermediate, not really functional 'bootstrap' script gets
copied into the package and gets executed. It notices that the 3
companion files are still missing, copies the 4 files into the package,
and re-execs the (now fully functional) 'bootstrap'. In other words,
the only purpose of the gnulib/build-aux/bootstrap is a trampoline
to fetch the other files.

For (b), I'll need to add a bit of comments/documentation somewhere.

> perhaps it'd be better for GNU tar to have these new files in a new 
> directory, i.e., boot/autogen.sh, boot/autopull.sh, 
> boot/bootstrap-funclib.sh, instead of autogen.sh, autopull.sh, 
> bootstrap-funclib.sh, so as to keep the top level simple. Is that easy to 
> arrange?

autopull.sh and autogen.sh are meant to be used by the developers,
several times a day. In the long term, they will be as important as
'INSTALL', 'HACKING', and 'configure', which are at the top-level.
I don't think it is useful to put them in a subdirectory.

bootstrap-funclib.sh could be put into a subdirectory; maybe
build-aux/ is the right one for this? (But determining $build_aux
requires parsing configure.ac...)

The forked bootstrap, by Gary Vaughan and others, at
https://github.com/gnulib-modules/bootstrap, uses another approach:
it presents itself as a git submodule. This is "clean", but resurfaces
the problems that 'bootstrap' / 'autopull.sh' were meant to solve,
namely
  - to present a simple, single command that can be executed
directly after 'git clone' or 'git pull',
  - to hide the complexities of 'git submodule' from the user.
(I have read about submodules several times over the last
few years, and still find that they are a usability mess.)


2022-07-24  Bruno Haible  

Fix comment.
Reported by Paul Eggert in
.
* top/bootstrap-funclib.sh: Fix comment about canonical location.
* top/autopull.sh: Likewise.
* top/autogen.sh: Likewise.
* top/bootstrap: Likewise.

diff --git a/top/autogen.sh b/top/autogen.sh
index f245fb11f2..02028f272b 100755
--- a/top/autogen.sh
+++ b/top/autogen.sh
@@ -20,12 +20,12 @@
 # along with this program.  If not, see .
 
 # Originally written by Paul Eggert.  The canonical version of this
-# script is maintained as build-aux/autogen.sh in gnulib.  However,
-# to be useful to your package, you should place a copy of it under
-# version control in the top-level directory of your package.  The
-# intent is that all customization can be done with a bootstrap.conf
-# file also maintained in your version control; gnulib comes with a
-# template build-aux/bootstrap.conf to get you started.
+# script is maintained as top/autogen.sh in gnulib.  However, to be
+# useful to your package, you should place a copy of it under version
+# control in the top-level directory of your package.  The intent is
+# that all customization can be done with a bootstrap.conf file also
+# maintained in your version control; gnulib comes with a template
+# build-aux/bootstrap.conf to get you started.
 #
 # Alternatively, you can use an autogen.sh script that is specific
 # to your package.
diff --git a/top/autopull.sh b/top/autopull.sh
index f4c0f49652..ffb4ba7cba 100755
--- a/top/autopull.sh
+++ b/top/autopull.sh
@@ -18,12 +18,12 @@
 # along with this program.  If not, see .
 
 # Originally written by Paul Eggert.  The canonical version of this
-# script is maintained as build-aux/autopull.sh in gnulib.  However,
-# to be useful to your package, you should place a copy of it under
-# version control in the top-level directory of your package.  The
-# intent is that all 

Re: split bootstrap in two phases, GNU tar

[Paul Eggert]

On 7/24/22 09:09, Bruno Haible wrote:


For GNU tar some adjustment of the bootstrap.conf is needed.


Thanks, I attempted to do that by installing the attached patch into GNU 
'tar'.


I got confused by the two Gnulib files build-aux/bootstrap and 
top/bootstrap. Although top/bootstrap says "The canonical version of this
 script is maintained as build-aux/bootstrap in gnulib", it's really in 
top/bootstrap.


More important, build-aux/bootstrap exists separately from top/bootstrap 
only to support packages that don't have bootstrap-funclib.sh; and yet 
build-aux/bootstrap invokes autopull.sh and autogen.sh, both of which 
require bootstrap-funclib.sh. One way to work around this problem would 
be to remove build-aux/bootstrap (and the machinery that generates it), 
thus requiring all downstream projects to add autogen.sh, autopull.sh, 
and bootstrap-funclib.sh. Another possibility would be to rename 
top/bootstrap to top/bootstrap.sh (so there aren't two confusing files 
with the same basename), and to fix top/gen-bootstrap.sed so that it 
inlines autopull.sh and autogen.sh as well as inlining 
bootstrap-funclib.sh. Or perhaps you can think of a better fix (maybe 
giving ./bootstrap --pull and --gen options instead of having separate 
scripts?).


Another idea: if we're going to break 'bootstrap' up into pieces, 
perhaps it'd be better for GNU tar to have these new files in a new 
directory, i.e., boot/autogen.sh, boot/autopull.sh, 
boot/bootstrap-funclib.sh, instead of autogen.sh, autopull.sh, 
bootstrap-funclib.sh, so as to keep the top level simple. Is that easy 
to arrange?From 5c4f8cadbd362497dc3c3136566589557ce00f1b Mon Sep 17 00:00:00 2001
From: Paul Eggert 
Date: Sun, 24 Jul 2022 11:44:12 -0700
Subject: [PATCH] Adjust to Gnulib bootstrap revamp

* autogen.sh, autopull.sh, bootstrap-funclib.sh:
New files, copied from gnulib/top.
* bootstrap: Copy from gnulib/top/bootstrap (as opposed
to copying from gnulib/build-aux/bootstrap, as we used to).
* bootstrap.conf (bootstrap_post_pull_hook)
(bootstrap_post_import_hook): New functions.
Move commands into these functions as needed.
---
 NEWS |5 +-
 README-alpha |4 +
 README-hacking   |4 +
 autogen.sh   |  486 ++
 autopull.sh  |  273 +++
 bootstrap| 1112 --
 bootstrap-funclib.sh |  594 ++
 bootstrap.conf   |   61 ++-
 8 files changed, 1501 insertions(+), 1038 deletions(-)
 create mode 100755 autogen.sh
 create mode 100755 autopull.sh
 create mode 100644 bootstrap-funclib.sh

diff --git a/NEWS b/NEWS
index 94529ab1..1378dbe5 100644
--- a/NEWS
+++ b/NEWS
@@ -1,4 +1,4 @@
-GNU tar NEWS - User visible changes. 2022-06-10
+GNU tar NEWS - User visible changes. 2022-07-24
 Please send GNU tar bug reports to 
 
 version 1.34.90 (git)
@@ -9,6 +9,9 @@ version 1.34.90 (git)
 * Leave the devmajor and devminor fields empty (rather than zero) for
   non-special files, as this is more compatible with traditional tar.
 
+* New build scripts ./autopull.sh and ./autogen.sh, to separate
+  fetching from autogenerating source files.  See README-hacking.
+
 * Bug fixes
 
 ** Warn "file changed as we read it" less often.
diff --git a/README-alpha b/README-alpha
index a27c66a8..baf982a7 100644
--- a/README-alpha
+++ b/README-alpha
@@ -33,6 +33,10 @@ Bootstrap reads its configuration from file bootstrap.conf located on the
 top of tar source tree.  Several options are provided that modify its
 behavior.  Run 'bootstrap --help' for a list.
 
+To only fetch auxiliary files from the network, run ./autopull.sh.
+To only generate files such as 'configure', without accessing the
+network, run ./autogen.sh.  The 'bootstrap' script does both.
+
 
 
 Copyright 2001-2022 Free Software Foundation, Inc.
diff --git a/README-hacking b/README-hacking
index 179df387..4c484a42 100644
--- a/README-hacking
+++ b/README-hacking
@@ -40,6 +40,10 @@ intend to hack on GNU tar, you might need to run it again later.
 There are lots of options that you may find useful in this case.
 See 'bootstrap --help' for a detailed list.
 
+To only fetch auxiliary files from the network, run ./autopull.sh.
+To only generate files such as 'configure', without accessing the
+network, run ./autogen.sh.  The 'bootstrap' script does both.
+
 
 * Copyright information
 
diff --git a/autogen.sh b/autogen.sh
new file mode 100755
index ..f245fb11
--- /dev/null
+++ b/autogen.sh
@@ -0,0 +1,486 @@
+#!/bin/sh
+# Convenience script for regenerating all autogeneratable files that are
+# omitted from the version control repository. In particular, this script
+# also regenerates all aclocal.m4, config.h.in, Makefile.in, configure files
+# with new versions of autoconf or automake.
+
+# Copyright (C) 2003-2022 Free Software Foundation, Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License 

Re: split bootstrap in two phases, GNU libidn

[Bruno Haible]

Hi dear GNU libidn developers (Simon et al.),

The 'bootstrap' script from Gnulib has been reorganized to work in two phases:
  (1) Fetch auxiliary files that are not in the git checkout.
  This is the part that requires network access and that has
  supply-chain concerns.
  (2) Generate files such as configure, config.h, Makefile.in etc.
  This includes running gnulib-tool.

See  for
details.

For GNU libidn some adjustment of the bootstrap.conf is needed. The 
bootstrap.conf
is meant to contain configuration settings, that is, define variables and
functions (hooks with special meanings). Instead, libidn/bootstrap.conf does 
some
actions directly. This will not work satisfactorily with the new two-phase
bootstrap. What is needed, is to move these actions into hooks:
  - Actions that may require network access belong in the
bootstrap_post_pull_hook function.
  - Actions that merely regenerate files (without network access) usually
belong in the bootstrap_post_import_hook.

Namely, these statements

printf "gdoc_MANS =\ngdoc_TEXINFOS =\n" > doc/Makefile.gdoc

and

GTKDOCIZE=$(which gtkdocize 2>/dev/null)
if test $? -ne 0; then
  echo "No gtk-doc support found. You can't build the docs."
  # rm because gtk-doc.make might be a link to a protected file
  rm -f gtk-doc.make 2>/dev/null
  echo "EXTRA_DIST =" >gtk-doc.make
  echo "CLEANFILES =" >>gtk-doc.make
  GTKDOCIZE=""
else
  mkdir -p lib/m4
  $GTKDOCIZE
fi

belong in the bootstrap_post_import_hook.

These adjustments will be needed the next time you use the '--bootstrap-sync'
option.

Bruno

Re: split bootstrap in two phases, GNU poke

[Bruno Haible]

Hi dear GNU poke developers (José et al.),

The 'bootstrap' script from Gnulib has been reorganized to work in two phases:
  (1) Fetch auxiliary files that are not in the git checkout.
  This is the part that requires network access and that has
  supply-chain concerns.
  (2) Generate files such as configure, config.h, Makefile.in etc.
  This includes running gnulib-tool.

See  for
details.

For GNU poke some adjustment of the bootstrap.conf is needed: The code
snippet

  if $use_git && test -d .git && check_exists git; then
echo 'Updating the Jitter submodule'
git submodule update --init -- ./jitter
JITTER_SRCDIR=jitter
  fi

needs to be moved from the bootstrap_post_import_hook to the
bootstrap_post_pull_hook.

This adjustment will be needed the next time you use the '--bootstrap-sync'
option.

Bruno

Re: split bootstrap in two phases, GNU wget2

[Bruno Haible]

Hi dear GNU wget developers (Tim et al.),

The 'bootstrap' script from Gnulib has been reorganized to work in two phases:
  (1) Fetch auxiliary files that are not in the git checkout.
  This is the part that requires network access and that has
  supply-chain concerns.
  (2) Generate files such as configure, config.h, Makefile.in etc.
  This includes running gnulib-tool.

See  for
details.

For GNU wget2 some adjustment of the bootstrap.conf is needed: The clone_wiki
invocation, which makes network accesses, needs to be moved from the
bootstrap_post_import_hook to the bootstrap_post_pull_hook.

This adjustment will be needed the next time you use the '--bootstrap-sync'
option.

Bruno

Re: split bootstrap in two phases, GNU tar

[Bruno Haible]

Hi dear GNU tar developers (Sergey et al.),

The 'bootstrap' script from Gnulib has been reorganized to work in two phases:
  (1) Fetch auxiliary files that are not in the git checkout.
  This is the part that requires network access and that has
  supply-chain concerns.
  (2) Generate files such as configure, config.h, Makefile.in etc.
  This includes running gnulib-tool.

See  for
details.

For GNU tar some adjustment of the bootstrap.conf is needed. The bootstrap.conf
is meant to contain configuration settings, that is, define variables and
functions (hooks with special meanings). Instead, tar/bootstrap.conf does the
actions directly. This will not work satisfactorily with the new two-phase
bootstrap. What is needed, is to move these actions into hooks:
  - Actions that may require network access belong in the
bootstrap_post_pull_hook function.
  - Actions that merely regenerate files (without network access) usually
belong in the bootstrap_post_import_hook.

These adjustments will be needed the next time you use the '--bootstrap-sync'
option.

Bruno

split bootstrap in two phases

[Bruno Haible]

Hi all 'bootstrap' users,

Over the last few years, it has become more and more clear that bootstrap
does two things:
  (1) Fetch auxiliary files that are not in the git checkout.
  This is the part that requires network access and that has
  supply-chain concerns.
  (2) Generate files such as configure, config.h, Makefile.in etc.
  This includes running gnulib-tool.

Recent discussion in gnu-prog-discuss has shown that making the separation
into two phases (1) and (2) explicit will have several benefits:

  * For reproducible builds, it is necessary to be execute the second
phase without the first one.

  * For people who have local modifications in dependency packages
(e.g. in gnulib), or for distros who want to apply patches to .m4 files,
it is important to have these phases separated.

  * The second phase is a way for people to regenerate files in a
supported way. So far, too many users have been running 'autoreconf -fvi',
which many GNU packages don't support.

  * For people who develop without an internet connection, the ability
to regenerate files explicitly is nice (even when, for some of the files,
Makefile rules would force a regeneration on the fly).

This patch does the separation.
The first phase is a script 'autopull.sh'.
The second phase is a script 'autogen.sh'.
Some packages may use the ones that are "generic" and customized
through a bootstrap.conf file. Other packages (e.g. GNU gettext) will
provide hand-written 'autopull.sh' and 'autogen.sh'.

The names of these scripts end with '.sh' in order to make them easily
editable in editors that recognize the file type from the suffix.

This patch is backward-compatible: A script named 'bootstrap' continues
to exist. But it will be useful for packages to document in their
INSTALL / HACKING / README-hacking / INSTALL.REPO files the sequence
of two commands:
  ./autopull.sh
  ./autogen.sh
because — as mentioned above — in some situations it is not necessary to
run ./autopull.sh.

I've tested this patch with GNU coreutils and GNU poke. The invocation
'./bootstrap --bootstrap-sync' can be used to upgrade from the older
bootstrap to the newer one with its companions autopull.sh and autogen.sh.


2022-07-24  Bruno Haible  

Split bootstrap into autopull.sh and autogen.sh.
* top/bootstrap-funclib.sh: New file, based on build-aux/bootstrap.
* top/autopull.sh: New file, based on build-aux/bootstrap.
* top/autogen.sh: New file, based on build-aux/bootstrap.
* top/bootstrap: New file, based on build-aux/bootstrap.
* top/gen-bootstrap.sed: New file.
* Makefile (build-aux/bootstrap): New rule.
(regen): Depend on it.
* build-aux/bootstrap: Regenerated using "make build-aux/bootstrap".

>From 096abe4908834d96a7225218c18d08425aa48b90 Mon Sep 17 00:00:00 2001
From: Bruno Haible 
Date: Sun, 24 Jul 2022 16:24:37 +0200
Subject: [PATCH] Split bootstrap into autopull.sh and autogen.sh.

* top/bootstrap-funclib.sh: New file, based on build-aux/bootstrap.
* top/autopull.sh: New file, based on build-aux/bootstrap.
* top/autogen.sh: New file, based on build-aux/bootstrap.
* top/bootstrap: New file, based on build-aux/bootstrap.
* top/gen-bootstrap.sed: New file.
* Makefile (build-aux/bootstrap): New rule.
(regen): Depend on it.
* build-aux/bootstrap: Regenerated using "make build-aux/bootstrap".
---
 ChangeLog|  12 +
 Makefile |   8 +-
 build-aux/bootstrap  | 967 +--
 top/autogen.sh   | 486 
 top/autopull.sh  | 273 +++
 top/bootstrap| 219 +
 top/bootstrap-funclib.sh | 594 
 top/gen-bootstrap.sed|   4 +
 8 files changed, 1918 insertions(+), 645 deletions(-)
 create mode 100755 top/autogen.sh
 create mode 100755 top/autopull.sh
 create mode 100755 top/bootstrap
 create mode 100644 top/bootstrap-funclib.sh
 create mode 100644 top/gen-bootstrap.sed

diff --git a/ChangeLog b/ChangeLog
index 46819bbfec..40c097a262 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,15 @@
+2022-07-24  Bruno Haible  
+
+	Split bootstrap into autopull.sh and autogen.sh.
+	* top/bootstrap-funclib.sh: New file, based on build-aux/bootstrap.
+	* top/autopull.sh: New file, based on build-aux/bootstrap.
+	* top/autogen.sh: New file, based on build-aux/bootstrap.
+	* top/bootstrap: New file, based on build-aux/bootstrap.
+	* top/gen-bootstrap.sed: New file.
+	* Makefile (build-aux/bootstrap): New rule.
+	(regen): Depend on it.
+	* build-aux/bootstrap: Regenerated using "make build-aux/bootstrap".
+
 2022-07-24  Bruno Haible  
 
 	bootstrap: Obey another environment variable GNULIB_REFDIR.
diff --git a/Makefile b/Makefile
index 85362c8c2c..2eaa15b6bb 100644
--- a/Makefile
+++ b/Makefile
@@ -142,7 +142,13 @@ sc_check_copyright:
 	@./check-copyright
 
 # Regenerate some files that are stored in the repository.
-regen: MODULES.html