[PATCH] maintainer-makefile: Silence announce-gen error with GNULIB_REVISION.
On running 'make release' I got this error message:
GEN release-prep
fatal: No names found, cannot describe anything.
make[1]: Entering directory '/home/jas/src/inetutils'
The error message is harmless since the code already handled this
situation, but the error message should be silenced since it looks
pretty alarming and the alternative code path using git rev-parse work
correctly as intended.
/Simon
From 0c52a761fbe563f2aa6731fbb18b0572005bc548 Mon Sep 17 00:00:00 2001
From: Simon Josefsson
Date: Sun, 12 May 2024 17:07:30 +0200
Subject: [PATCH] maintainer-makefile: Silence announce-gen error with
GNULIB_REVISION.
* top/maint.mk (gnulib-version): Silence git describe on failure.
---
ChangeLog| 5 +
top/maint.mk | 2 +-
2 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/ChangeLog b/ChangeLog
index 2e2311e7b2..b6aa21d7f7 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,8 @@
+2024-05-12 Simon Josefsson
+
+ maintainer-makefile: Silence announce-gen error with GNULIB_REVISION.
+ * top/maint.mk (gnulib-version): Silence git describe on failure.
+
2024-05-12 Bruno Haible
execinfo: Document known bugs.
diff --git a/top/maint.mk b/top/maint.mk
index 32228f4366..ecd8971900 100644
--- a/top/maint.mk
+++ b/top/maint.mk
@@ -1502,7 +1502,7 @@ vc-diff-check:
rel-files = $(DIST_ARCHIVES)
gnulib-version = $$(cd $(gnulib_dir)\
-&& { git describe || git rev-parse --short=10 HEAD; } )
+&& { git describe 2> /dev/null || git rev-parse --short=10 HEAD; } )
bootstrap-tools ?= autoconf,automake,gnulib
gpgv = $$(gpgv2 --version >/dev/null && echo gpgv2 || echo gpgv)
--
2.41.0
signature.asc
Description: PGP signature
Re: GNULIB_REVISION
On 4/25/24 10:43, Simon Josefsson wrote: https://gitlab.com/gsasl/inetutils/-/jobs/6706396721 That's an malloc failure on the server side. The savannah admins should be told about the issue soon after it happens. Routine clones of GNU projects shouldn't fail like that. (It's never happened to me, but then I don't clone from Savannah all that often - maybe Savannah is giving less server memory to clients that seem to hog?) Btw, using --depth 1 is incompatible with gnulib's git-version-gen We could fix that by not requiring git-version-gen for the Gnulib submodule. git-version-gen is pretty useless for that anyway, as for Gnulib it currently outputs a string like "0.1.7513-648ae" which is not much more useful than the commit ID "648aeb575db7af9ddd51cf17d1b56ee91e9ef3c5". Isn't the real problem that we don't put (for example) gzip's own commit ID into the coreutils tarball? If we did that, Gnulib's commit ID would come for free, since it can be derived from gzip's commit ID. I suppose you meant s/coreutils/gzip/ Yes, sorry. SHA1 git commits aren't long-term stable either, since SHA1 is broken As you say, they're good enough for now. And anyway I would think SHA1 is good enough longer term unless an adversary infects your Git repository (and in that case you probably have bigger problems...).
Re: GNULIB_REVISION
On 4/25/24 10:00 AM, Paul Eggert wrote: > Is there some way to cajole 'git clone' into using less memory, with a > '--depth 1' or similar options? Cloning shallowly would clone Gnulib a > lot faster, if you're cloning from a remote repository. Maybe '--single-branch' would help too. If space is really desperate the snapshots on gitweb are pretty small. https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=commit;h=648aeb575db7af9ddd51cf17d1b56ee91e9ef3c5 7.6 MB for me vs. my full clone $ du -s --human-readable .git 114M.git Collin
Re: GNULIB_REVISION
Paul Eggert writes: > You raise several good points. A couple of quick reaction: > > On 2024-04-25 09:26, Simon Josefsson via Gnulib discussion list wrote: > >> - the gnulib git submodule is huge. Not rarely I get out of memory >>errors during 'git clone' in CI/CD jobs. > > First I've heard of this problem (other than with Git LFS, which > Gnulib doesn't use). What part of the clone operation fails? Is this > server-side RAM, or client-side RAM, or something else? How much > memory does 'git clone' require now for Gnulib? > > Is there some way to cajole 'git clone' into using less memory, with a > '--depth 1' or similar options? Cloning shallowly would clone Gnulib a > lot faster, if you're cloning from a remote repository. It only happens once in a while, for example: https://gitlab.com/gsasl/inetutils/-/jobs/6706396721 This is gitlab's own git submodule checkout system working, and it is using --depth 150 which shouldn't be too heavy, so it not even getting to ./bootstrap's git clone. Btw, using --depth 1 is incompatible with gnulib's git-version-gen: it will not find a suitable version number for the build. Even gitlab's default depth of 50 (or if it was 100, can't remember) is often not enough if you have >50 commits since the last release. This cause problems when building from 'git archive' tarballs. >> - we don't offer any way for people receiving tarballs to learn which >>gnulib git commit was used > > Isn't the real problem that we don't put (for example) gzip's own > commit ID into the coreutils tarball? If we did that, Gnulib's commit > ID would come for free, since it can be derived from gzip's commit ID. I suppose you meant s/coreutils/gzip/, otherwise I don't follow? Yes that is a good idea! The git commit of the project should be part of the announce-gen output. The git tag name could be mentioned too. Tags are not long-term stable since they can be moved later on, so I think the full git commit id should be mentioned too. Current SHA1 git commits aren't long-term stable either, since SHA1 is broken, but at least this approach is the best we can do right now and when we move to SHA256 git things will be better. /Simon signature.asc Description: PGP signature
Re: GNULIB_REVISION
You raise several good points. A couple of quick reaction: On 2024-04-25 09:26, Simon Josefsson via Gnulib discussion list wrote: - the gnulib git submodule is huge. Not rarely I get out of memory errors during 'git clone' in CI/CD jobs. First I've heard of this problem (other than with Git LFS, which Gnulib doesn't use). What part of the clone operation fails? Is this server-side RAM, or client-side RAM, or something else? How much memory does 'git clone' require now for Gnulib? Is there some way to cajole 'git clone' into using less memory, with a '--depth 1' or similar options? Cloning shallowly would clone Gnulib a lot faster, if you're cloning from a remote repository. - we don't offer any way for people receiving tarballs to learn which gnulib git commit was used Isn't the real problem that we don't put (for example) gzip's own commit ID into the coreutils tarball? If we did that, Gnulib's commit ID would come for free, since it can be derived from gzip's commit ID.
Re: GNULIB_REVISION
Bruno Haible writes: > Hi Simon, > >> you can ... via >> GNULIB_REVISION pick out exactly the gnulib git revision that libpaper >> needs. ... >> [1] >> https://blog.josefsson.org/2024/04/13/reproducible-and-minimal-source-only-tarballs/ >> [2] https://salsa.debian.org/auth-team/libntlm/-/tree/master/debian > > I see GNULIB_REVISION as an obsolete alternative to git submodules, and > would therefore discourage rather than propagate its use. I think it will be challenging for gnulib to insists on always being used as a git submodule, and I would prefer if we continue support multiple ways of working. Personally I have been migrating towards gnulib git submodules because most other projects use gnulib like that, but I've never really felt comfortable with them. Some of the concerns I have: - git submodules leads to -- in my subjective opinion -- complexity which leads to a worse user experience for developers. I have learned to work with git submodules over the years, but it was a hurdle that I don't want to force on everyone. - the gnulib git submodule is huge. Not rarely I get out of memory errors during 'git clone' in CI/CD jobs. I can restart the jobs manually, but this indicate that there is a resource drain here. For a tiny project like libntlm the imbalance if the small project code and large gnulib is troubling. - often CI/CD platforms have different ways of working with git submodules which adds complexity which leads to bugs. Allowing maintainers to decide if they want to work with git submodules or not seems like a good thing. - we don't offer any way for people receiving tarballs to learn which gnulib git commit was used (you noticed this too below) but with a GNULIB_REVISION approach this is part of the tarball, just like any other versioned dependency on autoconf, automake etc - I think gnulib could be regarded as any other external dependency, just like autoconf, automake, libtool etc that also generate files in my build tree during bootstrapping. I don't put autoconf as a git submodule, why should I put gnulib as one? Granted, these concerns are a bit vague and subjective. > Currently libntlm has this in its bootstrap.conf: > > GNULIB_REVISION=dfb71172a46ef41f8cf8ab7ca529c1dd3097a41d > > and GNU make has this: > > GNULIB_REVISION=stable-202307 Interesting. This suggests the GNULIB_REVISION approach isn't the entire solution either. I think it is useful to record the gnulib git commit used to prepare a tarball, and have that git commit id be part of the shipped tarball, and stored inside the git repository. The first use above achieve this, but the second one doesn't (branches/tags are moving targets). If I download the gzip tarball I can't find anywhere what gnulib commit was used for bootstrapping. It is quite cumbersome to verify that the tarball didn't contain any modified gnulib code. This is even harder when projects INTENTIONALLY modify gnulib code compared to what's in gnulib git, which coreutils and several others projects does through gnulib *.diff/*.patch files. Ultimately, I think there is an important use-case to build projects directly from source code without having tarballs with pre-generated files that are not reproduced by the user. > The differences between both approaches are: > > - GNULIB_REVISION works only with the 'bootstrap' program. The submodules > approach works also without 'bootstrap'. What use case are you thinking of? The gnulib git commit information consumers that I can think of are gnulib-aware. > - For GNULIB_REVISION, the user is on their own regarding tooling, aside > from 'bootstrap'. In the submodules approach, the 'git' suite provides > the tooling, and many developers are familiar with it. Yes, but developers also like flexibility, and in some situations I think the git approach is not the best way of working. > - .tar.gz files created by the gitweb "snapshot" link, by the cgit "refs > > Download" section, or the GitHub "Download ZIP" button contain an empty > directory in place of the submodule, and no information about the > revision. > Whereas they contain the file with the GNULIB_REVISION assignment. Indeed, this was the main challenge for me. That is critical information for anyone who wants to avoid touching tarballs with pre-generated content. >> I should write a post to debian-devel describing this pattern on >> how to use gnulib in Debian packages > > It feels wrong to me if, in order to get meta-information about required > dependencies of a package, Debian tools grep a particular file for a specific > string. This approach is simply too limited. Meta-information about dependencies are normally always hand-curated in Debian (the Build-Depends: header). The simplest
Re: GNULIB_REVISION
Hi Simon, > you can ... via > GNULIB_REVISION pick out exactly the gnulib git revision that libpaper > needs. ... > [1] > https://blog.josefsson.org/2024/04/13/reproducible-and-minimal-source-only-tarballs/ > [2] https://salsa.debian.org/auth-team/libntlm/-/tree/master/debian I see GNULIB_REVISION as an obsolete alternative to git submodules, and would therefore discourage rather than propagate its use. Currently libntlm has this in its bootstrap.conf: GNULIB_REVISION=dfb71172a46ef41f8cf8ab7ca529c1dd3097a41d and GNU make has this: GNULIB_REVISION=stable-202307 Both can be done with git submodules. Git submodules do support branches [1], and the command 'git submodule update --remote gnulib' updates the submodule while staying on the indicated branch. When using submodules, the target revision is stored in a versionable way, that both gitweb and cgit can show appropriately [2][3]. If a branch is used, it is stored in the .gitmodules file. The differences between both approaches are: - GNULIB_REVISION works only with the 'bootstrap' program. The submodules approach works also without 'bootstrap'. - For GNULIB_REVISION, the user is on their own regarding tooling, aside from 'bootstrap'. In the submodules approach, the 'git' suite provides the tooling, and many developers are familiar with it. - .tar.gz files created by the gitweb "snapshot" link, by the cgit "refs > Download" section, or the GitHub "Download ZIP" button contain an empty directory in place of the submodule, and no information about the revision. Whereas they contain the file with the GNULIB_REVISION assignment. > I should write a post to debian-devel describing this pattern on > how to use gnulib in Debian packages It feels wrong to me if, in order to get meta-information about required dependencies of a package, Debian tools grep a particular file for a specific string. This approach is simply too limited. The correct way, IMO, would be that 'git' provides this meta-information, either embedded in the .tar.gz generated by the web tooling, or in a separate .tar.gz. AFAICT, 'git' currently does not have this ability. Therefore we need to approach the 'git' team, in order to find a solution that scales across the whole set of software package — not specific to gnulib and not specific to 'bootstrap'. Bruno [1] https://stackoverflow.com/questions/1777854/how-can-i-specify-a-branch-tag-when-adding-a-git-submodule [2] https://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=tree [3] https://git.savannah.gnu.org/cgit/coreutils.git/tree/
Re: ./bootstrap --gnulib-srcdir and GNULIB_REVISION
Simon Josefsson wrote: > Alas, GNULIB_REVISION is not documented in doc/gnulib.texi. Yes, that's because the more modern way is to use a git submodule instead. This is documented in https://www.gnu.org/software/gnulib/manual/html_node/VCS-Issues.html https://www.gnu.org/software/gnulib/manual/html_node/Developer-tools.html > My impression is that the ./bootstrap script has gained a lot of > complexity that has evolved organically. For some projects this > complexity is unwanted -- e.g., guile-gnutls is used early in the > bootstrapping of Guix and we eventually resolved to putting all needed > gnulib files in .git and used a naive ./bootstrap script: > > https://gitlab.com/gnutls/guile/-/blob/master/bootstrap?ref_type=heads Sure. Especially when there are several gnulib-tool invocations, the bootstrap architecture and configuration file are less than adequate. [1] In this case, hand-written autopull.sh and autogen.sh files fulfil the same need and are easier to maintain. Bruno [1] https://git.savannah.gnu.org/gitweb/?p=poke.git;a=blob;f=bootstrap.conf
Re: ./bootstrap --gnulib-srcdir and GNULIB_REVISION
Hi Simon, > Bug #2: ./bootstrap writes to the path indicated by --gnulib-srcdir with > the 'git checkout' command, and leaves the --gnulib-srcdir path at that > commit after ./bootstrap is finished. This happens to work in my > example since I pointed it to a writable work tree, but I think altering > that path is unexpected and not documented. Imagine pointing this to a > system-wide gnulib .git store like --gnulib-srcdir=/usr/share/src/gnulib > or similar read-only place. Or imagine multiple ./bootstrap running at > the same time for different projects, both pointing to the same gnulib > .git work tree. I think the path indicated by --gnulib-srcdir should be > read-only. If --gnulib-srcdir is given and GNULIB_REVISION is set in bootstrap.conf: * The current behaviour (now independent of submodule or not) is that - Without --no-git, the GNULIB_REVISION is checked out in $GNULIB_SRCDIR. If this directory has local modifications, git reports an error, and 'bootstrap' fails. If it does not have local modifications, the $GNULIB_SRCDIR is left in that modified state. - With --no-git, the contents of $GNULIB_SRCDIR is left unmodified. So, the user has the possibility to force the behaviour they desire. * But I agree that it would be useful to change "If the environment variable GNULIB_SRCDIR is set, then sources are fetched from that local directory. If it is a git repository and the configuration variable GNULIB_REVISION is set in bootstrap.conf, then that revision is checked out." to "If the environment variable GNULIB_SRCDIR is set, then sources are fetched from that local directory. If the configuration variable GNULIB_REVISION is set in bootstrap.conf, it is ignored." for 3 reasons: - As a simple-to-remember rule, $GNULIB_SRCDIR would always be left unmodified. - The purpose of GNULIB_REVISION is to simulate a submodule without actually having a git submodule. Only very few packages use this: GNU make, man-db, libpipeline, binfmt-support. Probably they wanted to avoid submodules because of the extra learning curve associated with git submodules. The submodule is only considered in the case that GNULIB_SRCDIR is not set. Therefore, it would be consistent to consider GNULIB_REVISION also only when GNULIB_SRCDIR is not set. - The recipe for using a modified gnulib ./bootstrap --no-git --gnulib-srcdir=... would be simplified to ./bootstrap --gnulib-srcdir=... And what would a user do who was using GNULIB_SRCDIR and _wants_ the GNULIB_REVISION to be respected? They would set GNULIB_REFDIR instead of GNULIB_SRCDIR. (This is precisely the point of GNULIB_REFDIR!) Bruno
Re: ./bootstrap --gnulib-srcdir and GNULIB_REVISION
Hi Simon, After fixing the --help output, I'm changing the 'bootstrap' logic to follow this documentation. Two patches: * 0001 simplifies the logic by removing fallback code for git versions < 1.6.4. Version 1.6.4 was released in 2009, that is, 15 years ago. When encountering such an old version, now an error message is output. * 0002 implements prepare_GNULIB_SRCDIR as documented. The real change is that instead of having a single occurrence of git checkout "$GNULIB_REVISION" at the end, we have one in each possible case. Which allows to tailor when to invoke it. > Bug #1: it seems GNULIB_REVISION in bootstrap.conf has no effect, and > this code is the reason (quoting bootstrap-funclib.sh): > > # XXX Should this be done if $use_git is false? > if test -d "$GNULIB_SRCDIR"/.git && test -n "$GNULIB_REVISION" \ > && ! git_modules_config submodule.gnulib.url >/dev/null; then > (cd "$GNULIB_SRCDIR" && git checkout "$GNULIB_REVISION") || cleanup_gnulib > fi > > The reason is that the tarball has .gitmodules looking like this: > > [submodule "gnulib"] > path = gnulib > url = https://git.savannah.gnu.org/git/gnulib.git > > Which trigger the '! git_modules_config submodules.gnulib.url'. This is fixed as part of 0002. Bruno >From ac8c4111ce570e140c819f231493fa573d3b0904 Mon Sep 17 00:00:00 2001 From: Bruno Haible Date: Sun, 14 Apr 2024 00:08:50 +0200 Subject: [PATCH 1/2] bootstrap: Simplify git submodule initialization. * top/bootstrap-funclib.sh (prepare_GNULIB_SRCDIR): Err out if git is older than version 1.6.4. Remove fallback code for older versions. * build-aux/bootstrap: Regenerated. --- ChangeLog| 7 +++ build-aux/bootstrap | 27 --- top/bootstrap-funclib.sh | 27 --- 3 files changed, 23 insertions(+), 38 deletions(-) diff --git a/ChangeLog b/ChangeLog index c9349f22ac..a6878807ba 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,10 @@ +2024-04-13 Bruno Haible + + bootstrap: Simplify git submodule initialization. + * top/bootstrap-funclib.sh (prepare_GNULIB_SRCDIR): Err out if git is + older than version 1.6.4. Remove fallback code for older versions. + * build-aux/bootstrap: Regenerated. + 2024-04-13 Collin Funk Improve 'git diff' of Python files. diff --git a/build-aux/bootstrap b/build-aux/bootstrap index ef9161d75e..70e567b2ad 100755 --- a/build-aux/bootstrap +++ b/build-aux/bootstrap @@ -500,6 +500,12 @@ prepare_GNULIB_SRCDIR () || die "Error: --gnulib-srcdir or \$GNULIB_SRCDIR is specified," \ "but does not contain gnulib-tool" elif $use_git; then +if git submodule -h | grep -- --reference > /dev/null; then + : +else + die "git version is too old, git >= 1.6.4 is required" +fi + gnulib_path=$(git_modules_config submodule.gnulib.path) test -z "$gnulib_path" && gnulib_path=gnulib @@ -510,25 +516,8 @@ prepare_GNULIB_SRCDIR () && git_modules_config submodule.gnulib.url >/dev/null; then # Use GNULIB_REFDIR as a reference. echo "$0: getting gnulib files..." - if git submodule -h|grep -- --reference > /dev/null; then -# Prefer the one-liner available in git 1.6.4 or newer. -git submodule update --init --reference "$GNULIB_REFDIR" \ - "$gnulib_path" || exit $? - else -# This fallback allows at least git 1.5.5. -if test -f "$gnulib_path"/gnulib-tool; then - # Since file already exists, assume submodule init already complete. - git submodule update -- "$gnulib_path" || exit $? -else - # Older git can't clone into an empty directory. - rmdir "$gnulib_path" 2>/dev/null - git clone --reference "$GNULIB_REFDIR" \ -"$(git_modules_config submodule.gnulib.url)" "$gnulib_path" \ -&& git submodule init -- "$gnulib_path" \ -&& git submodule update -- "$gnulib_path" \ -|| exit $? -fi - fi + git submodule update --init --reference "$GNULIB_REFDIR" \ +"$gnulib_path" || exit $? else # GNULIB_REFDIR is not set or not usable. Ignore it. if git_modules_config submodule.gnulib.url >/dev/null; then diff --git a/top/bootstrap-funclib.sh b/top/bootstrap-funclib.sh index 64896ff4e5..e0d3ab900d 100644 --- a/top/bootstrap-funclib.sh +++ b/top/bootstrap-funclib.sh @@ -463,6 +463,12 @@ prepare_GNULIB_SRCDIR () || die "Error: --gnulib-srcdir or \$GNULIB_SRCDIR is specified," \ "but does not contain gnulib-tool" elif $use_git;
Re: ./bootstrap --gnulib-srcdir and GNULIB_REVISION
Simon Josefsson wrote:
> My reaction was initially exactly the same as yours, until I found this
> piece of --help documentation, which actually is the first (and
> presumably highest priority) rule:
>
> * If the environment variable GNULIB_SRCDIR is set (either as an
>environment variable or via the --gnulib-srcdir option), then sources
>are fetched from that local directory. If it is a git repository and
>the configuration variable GNULIB_REVISION is set in bootstrap.conf,
>then that revision is checked out.
This piece of documentation is outdated. It also does not mention the
role of GNULIB_REFDIR. In the commit 2122284380cc0d1b3b6f11d92c04652616da79c7
<https://lists.gnu.org/archive/html/bug-gnulib/2022-07/msg00052.html>
I updated meant to update the --help string:
bootstrap: Obey another environment variable GNULIB_REFDIR.
* build-aux/bootstrap (usage): Document option --gnulib-refdir and
environment variable GNULIB_REFDIR.
but apparently botched it (maybe because I was developing this patch
concurrently with the next one, which created autopull and autogen).
As a first step, I'm correcting the --help strings, to match what I meant
to implement on 2022-07-24:
2024-04-13 Bruno Haible
bootstrap: Fix --help messages.
Reported by Simon Josefsson in
<https://lists.gnu.org/archive/html/bug-gnulib/2024-04/msg00162.html>.
* top/bootstrap (usage): Fix the description of GNULIB_SRCDIR. Document
GNULIB_REFDIR. Use full-sentences style. Tweak wording.
* top/bootstrap-funclib.sh (autopull_usage): Clarify the phase. Don't
mention nonexistent options --gnulib-srcdir and --gnulib-refdir. Use
full-sentences style.
(autogen_usage): Clarify the phase. Use full-sentences style.
* build-aux/bootstrap: Regenerated.
diff --git a/build-aux/bootstrap b/build-aux/bootstrap
index 3173829dab..ef9161d75e 100755
--- a/build-aux/bootstrap
+++ b/build-aux/bootstrap
@@ -3,7 +3,7 @@
# Bootstrap this package from checked-out sources.
-scriptversion=2023-12-10.18; # UTC
+scriptversion=2024-04-13.15; # UTC
# Copyright (C) 2003-2024 Free Software Foundation, Inc.
#
@@ -37,7 +37,7 @@ medir=`dirname "$me"`
# A library of shell functions for autopull.sh, autogen.sh, and bootstrap.
-scriptlibversion=2023-12-10.18; # UTC
+scriptlibversion=2024-04-13.15; # UTC
# Copyright (C) 2003-2024 Free Software Foundation, Inc.
#
@@ -647,7 +647,8 @@ fi
autopull_usage() {
cat <
Re: ./bootstrap --gnulib-srcdir and GNULIB_REVISION
Simon Josefsson via Gnulib discussion list writes: > My reaction was initially exactly the same as yours, until I found this > piece of --help documentation, which actually is the first (and > presumably highest priority) rule: > > * If the environment variable GNULIB_SRCDIR is set (either as an >environment variable or via the --gnulib-srcdir option), then sources >are fetched from that local directory. If it is a git repository and >the configuration variable GNULIB_REVISION is set in bootstrap.conf, >then that revision is checked out. > > So I think this combination is intended to be supported, it is just not > working when a .gitmodules file is present in $CWD -- something that is > not mentioned as a requirement. Sorry, I found this piece of --help contradict the last sentence I wrote: If you maintain a package and want to pin a particular revision of the Gnulib sources that has been tested with your package, then there are two possible approaches: either configure a 'gnulib' submodule with the appropriate revision, or set GNULIB_REVISION (and if necessary GNULIB_URL) in bootstrap.conf. So it seems having a git submodule for gnulib and using GNULIB_REVISION at the same time is not supported. FWIW, I'm going to experiment in libntlm and use GNULIB_REVISION in bootstrap.conf instead of a git submodule, to allow a downloaded tarball of git HEAD to be a supported way of building the package from source. This hasn't worked historically for reasons discussed in this thread, but given the xz backdoor I think there is value in supporting this approach. /Simon signature.asc Description: PGP signature
Re: ./bootstrap --gnulib-srcdir and GNULIB_REVISION
Bruno Haible writes: > Hi Simon, > >> Bug #2: ./bootstrap writes to the path indicated by --gnulib-srcdir with >> the 'git checkout' command, and leaves the --gnulib-srcdir path at that >> commit after ./bootstrap is finished. This happens to work in my >> example since I pointed it to a writable work tree, but I think altering >> that path is unexpected and not documented. Imagine pointing this to a >> system-wide gnulib .git store like --gnulib-srcdir=/usr/share/src/gnulib >> or similar read-only place. Or imagine multiple ./bootstrap running at >> the same time for different projects, both pointing to the same gnulib >> .git work tree. I think the path indicated by --gnulib-srcdir should be >> read-only. >> >> Should the 'git checkout' code be replaced with something like >> >> git clone --reference "$GNULIB_SRCDIR" "$gnulib_path" \ >> && git checkout -C "$gnulib_path" $GNULIB_REVISION >> GNULIB_SRCDIR="$gnulib_path" >> >> Discussion before suggesting patches would be useful, to establish some >> agreement on how we want this to behave. > > You're right, --gnulib-srcdir and the $GNULIB_SRCDIR variable denote > "the local directory where gnulib >sources reside. Use this if you already >have gnulib sources on your machine, and >you want to use these sources." > (I introduced the distinction between GNULIB_SRCDIR and GNULIB_REFDIR > in commit 2122284380cc0d1b3b6f11d92c04652616da79c7.) > > Thus the behaviour you observed is a bug. Even worse, 'bootstrap' does > it even when the option --no-git is given! > > How to reproduce: > $ git clone git://git.savannah.gnu.org/make.git > $ cd make > $ ./bootstrap --no-git --gnulib-srcdir=$GNULIB_SRCDIR > > I think the use of --gnulib-srcdir when GNULIB_REVISION is specified > in bootstrap.conf is a classical example of conflicting requests. > Which one should take precedence? > - IMO --gnulib-srcdir is documented in such a way that it takes > precedence. > - But one may also argue that it should produce an error, to make > the user aware of the conflict. Something like > "The option --gnulib-srcdir cannot be honored together because the > package specifies a GNULIB_REVISION." > The user should be able to resolve the conflict either way, > by choosing different command-line options. My reaction was initially exactly the same as yours, until I found this piece of --help documentation, which actually is the first (and presumably highest priority) rule: * If the environment variable GNULIB_SRCDIR is set (either as an environment variable or via the --gnulib-srcdir option), then sources are fetched from that local directory. If it is a git repository and the configuration variable GNULIB_REVISION is set in bootstrap.conf, then that revision is checked out. So I think this combination is intended to be supported, it is just not working when a .gitmodules file is present in $CWD -- something that is not mentioned as a requirement. I would certainly agree that trying to understand the interaction between: 1) --gnulib-srcdir 2) --gnulib-refdir 3) GNULIB_REVISION 4) --no-git 4) building from a tarball with .gitmodules file 5) building from a tarball without .gitmodules file 6) building from a git clone with a .git sub-directory 7) building from a git clone with an indirect .git file 8) building with GNULIB_REVISION provided as an environment variable outside of bootstrap.conf and maybe other factors is really complicated, and I have had to read both --help and source code to feel close to understanding things. Alas, GNULIB_REVISION is not documented in doc/gnulib.texi. My impression is that the ./bootstrap script has gained a lot of complexity that has evolved organically. For some projects this complexity is unwanted -- e.g., guile-gnutls is used early in the bootstrapping of Guix and we eventually resolved to putting all needed gnulib files in .git and used a naive ./bootstrap script: https://gitlab.com/gnutls/guile/-/blob/master/bootstrap?ref_type=heads /Simon signature.asc Description: PGP signature
Re: ./bootstrap --gnulib-srcdir and GNULIB_REVISION
Hi Simon, > Bug #2: ./bootstrap writes to the path indicated by --gnulib-srcdir with > the 'git checkout' command, and leaves the --gnulib-srcdir path at that > commit after ./bootstrap is finished. This happens to work in my > example since I pointed it to a writable work tree, but I think altering > that path is unexpected and not documented. Imagine pointing this to a > system-wide gnulib .git store like --gnulib-srcdir=/usr/share/src/gnulib > or similar read-only place. Or imagine multiple ./bootstrap running at > the same time for different projects, both pointing to the same gnulib > .git work tree. I think the path indicated by --gnulib-srcdir should be > read-only. > > Should the 'git checkout' code be replaced with something like > > git clone --reference "$GNULIB_SRCDIR" "$gnulib_path" \ > && git checkout -C "$gnulib_path" $GNULIB_REVISION > GNULIB_SRCDIR="$gnulib_path" > > Discussion before suggesting patches would be useful, to establish some > agreement on how we want this to behave. You're right, --gnulib-srcdir and the $GNULIB_SRCDIR variable denote "the local directory where gnulib sources reside. Use this if you already have gnulib sources on your machine, and you want to use these sources." (I introduced the distinction between GNULIB_SRCDIR and GNULIB_REFDIR in commit 2122284380cc0d1b3b6f11d92c04652616da79c7.) Thus the behaviour you observed is a bug. Even worse, 'bootstrap' does it even when the option --no-git is given! How to reproduce: $ git clone git://git.savannah.gnu.org/make.git $ cd make $ ./bootstrap --no-git --gnulib-srcdir=$GNULIB_SRCDIR I think the use of --gnulib-srcdir when GNULIB_REVISION is specified in bootstrap.conf is a classical example of conflicting requests. Which one should take precedence? - IMO --gnulib-srcdir is documented in such a way that it takes precedence. - But one may also argue that it should produce an error, to make the user aware of the conflict. Something like "The option --gnulib-srcdir cannot be honored together because the package specifies a GNULIB_REVISION." The user should be able to resolve the conflict either way, by choosing different command-line options. Bruno
Re: ./bootstrap --gnulib-srcdir and GNULIB_REVISION
My quick reaction is that you've identified two bugs. --gnulib-srcdir should only read from that directory, and GNULIB_REVISION should work in bootstrap.conf.
./bootstrap --gnulib-srcdir and GNULIB_REVISION
Hi I'm trying to get ./bootstrap from a minimal source-only archive generated via 'git archive' that has GNULIB_REVISION set in bootstrap.conf, expecting this to work: ./bootstrap --gnulib-srcdir=/home/jas/src/gnulib Bug #1: it seems GNULIB_REVISION in bootstrap.conf has no effect, and this code is the reason (quoting bootstrap-funclib.sh): # XXX Should this be done if $use_git is false? if test -d "$GNULIB_SRCDIR"/.git && test -n "$GNULIB_REVISION" \ && ! git_modules_config submodule.gnulib.url >/dev/null; then (cd "$GNULIB_SRCDIR" && git checkout "$GNULIB_REVISION") || cleanup_gnulib fi The reason is that the tarball has .gitmodules looking like this: [submodule "gnulib"] path = gnulib url = https://git.savannah.gnu.org/git/gnulib.git Which trigger the '! git_modules_config submodules.gnulib.url'. The result is that GNULIB_REVISION is not respected, and I get whatever gnulib code happens to be checked out in --gnulib-srcdir. What's the reason for that check? The logic here isn't that clear. How about simply using: if test -d "$GNULIB_SRCDIR"/.git && test -n "$GNULIB_REVISION"; then (cd "$GNULIB_SRCDIR" && git checkout "$GNULIB_REVISION") || cleanup_gnulib fi At least it seems like a bug that GNULIB_REVISION is not respected, the --help output suggests this should work, which doesn't say anything about got submodules affecting behaviour: * If the environment variable GNULIB_SRCDIR is set (either as an environment variable or via the --gnulib-srcdir option), then sources are fetched from that local directory. If it is a git repository and the configuration variable GNULIB_REVISION is set in bootstrap.conf, then that revision is checked out. I can work around bug#1 with the following: rm .gitmodules ./bootstrap --gnulib-srcdir=/home/jas/src/gnulib That result in the correct gnulib commit being used, and all is fine. Bug #2: ./bootstrap writes to the path indicated by --gnulib-srcdir with the 'git checkout' command, and leaves the --gnulib-srcdir path at that commit after ./bootstrap is finished. This happens to work in my example since I pointed it to a writable work tree, but I think altering that path is unexpected and not documented. Imagine pointing this to a system-wide gnulib .git store like --gnulib-srcdir=/usr/share/src/gnulib or similar read-only place. Or imagine multiple ./bootstrap running at the same time for different projects, both pointing to the same gnulib .git work tree. I think the path indicated by --gnulib-srcdir should be read-only. Should the 'git checkout' code be replaced with something like git clone --reference "$GNULIB_SRCDIR" "$gnulib_path" \ && git checkout -C "$gnulib_path" $GNULIB_REVISION GNULIB_SRCDIR="$gnulib_path" Discussion before suggesting patches would be useful, to establish some agreement on how we want this to behave. /Simon signature.asc Description: PGP signature
