Re: AM_ICONV and integration into sagemath
Dima Psechnik wrote:
> We are in fact having a fight now over whether to check in the
> iconv pieces produced by gnulib-tool into the source tree, or call
> gnulib-tool at bootstrap.
> https://trac.sagemath.org/ticket/34152
OK, now let's talk about the actual problem at hand.
I won't jump into the discussion there, because
- it's already a very long thread,
- I don't know what an "SPKG" is,
- I don't know about the preferred habits in the sagemath dev community.
But there I read:
> $ du -sh config/config.rpath m4/lib-*.m4 m4/iconv.m4 m4/host-cpu-c-abi.m4
> 20K config/config.rpath
> 8.0K m4/lib-ld.m4
> 36K m4/lib-link.m4
> 12K m4/lib-prefix.m4
> 12K m4/iconv.m4
> 24K m4/host-cpu-c-abi.m4
Yes, these is currently the list of files needed for AM_ICONV.
> It's some kind of a weird tradition of jumping through hoops to get
> config.rpath
If you need only a few files from Gnulib, the common approach is
to invoke either gnulib-tool (if present, indicated through an
environment variable GNULIB_SRCDIR that the user can set) or
fetch the files from the git repository one by one.
To fetch only config.rpath:
GNULIB_REPO_URL="https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob_plain;hb=HEAD;f=;
for file in build-aux/config.rpath; do
if test -n "$GNULIB_SRCDIR" && test -d "$GNULIB_SRCDIR"; then
"GNULIB_SRCDIR/gnulib-tool" --copy-file $file $file
else
wget -q --timeout=5 -O $file.tmp "${GNULIB_REPO_URL}$file" \
&& mv $file.tmp $file
fi
done
To fetch all 6 files:
GNULIB_REPO_URL="https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob_plain;hb=HEAD;f=;
for file in build-aux/config.rpath m4/lib-ld.m4 m4/lib-link.m4 m4/lib-prefix.m4
m4/iconv.m4 m4/host-cpu-c-abi.m4; do
if test -n "$GNULIB_SRCDIR" && test -d "$GNULIB_SRCDIR"; then
"GNULIB_SRCDIR/gnulib-tool" --copy-file $file $file
else
wget -q --timeout=5 -O $file.tmp "${GNULIB_REPO_URL}$file" \
&& mv $file.tmp $file
fi
done
This is the quick solution, which minimizes network traffic. But it is not
future-proof: At any moment, some more files may become needed, due to
changes in the *.m4 files. The future-proof solution is to download
the gnulib git repository (shallow, if you want) and run
$GNULIB_SRCDIR/gnulib-tool [OPTIONS] --import iconv
> I don't quite understand the relationship between automake, gettext, and
> gnutools here. Is AM_ICONV even meant to work without gnulib?
> Or is it a libtool bug?
AM_ICONV is documented in two places:
- https://www.gnu.org/software/gettext/manual/html_node/AM_005fICONV.html
- https://www.gnu.org/software/gnulib/MODULES.html#module=iconv
The consumption as a gnulib module is meant to be easier than through
gettext. A lot of work has been put into gnulib to make such a consumption
easy; gettext (autopoint) lags behind in this aspect.
> https://lists.gnu.org/archive/html/bug-gettext/2011-10/msg00012.html
Already in 2011, if you need only AM_ICONV, the gnulib module 'iconv'
was the better choice.
> config.rpath (in gnulib) appears to be an orphan.
No. It is maintained in gnulib, as you can see from the commit history.
> (Does gettext misappropriate prefixes AM_ and
> AC_ for its macros, making an impression that they are from autotools?)
Yes, the prefix AM_ here is a mistake. It should better be gl_. But
changing it now would cause more annoyance than benefit.
Bruno
Re: gnulib and version control
In [1] we document three approaches, dependending on the package's usual handling of imported/generated files. Dima Pasechnik wrote: > If a project does not use git's submodules, noone wants an extra > complexity of submodules to be added to the build system just due to > the need for gnulib. Sure. No one forces you to use a git submodule. As written in [1]: "The alternative is to always follow the newest Gnulib automatically. Note that this can cause breakages at unexpected moments, namely when a broken commit is pushed in Gnulib. It does not happen often, but it does happen." > In some projects I am or was involved, it's basically the case that the code > injected by > the gnulib-tool got committed into the tree, and blissfully forgotten > about. Yes, that's one possibility. > Why you at gnulib headquarters want large chunks of gnulib > scattered accross many, many thousands of downstream git trees, I don't know. This is not what we "want". The documentation [1] gives three options. You choose the one that fits best for your package. > My point is that it's used "elsewhere" not in the way you envision, but > by simply committing the code produced by gnulib-tool into the source > tree, and forgetting all about it. You must have studied the various packages that use gnulib quite extensively, but I have done that too. My impression is that the vast majority follows the principle "don't commit generated files into version control", that is, they use approach 2, not approach 1, from [1]. Bruno [1] https://www.gnu.org/software/gnulib/manual/html_node/VCS-Issues.html
Re: steady development
Hi Dima, In your messages, there are several topics, which I'll reply to separately. Focusing on one topic means to simplify the discussion. > regular releases are badly needed. You make it sound like releases are so much better than steady development. But releases are not ideal: When there is an issue, 1. users or developers need to find a workaround, 2. once there is a release that fixes the issue, the users need to revert the workaround and use the official feature instead. With steady development, you get and can install a fix within days. For example, just last week, Emacs wanted a different behaviour of gen_tempname() and got it within days. This matters especially for gnulib, because gnulib is used by package authors _before_ they create their tarballs. If gnulib were rolled out as releases, the time delay until a fix reaches the users would be - the time from the fix until the next gnulib release PLUS - the time from that point to the next release of the particular package. PLUS - the time it takes for your favorite distro to upgrade to that release. > As well, talking about "taking QA steps" does not inspire much confidence. > Stable, well-used, versions have obvious advantages. The QA steps I talked about were unit tests and continuous integration. I don't know about you, but I do trust a package with unit tests and a CI that verifies that the tests pass more than a package with releases but no unit tests or a package which does releases at dates that were fixed in advance. Bruno
Re: Creating a formula for Homebrew
On Thu, Aug 25, 2022 at 11:02:59PM -0500, Paul Eggert wrote: > On 8/25/22 18:20, dmitrii.pasech...@cs.ox.ac.uk wrote: > > > > There has been no official gnulib release in like 8 years... > > As far as I'm concerned none of the releases have been "official". > Gnulib simply doesn't work that way. If you want something that does > work that way, feel free to use Gnulib to build it. However, I expect > you'll find it's more trouble than it's worth; I don't recommend it. > > > > Some people want to follow gnulib guidelines to the letter - even though > > as you admit it's meant mostly for a small range of mostly FSF's > > projects crucially dependent on gnulib, > > I didn't say that. Gnulib can be used elsewhere. > > > > For 3+ years we didn't see a bug while using gettext's m4 macros needed > > for iconv support > > This sounds like it's more a gettext issue, so I suggest filing a bug > report there. Here is a gettext bug report addressing this issue, basically. Posted in 2011, no followups. https://lists.gnu.org/archive/html/bug-gettext/2011-10/msg00012.html > > > > It's some kind of a weird tradition of jumping through > > hoops to get config.rpath, while config.guess and config.sub are > > provided by automake. :-) > > Automake (and Gnulib) are actually downstream from the source for > config.guess and config.sub. I don't know about config.rpath. > > > > Imagine gnulib was a C macro library, for which you provided a tool to > > copy select sets of headers into the downstream source trees > > I don't have to imagine that, as that's basically what Gnulib is (though > I would drop the word "macro"). >
Re: Creating a formula for Homebrew
On Thu, Aug 25, 2022 at 11:02:59PM -0500, Paul Eggert wrote: > On 8/25/22 18:20, dmitrii.pasech...@cs.ox.ac.uk wrote: > > > > There has been no official gnulib release in like 8 years... > > As far as I'm concerned none of the releases have been "official". > Gnulib simply doesn't work that way. If you want something that does > work that way, feel free to use Gnulib to build it. However, I expect > you'll find it's more trouble than it's worth; I don't recommend it. > > > > Some people want to follow gnulib guidelines to the letter - even though > > as you admit it's meant mostly for a small range of mostly FSF's > > projects crucially dependent on gnulib, > > I didn't say that. Gnulib can be used elsewhere. My point is that it's used "elsewhere" not in the way you envision, but by simply committing the code produced by gnulib-tool into the source tree, and forgetting all about it. Basically, planting bugs to be fixed later everywhere. (Cf. the message from a Gentoo contributor in this thread). > > > > For 3+ years we didn't see a bug while using gettext's m4 macros needed > > for iconv support > > This sounds like it's more a gettext issue, so I suggest filing a bug > report there. It's the issue of AM_ICONV being broken, unless one either uses gnulib-tool to seed the project with config.rpath, or getting config.rpath in a right place for automake for some other means. (Does gettext misappropriate prefixes AM_ and AC_ for its macros, making an impression that they are from autotools?) I don't quite understand the relationship between automake, gettext, and gnutools here. Is AM_ICONV even meant to work without gnulib? Or is it a libtool bug? > > It's some kind of a weird tradition of jumping through > > hoops to get config.rpath, while config.guess and config.sub are > > provided by automake. :-) > > Automake (and Gnulib) are actually downstream from the source for > config.guess and config.sub. I don't know about config.rpath. config.rpath (in gnulib) appears to be an orphan. There is no contact address to report bugs in its header. It's only # Copyright 1996-2022 Free Software Foundation, Inc. # Taken from GNU libtool, 2001 # Originally by Gordon Matzigkeit , 1996 Is it maintained by libtool, or forked from libtool and maintained by gnulib? > > > > Imagine gnulib was a C macro library, for which you provided a tool to > > copy select sets of headers into the downstream source trees > > I don't have to imagine that, as that's basically what Gnulib is (though > I would drop the word "macro"). Yes, gnulib is ATM a great tool to plant bugs downstream, long-term. :-)
Re: Creating a formula for Homebrew
Bruno Haible writes: > Hi, > > Wesley Viana wrote: >> So I was wondering how to contribute by "packing" gnulib into a brew >> formula. > > Packaging gnulib through a packaging system (such as Debian, pkg, BSD ports, > or brew) is, in the current state of things, not desirable. > > Gnulib is a source code library [1], and, although the documentation states > that the user has the choice between using the git repo and stable releases > [2], > there have not been such stable releases for 4 years. That is, everyone uses > the git repo. And we are taking QA steps to ensure a high quality of the > code in the git repo. Whilke I agree with the above, I do believe there is a way to package gnulib in packaging systems that make sense: by packging the gnulib git repository including its entire git history, and keep that "package" up to date regulary. After all, that is the way developers use gnulib: it is a source-archive, and every git commit is a release that somebody may want to use. The source of the problem I see described in this thread is that distributions (for good reasons) wants to rebuild everything from source, and building from tarballs is quite fragile: it is difficult to know if you rebuilt everything from its pure source code or accidentally just re-used a pre-built artifact. The alternative that people have adopted in recent years is to build from version controlled sources and going through the bootstraps steps themselves (usually quite poorly, by just calling autoreconf -fvi which was never intended for that purpose), and if done right I think this results in something that is better for distributions. Building from git introduces two new problems: 1) not using officiall release tarball artifacts, introducing unreliability in what a release is, including what cryptographic signatures to trust, and 2) any git submodules (such as gnulib) and external resources (like translation files) needs to be available offline locally in a package. Translation files is tricky, and I'm not sure how to resolve that in a good way. There is no strong connection between a git repo and the translation file intended to be used with it, introducing unreliability and supply-chain issues. Perhaps the solution is similar to the gnulib packaging: just package all translationproject.org *.po files in a separate package in the distribution, and keep that up to date, and use that as the source during source-code rebuilds. /Simon signature.asc Description: PGP signature
Re: Creating a formula for Homebrew
> On 25 Aug 2022, at 15:53, dmitrii.pasech...@cs.ox.ac.uk wrote: > > On Thu, Feb 08, 2018 at 11:43:08AM +0100, Bruno Haible wrote: >> Wesley Viana wrote: >>> So I was wondering how to contribute by "packing" gnulib into a brew >>> formula. >> >> Packaging gnulib through a packaging system (such as Debian, pkg, BSD ports, >> or brew) is, in the current state of things, not desirable. >> >> Gnulib is a source code library [1], and, although the documentation states >> that the user has the choice between using the git repo and stable releases >> [2], >> there have not been such stable releases for 4 years. That is, everyone uses >> the git repo. And we are taking QA steps to ensure a high quality of the >> code in the git repo. > > I don't think this is how gnulib is usually used, and that's why regular > releases are badly needed. > By the way, several Linux distributions, e.g. Debian and Gentoo, do provide > gnulib packages (updated at random moments). Note that while Gentoo does have a gnulib package, it's mostly there for esoteric reasons, and we don't use it in builds of packages. So, we don't use it in builds, but I wish we could. I do wish there was a way to unbundle it but that would require solid versioning of gnulib. Right now, there's often (seemingly) API breaks which mean consumers have to be adapted. Often, bugs occur in e.g. coreutils which involve patching gnulib rather than coreutils itself. It's even more frustrating when fixing a build failure on e.g. a new platform that has its roots in gnulib and we have to patch every single gnulib consumer until they: 1. pull down a new gnulib, And 2. make new releases (which might be years). Best, sam signature.asc Description: Message signed with OpenPGP
