Re: Goodbye to GNU make's "build.sh" ... ?
On Sun, Jun 26, 2022 at 5:48 PM Paul Smith wrote: > Seems like it's not so much that the patch was rejected; indeed they > seem to agree there's a bug. But they wanted more work on it and it > sort of fell apart. I have no idea how much effort it would be to push > hard enough to get a change merged. i think, people use shell globbing much more often than $(wildcard) or glibc's glob impl. bash's glob impl is derived from the same old rms impl and has the bug fixed. And this allows the situation to continue. regards, Dmitry
Re: Goodbye to GNU make's "build.sh" ... ?
On Sunday, June 26, 2022, Paul Smith wrote > > > I would prefer to have that fixed in gnulib, than continue to maintain > a local copy just to avoid this issue. Then there is no choice to drop gnulib and build.sh should go. Since you asked for opinions, my opinion is that maintaining a mini make along with the real make can complicate the release procedure. It is not clear that anybody would benefit from a mini make. What was the original driving force to introduce build.sh? > Seems like it's not so much that the patch was rejected; indeed they > seem to agree there's a bug. But they wanted more work on it and it > sort of fell apart. I have no idea how much effort it would be to push > hard enough to get a change merged. > > > > >
Re: Goodbye to GNU make's "build.sh" ... ?
On Sun, 2022-06-26 at 16:57 -0400, Dmitry Goncharov wrote: > Have you considered to to avoid glob from gnulib? Make has its one > impl of glob which it uses on certain systems. > We could use this impl and maintain it on all systems. While the original issue came up WRT glob, in actuality the problem is far deeper. The later versions of gnulib will pull significant content for the findprog-in module, for example, which we do rely on quite significantly in the GNU make fastpath. However my plan was to preserve this internal version of glob and use it for systems where we couldn't use the gnulib version, such as on Windows. > Aside from dependencies, one problem with the gnulib impl of glob is > that it has a bug on some filesystems. I would prefer to have that fixed in gnulib, than continue to maintain a local copy just to avoid this issue. Seems like it's not so much that the patch was rejected; indeed they seem to agree there's a bug. But they wanted more work on it and it sort of fell apart. I have no idea how much effort it would be to push hard enough to get a change merged.
Re: Goodbye to GNU make's "build.sh" ... ?
On Sun, Jun 26, 2022 at 10:11 AM Paul Smith wrote: > This really came up because I was trying to find a way to include the > latest gnulib globbing library. Adding "glob" to this pulls in a HUGE > number of prerequisites. Have you considered to to avoid glob from gnulib? Make has its one impl of glob which it uses on certain systems. We could use this impl and maintain it on all systems. Aside from dependencies, one problem with the gnulib impl of glob is that it has a bug on some filesystems. i submitted a patch to both glibc and gnulib. https://sourceware.org/pipermail/libc-alpha/2017-November/088984.html and https://lists.gnu.org/archive/html/bug-gnulib/2019-11/msg6.html. The code is complex enough that review requires significant effort and the patch is not merged. The result is that the gnulib impl, which is supposed to be preferred, is the only one that has the bug. Code like this $(wildcard lib/ src/) works on any system other than gnu (again on some filesystems). regards, Dmitry
Re: Goodbye to GNU make's "build.sh" ... ?
Herring, Daniel - 0447 - MITLL wrote: > On Jun 26, 2022, at 3:30 PM, Eli Zaretskii wrote: > > How do people bootstrap systems nowadays? Is it only via > cross-compilation? > > > Another way to bootstrap is to have a native tool chain that is less > capable…. For example, an embedded device may come with an old or proprietary > shell and compiler. But it would still include something resembling "make" - embedded systems toolkits have shipped with clones of make since the 1980s. I'm not convinced that there are any notable systems out there capable of supporting gmake that don't already have a make utility. Or that telling anyone so unlucky "first bootstrap with gmake 4.3" is a bad answer. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Re: Goodbye to GNU make's "build.sh" ... ?
On Jun 26, 2022, at 3:30 PM, Eli Zaretskii wrote: How do people bootstrap systems nowadays? Is it only via cross-compilation? Another way to bootstrap is to have a native tool chain that is less capable…. For example, an embedded device may come with an old or proprietary shell and compiler. — Daniel smime.p7s Description: S/MIME cryptographic signature
Re: Goodbye to GNU make's "build.sh" ... ?
> From: Paul Smith > Cc: bug-make@gnu.org > Date: Sun, 26 Jun 2022 15:06:36 -0400 > > > No, I'm talking about not using Gnulib only for stage-1 build, the > > one that produces a Make binary capable of building the full > > version. I was not talking about avoiding to use Gnulib for the > > final build whose results are installed. > > This is sort of what Sven was suggesting. It could work, but configure > does a lot of things: not only does it generate a config.h but it also > finds compilers, figures out what options they accept, etc. Obviously > we don't need to do all that for a minimal build, but still the Windows > ecosystem is much simpler because it's so homogeneous; even then > build_w32.bat doesn't support building with Clang for Windows, for > example. Generating a config.h on any random system is much harder > than having one pre-defined for the Windows32 API. But this part was already supported by the bootstrap process, right? So maybe we will need 2 configure scripts: one that only configures for build.sh (and doesn't configure Gnulib), and the other which does everything. > The goal of this "mini-make" must be to be able to compile automake- > generated makefiles (since that's what's needed in order to work with a > gnulib-enabled system). Or at least, the Makefile.am that GNU make and > its gnulib modules use. > > Since POSIX make does support wildcard syntax (e.g., "foo: *.c" is > legal) we would need glob/fnmatch of some sort, unless we can guarantee > that automake won't use it, for example. Using wildcards in Makefiles is pretty rare. Do we have that now? > There's some amount of disingenuity here since make is really just the > top of the iceberg when it comes to bootstrapping: you can't run > configure without a full libc, compiler, coreutils etc. and how can you > build those without a make? So either we're talking about cross- > compilation, or else this "minimake" would need to be sufficient to > build more than just make itself. How do people bootstrap systems nowadays? Is it only via cross-compilation?
Re: Goodbye to GNU make's "build.sh" ... ?
On Sun, 2022-06-26 at 17:39 +0300, Eli Zaretskii wrote: > > The problem is that the process for "make-less" systems in the past > > has been: run configure, then run build.sh. But we won't be able > > to use the environment created by configure in this "new" build.sh, > > because it works in tandem with the makefiles. The generated > > config.h etc. will assume lots of things that simply aren't true > > anymore, since we don't have make. > > I thought of making build.sh compile and link make without any > configure step, similar to what build_w32.bat does. > > > So to me this is equivalent to my option #1, don't use gnulib at > > all (or at least, use it incredibly sparingly). > > No, I'm talking about not using Gnulib only for stage-1 build, the > one that produces a Make binary capable of building the full > version. I was not talking about avoiding to use Gnulib for the > final build whose results are installed. This is sort of what Sven was suggesting. It could work, but configure does a lot of things: not only does it generate a config.h but it also finds compilers, figures out what options they accept, etc. Obviously we don't need to do all that for a minimal build, but still the Windows ecosystem is much simpler because it's so homogeneous; even then build_w32.bat doesn't support building with Clang for Windows, for example. Generating a config.h on any random system is much harder than having one pre-defined for the Windows32 API. The goal of this "mini-make" must be to be able to compile automake- generated makefiles (since that's what's needed in order to work with a gnulib-enabled system). Or at least, the Makefile.am that GNU make and its gnulib modules use. Since POSIX make does support wildcard syntax (e.g., "foo: *.c" is legal) we would need glob/fnmatch of some sort, unless we can guarantee that automake won't use it, for example. We can hopefully disable other advanced features such as ar parsing, job control, etc. which will certainly help simplify things. There's some amount of disingenuity here since make is really just the top of the iceberg when it comes to bootstrapping: you can't run configure without a full libc, compiler, coreutils etc. and how can you build those without a make? So either we're talking about cross- compilation, or else this "minimake" would need to be sufficient to build more than just make itself.
Re: Goodbye to GNU make's "build.sh" ... ?
> From: Paul Smith > Date: Sun, 26 Jun 2022 10:11:12 -0400 > > > Can you list the Gnulib functions we currently use? > > You can find them in the bootstrap.conf file: > > https://git.savannah.gnu.org/cgit/make.git/tree/bootstrap.conf > > See the list at the end. OK, thanks. The list is very short. And I think most if not all of the functions there can be stubbed out. For example, strtoll can be stubbed by a trivial wrapper around strtol, mempcpy can be easily implemented using memcpy, and getloadavg and glob could be replaced by functions that return trivial results, since building Make doesn't really need those functions, does it? > > Perhaps we could then prepare a fake-gnulib module with trivial > > implementations of those functions, which could be used by build.sh > > instead of the real Gnulib functions. > > This seems odd to me: I mean, isn't that just replacing gnulib with > what it already is? No, because we only need this to build the initial binary of Make, which then could be used to build the full version, and that full version will use Gnulib. So the idea is to create a small file with trivially simple implementation for functions that might be missing, and for the rest rely on libc -- just for the first "stage-1", so-to-say, binary of Make. Then "stage 2" will use this stage-1 binary to configure and build the full version which uses Gnulib. > The point of gnulib is to provide replacements on systems that don't > provide these. That's not the only purpose, perhaps not only the main one. The main one (AFAIU) is to replace library functions with versions that are either devoid of known bugs or are more compatible to Posix and glibc. > > (Some Gnulib functions are replacements for those found in every C > > library, and those could be simply ignored in the build.sh build, > > relying on libc to do a reasonably good job.) > > Yes, we could say that build.sh can only be used on systems which are > essentially POSIX-compliant and don't need a lot of fixup. I think it's more than that: to build Make we need much less than full Posix compliance. > The problem is that the process for "make-less" systems in the past has > been: run configure, then run build.sh. But we won't be able to use > the environment created by configure in this "new" build.sh, because it > works in tandem with the makefiles. The generated config.h etc. will > assume lots of things that simply aren't true anymore, since we don't > have make. I thought of making build.sh compile and link make without any configure step, similar to what build_w32.bat does. > So to me this is equivalent to my option #1, don't use gnulib at all > (or at least, use it incredibly sparingly). No, I'm talking about not using Gnulib only for stage-1 build, the one that produces a Make binary capable of building the full version. I was not talking about avoiding to use Gnulib for the final build whose results are installed.
Re: Goodbye to GNU make's "build.sh" ... ?
On Sun, 2022-06-26 at 07:34 +0100, Sven C. Dack wrote: > 3. Allow to build a minimal Make executable, which provides basic and > traditional Make functionality and does not rely on gnulib, and then > use it as a bootstrap. It's certainly a possibility. But there is already so much ifdef etc. in GNU make, I'm loathe to increase it dramatically by trying to carve out some sort of minimalist, but still sufficiently powerful, version of make. And, the things we get from gnulib are not really "extra things" that we can just jettison, so much as replacements for basic POSIX functions that not all platforms provide, or where the provided versions are broken in some way on some platform. If the GNU make source code was well-organized and cleanly modular this would probably be a far more reasonable proposition but anyone who's looked at the code can attest that this is far from the case.
Re: Goodbye to GNU make's "build.sh" ... ?
On Sun, 2022-06-26 at 08:41 +0300, Eli Zaretskii wrote: > It is sad that Gnulib maintainers aren't prepared to cater to a GNU > project, and an important one such as Make. These are special > requirements that only a handful of GNU project could ever have, and > for good reasons, so supporting their needs should be a no-brainer. I suspect their position is that it IS a no-brainer, but with exactly the opposite result: this requirement is needed by exactly one project (GNU make, since once you have it you're all set) and so undertaking all that work in gnulib is not resource-effective. > > This leaves me with two options: > > > > 1. Stop using gnulib, or at least sharply limit the modules we > > will include to those with trivial-enough configurations. > > 2. Abandon the build.sh script and require an existing make > > program in order to build a new version of GNU make. > > #2 would require that we promise to have Make 4.3 available from here > to eternity. Is that feasible? Well, it doesn't have to be GNU make. It just needs to be some make that can use automake generated makefiles. However, it does require that such a make exists. There are two different situations: first, bringing up GNU make on a non-GNU system where you have some make, but not GNU make. Maybe you have BSD make or something. Then the question is, is your existing make capable of using automake-generated makefiles? Second, you might be bootstrapping an entirely new system without ANY make at all. Now you need to get one. I don't know whether the other versions of make available have facilities to build without an existing make. Here you might need an older version of GNU make which has a build.sh. Of course here you likely already have a cross-build environment so you could just cross-compile make as well. > Can you list the Gnulib functions we currently use? You can find them in the bootstrap.conf file: https://git.savannah.gnu.org/cgit/make.git/tree/bootstrap.conf See the list at the end. The newly-added strtoll() is causing the proximate issue because it relies on unistd.h which generates an enormous sed script in the makefile. This really came up because I was trying to find a way to include the latest gnulib globbing library. Adding "glob" to this pulls in a HUGE number of prerequisites. But even after undoing this I discovered strtoull() is bad enough. My idea was to keep the current glob/fnmatch to use on systems which couldn't run "configure" (for example with the build_w32.bat script) and use the gnulib glob/fnmatch for all systems that could run configure. But, configure is not enough: you also need make. The current build.sh script contains some minimal amount of magic needed to transform the simple files we used in GNU make 4.3; see https://git.savannah.gnu.org/cgit/make.git/tree/build.sh#n69 but making this work for the complex targets is frankly more than I want to take on. So then I wondered, why even both with gnulib glob? Maybe we should just take ownership of the ancient glob we already have and say, if you have a system with GNU libc you get the latest, else you get this basically functional version. > Perhaps we could then prepare a fake-gnulib module with trivial > implementations of those functions, which could be used by build.sh > instead of the real Gnulib functions. This seems odd to me: I mean, isn't that just replacing gnulib with what it already is? The point of gnulib is to provide replacements on systems that don't provide these. I don't want to create a new "micro- gnulib" project. > (Some Gnulib functions are replacements for those found in every C > library, and those could be simply ignored in the build.sh build, > relying on libc to do a reasonably good job.) Yes, we could say that build.sh can only be used on systems which are essentially POSIX-compliant and don't need a lot of fixup. The problem is that the process for "make-less" systems in the past has been: run configure, then run build.sh. But we won't be able to use the environment created by configure in this "new" build.sh, because it works in tandem with the makefiles. The generated config.h etc. will assume lots of things that simply aren't true anymore, since we don't have make. So to me this is equivalent to my option #1, don't use gnulib at all (or at least, use it incredibly sparingly).
Re: Goodbye to GNU make's "build.sh" ... ?
On Sun, 2022-06-26 at 02:14 +0200, Henrik Carlqvist wrote: > Would this "old version of make" have to be GNU make? It would need to be some version of make that is supported by automake. As far as I'm aware, automake-generated makefiles are currently intended to run with any POSIX-compliant version of make; I believe that BSD make qualifies. Certainly SunPro make from the late Jörg Shilling would qualify, but I believe it requires a pre-existing make to build (I haven't tried in quite a while). And now that we've lost him I'm not sure who will take up maintenance of that project. Also, I've heard calls for automake maintainers to simplify the generated makefiles by assuming GNU make and taking advantage of GNU make features: then we really WOULD be in a quandary. But I have no idea how likely such a thing is to happen. > The scary part of relying on an old version of some software is that > old versions sometimes suffer from bit rot caused by changes in > language standards or API changes in libraries. Indeed. > Maybe we would have to rely on cross compiling GNU make for new > systems? Another solution, to be sure.
Re: Goodbye to GNU make's "build.sh" ... ?
The obvious end-point of #2 is to add a make-4.3.tar.gz file to future make packages and modify build.sh to unpack and use it. This might involve renaming the "inner" script to "build-4.3.sh" or similar. But I also have the same question: what are the real-world cases where build.sh is needed? > On Jun 26, 2022, at 10:44 AM, Philip Guenther wrote: > > >> On Sat, Jun 25, 2022 at 12:48 PM Paul Smith wrote: > >> I'm trying to decide what the future is for GNU make's "build.sh" >> bootstrapping script. As you may recall, this script is provided to >> allow GNU make to build on systems which don't already have an instance >> of make installed. Its goal is to build the first make binary, without >> of course all the fancy parts of avoiding rebuilds, generating >> dependency files, etc. > > Can anyone contribute information about actual cases where build.sh was used? > > I've been using GNU make since 1992 or so, on only 9 or so different UNIX > platforms, but they all had _some_ form of make available, however crippled, > that was sufficient to bootstrap GNU make. So, I've avoided being its target > audience. Who has been and can they speak to what they would do without it? > "I have a native compiler, but not make" is kinda a weird place to be; those > bootstrapping a new OS typically cross-compile the entire toolchain until > they can self-host: make is just another small divot in that bumpy road and > using build.sh when they still have to cross-compile the rest of the > toolchain doesn't actually help. > > >> This leaves me with two options: >> >>1. Stop using gnulib, or at least sharply limit the modules we will >> include to those with trivial-enough configurations. >>2. Abandon the build.sh script and require an existing make program >> in order to build a new version of GNU make. > > Lacking a description of a case in the past where build.sh was actually the > only sane option, I would go with #2. > > > Philip Guenther >
Re: Goodbye to GNU make's "build.sh" ... ?
On Sat, Jun 25, 2022 at 12:48 PM Paul Smith wrote: > I'm trying to decide what the future is for GNU make's "build.sh" > bootstrapping script. As you may recall, this script is provided to > allow GNU make to build on systems which don't already have an instance > of make installed. Its goal is to build the first make binary, without > of course all the fancy parts of avoiding rebuilds, generating > dependency files, etc. > Can anyone contribute information about actual cases where build.sh was used? I've been using GNU make since 1992 or so, on only 9 or so different UNIX platforms, but they all had _some_ form of make available, however crippled, that was sufficient to bootstrap GNU make. So, I've avoided being its target audience. Who has been and can they speak to what they would do without it? "I have a native compiler, but not make" is kinda a weird place to be; those bootstrapping a new OS typically cross-compile the entire toolchain until they can self-host: make is just another small divot in that bumpy road and using build.sh when they still have to cross-compile the rest of the toolchain doesn't actually help. This leaves me with two options: > >1. Stop using gnulib, or at least sharply limit the modules we will > include to those with trivial-enough configurations. >2. Abandon the build.sh script and require an existing make program > in order to build a new version of GNU make. Lacking a description of a case in the past where build.sh was actually the only sane option, I would go with #2. Philip Guenther
Re: Goodbye to GNU make's "build.sh" ... ?
Hello, developers causing a "Catch 22" paradox is not new. You want to avoid such software, or at least work around it with a big margin. Sharply limiting the use of gnulib here may not provide such a margin and cause repeated trouble in the future. Your condition of "to those with trivial-enough configurations" can fail with the next update to gnulib. How about a third option: 3. Allow to build a minimal Make executable, which provides basic and traditional Make functionality and does not rely on gnulib, and then use it as a bootstrap. Then make the gnulib maintainers aware of the "Catch 22" and tell them not rely on newer GNU Make versions. Offering a minimal Make executable next to the full GNU Make may also have uses for other projects. -Sven On 25/06/2022 22:47, Paul Smith wrote: I'm trying to decide what the future is for GNU make's "build.sh" bootstrapping script. As you may recall, this script is provided to allow GNU make to build on systems which don't already have an instance of make installed. Its goal is to build the first make binary, without of course all the fancy parts of avoiding rebuilds, generating dependency files, etc. Unfortunately, this really cannot work if GNU make is going to rely on gnulib, because a MANY gnulib modules require not only autoconf to work, but also automake. That is, it's not enough to just run configure and then you get a set of source files and header files that you can compile by hand by just invoking the compiler. You must ALSO run make, because the modules provide makefile recipes that invoke sed, etc. to convert files into their final form. I had a discussion about this with the Gnulib maintainers a while ago: https://lists.gnu.org/archive/html/bug-gnulib/2019-09/msg00041.html However the gnulib maintainers were disinclined to modify the practices of the gnulib modules. This leaves me with two options: 1. Stop using gnulib, or at least sharply limit the modules we will include to those with trivial-enough configurations. 2. Abandon the build.sh script and require an existing make program in order to build a new version of GNU make. #1 is what I followed for GNU make 4.3, which has a limited subset of carefully-chosen modules. However this becomes harder over time. For example any module that needs unistd.h requires a very complex automake rule. If #2 is chosen, then a bootstrap process would involve first obtaining an older version of make, such as GNU make 4.3 or lower, and building that with its build.sh, then using the resulting make to build the newer version. I'm interested in thoughts about these options.
