Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
Dave Hart davehart_gmail_exchange_...@davehart.net writes: If anyone knows of examples of non-recursive Makefile implementations that manage to preserve the recursive make property of being able to make in a subdir to make a subset, please share so we can learn from their pioneering. Could you just put a (probably constant) stub Makefile in each subdirectory, which simply invokes the top-level Makefile with an appropriate parameter to indicate which subdirectory it came from...? -Miles -- Religion, n. A daughter of Hope and Fear, explaining to Ignorance the nature of the Unknowable.
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
I agree the reason becomes less compelling as more capable systems become more commonplace, but I do not agree ancient RISC boxes are no longer an interesting target for current NTP builds. The machine I use (and many of us, too) has a MIPS-like chip, the Loongson. -- Dr Richard Stallman President, Free Software Foundation 51 Franklin St Boston MA 02110 USA www.fsf.org www.gnu.org Skype: No way! That's nonfree (freedom-denying) software. Use free telephony http://directory.fsf.org/category/tel/
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On 11/24/2011 04:51 PM, Richard Stallman wrote: I agree the reason becomes less compelling as more capable systems become more commonplace, but I do not agree ancient RISC boxes are no longer an interesting target for current NTP builds. The machine I use (and many of us, too) has a MIPS-like chip, the Loongson. I don't think portability has anything to do with age or family of the processor, but more with the operating system. Ancient RISC boxes were running only very small parts of the GNU userspace. Instead, modern ARM/MIPS/SPARC/PowerPC machines are often GNU/Linux or Busybox/Linux (and if the latter, they are cross-compiled with a GNU toolchain). Paolo
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On 11/22/2011 8:03 PM, Dave Hart wrote: If Autotools are primarily intended to support those using GNU/Linux systems and portability is not a goal, your argument that GNU has won and BSD compatibility of free software is no longer worthwhile makes sense. Where did I say automake 2 = Linux-only? What I said is that if this next-generation version of automake were to require GNU make, it would probably still support 99.999% of all hosts it currently targets out of the box, and could pick up the last 0.001% with a widely available prerequisite install. The 99.999% means Linux, OS X, Cygwin, and MinGW, at least, plus those systems where GNU make wasn't the default but is installed anyway for some other reason. For those that think it's a huge problem for an autotools based package to have *any* build prerequisites, I used Unices in the bad old days where sed and awk were optional installs, and that didn't stall autotools' deployment back then, when it was more vulnerable. A requirement to install GNU make to build a Makefile generated via automake 2 won't kill the autotools now. My point is not to argue in favor of GNU vs BSD or anything else religious like that. I am simply pointing out that a requirement for GNU make is scarcely more stringent these days than requiring a vaguely POSIX compatible shell. As Ralf suggests, Automake has made it this far without requiring more than portable make. Yes, and we've bought that last 0.001% of compatibility with bigger, slower, and harder to read generated Makefiles and configure scripts. TANSTAAFL. If the price to lose some bloat, gain some speed, and increase the clarity of these files is that I have to install GNU make on the 0.001% of systems where it isn't installed already, that seems a fair trade.
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On 2011-11-22 17:38 -0700, Warren Young wrote: - Reduced build overhead. GNU make can do many things internally that standard make(1) needs to shell out to do. For example, given a list of *.cpp names, generate *.o: OBJECTS=$(SOURCES:.cpp=.o) You might think this is a bad example in the context of this discussion, since Automake can do that conversion itself, since it makes you give it the list of source files. Yes, this is a bad example, because the above line is portable make! Cheers, -- Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On Wed, 23 Nov 2011, Warren Young wrote: Yes, and we've bought that last 0.001% of compatibility with bigger, slower, and harder to read generated Makefiles and configure scripts. TANSTAAFL. If the price to lose some bloat, gain some speed, and increase the clarity of these files is that I have to install GNU make on the 0.001% of systems where it isn't installed already, that seems a fair trade. There is no argument that depending on GNU make won't enable the Makefiles to become much more compact. My argument continues to be that if autotools is going to demand a GNU prerequisite to build software (for the first time since the dawn of GNU) that we should at least get something really useful from it such as the ability to use Makefile.am files directly (without having to perform sed substitutions on a packaged template Makefile.in), and to be able to easily create non-recursive build environments (which are really the only case that 'make' works properly). Beyond that, it mostly comes down to this simple question: What do I type in order to invoke GNU make on this system? Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On Wed, Nov 23, 2011 at 14:39, Warren Young war...@etr-usa.com wrote: Yes, and we've bought that last 0.001% of compatibility with bigger, slower, and harder to read generated Makefiles and configure scripts. TANSTAAFL. If the price to lose some bloat, gain some speed, and increase the clarity of these files is that I have to install GNU make on the 0.001% of systems where it isn't installed already, that seems a fair trade. It's unclear whether Autoconf + Automire will be able to detect and use GNU make which is installed but is not named 'make'. As I said, the basic tarball user's build instructions are configure make. I realize there can be automation to use, for example, gmake via Makefile vs. Makefile.gnu, and if that sort of change is in place before the first requirement for GNU make, the pain is reduced. In the realm of people building NTP from source, far more than 1 in 100,000 seem to be using systems where 'make' is available but is not GNU make. Even if we define the requirement as $(MAKE-make) is GNU make or gmake is GNU make, more than 1 in 100,000 that I've dealt with using NTP tarballs are using systems where GNU is not built-in, and would be negatively impacted by the additional prerequisite required to build NTP. However, as long as this experimentation with requiring GNU make is done in an Automire fork and not Automake, I have no qualms greater than concern for maintainer attention to Automake fading over time in favor of Automire. Cheers, Dave Hart
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On Wed, 23 Nov 2011, Dave Hart wrote: In other words, Harlan has indicated to me a non-recursive Makefile sounds good in eliminating build system bugs due to each directory's Makefile having a limited view of the whole, and I agree, but he doesn't like the idea of giving up the ability to make in, say, the ntpd subdirectory and have only ntpd and its prerequisite directories components build, and I agree again. If anyone knows of examples of non-recursive Makefile implementations that manage to preserve the recursive make property of being able to make in a subdir to make a subset, please share so we can learn from their pioneering. It is certainly possible to build a subset but you have to specify full paths to the targets and the current directory needs to be where the Makefile is. Automatically finding the root of the build tree (as Mercurial and git do) in order to simplify building a subcomponent is something that an improved automake could support. Unfortunately, 'make' is so flimsy that a non-recursive build (using a single 'make' process to produce all build products) is the only way for make to know exactly what needs to be done. Unfortunately, Automake does not fully support non-recursive builds since it always recurses at least once: % make make all-am make[1]: Entering directory `/scratch/bfriesen/build/GM-16-static' make[1]: Leaving directory `/scratch/bfriesen/build/GM-16-static' and each time make recurses, it needs to parse and re-evaluate the encountered Makefile from scratch based on what it sees in the filesystem. The costs of recursion (using 'make') are cumulatively quite expensive and something that Automake developers should consider to be a primary focus to eliminate if Automake will continue to depend on a 'make' type tool. The relative costs of recursion will only continue to grow as software projects grow, and as computing systems are delivered with tens of CPU cores (which recursion prevents from being efficiently used). Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On 11/22/2011 8:06 PM, Harlan Stenn wrote: Warren wrote: On 11/22/2011 6:02 PM, Harlan Stenn wrote: The BSDs have their good reasons to want to avoid GPL'd code, especially GPL3. Besides, why should BSD purity get to hold back the Autotools? So GNU/Linux purity is fine but BSD purity is not? Where did I advocate GNU/Linux purity? Making GNU make a prerequisite does not require that you convert your systems to Linux, nor to relicense everything else under a GNU license. How does installing GNU make on an otherwise pure BSD system compromise its integrity? It doesn't occupy an especially privileged position. It isn't a library. It doesn't itself generate code. It isn't used to boot and run anything on a Unixy system, only to modify things, and the new things generated aren't tainted by it. For that matter, where are these pure BSD systems anyway? The BSDs all still ship with gcc, don't they? I know FreeBSD is trying to move to clang, but even that accomplished, I doubt that's the only other GPL program on your average BSD system. If the distrowatch.com stats are to be believed, *BSD's market share is under 1% that of Linux There are %s and there are numbers. And we all know what they say about statistics. Yes, yes, Disraeli and Twain, very good. I like Voltaire better: A witty saying proves nothing. (Very meta.) I don't mind that you don't like my stats, but if you want to persuade me to think past them, you'll need to find or collect numbers that paint a significantly different picture. Present them, and we can argue the relative merits of their collection methodologies. Impugning mine just because they are statistics (w...) and therefore obviously lies will only persuade the innumerate. People like to say that statistics lie, but that's only when you're doing it wrong. I told you where I got my stats, and DistroWatch tells you how they collect them. I didn't come out and say exactly how I crunched them to get my percentages, but only because I think anyone who follows those vague footprints will arrive at the same destination. I'm refraining, mightily and successfully, from escalating. At least so far. Well, I know I come across as a strong advocate here, but more an advocate for clear thinking than anything else. I stepped into this thread because I see a lot of wooly arguments against the proposal. I'm not offering to do any of this rewrite work, and I won't lose any sleep if no one else does it, either. I just hate to see the idea shot down with weak arguments. I think if we're honest, the most powerful explanation of why this probably won't happen is that it's just a lot of work. It's not really about GNU vs BSD make, it isn't about the 0.001% against the 99.999%, and it isn't about licenses. I wouldn't have even bothered butting in if that were the stated reason for not doing it, but we're getting these weak reasons instead.
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On 11/23/2011 8:08 AM, Bob Friesenhahn wrote: Beyond that, it mostly comes down to this simple question: What do I type in order to invoke GNU make on this system? No, that's easily wrapped. Aautoconf can write out both a GNUmakefile and a Makefile. When both are present, GNU make will read GNUmakefile unless forced to do otherwise via -f. So, it will contain the actual build instructions. The generated Makefile is just a compatibility wrapper, to sit across the many paths whereby a non-GNU make could be run instead. Basically, it looks like this: all clean install [more targets]: gmake $@ The name 'gmake' is detected by autoconf, in the same sort of way it currently figures out what your C compiler is called.
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On 11/21/2011 09:56 PM, Stefano Lattarini wrote: Here is my tentative plan to act on the proposal: 1. We start requiring GNU make in an experimental automake 2.0 development line (which might, and will, break whathever backward-compatibility gets in its way). 2. Concurrently, we continue to support the more portable (and tested, and used-in-the-real-world) 1.x line, with bugfixes at least (and probably also with addition of new not-too-big features). 3. We publicize this move in the automake (1.x) web pages, documentation, etc, inviting users and developers to try out the new automake 2.0 pre-alpha, and to send cricisims, suggestions, praise and ranting to the automake lists. 4. Time and user responses decide wether automake 2.0 will succeed or die out. WDYT? It seems very hard to be successful and break backwards-compatibility. When we introduced shell functions into Autoconf, and in general updated Autoconf/M4sh/libtool for relatively new shells (new = newer than Ultrix), it was successful exactly because no one noticed! Paolo
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
Hi Paolo, thanks for the reply. On Tuesday 22 November 2011, Paolo Bonzini wrote: On 11/21/2011 09:56 PM, Stefano Lattarini wrote: Here is my tentative plan to act on the proposal: 1. We start requiring GNU make in an experimental automake 2.0 development line (which might, and will, break whathever backward-compatibility gets in its way). 2. Concurrently, we continue to support the more portable (and tested, and used-in-the-real-world) 1.x line, with bugfixes at least (and probably also with addition of new not-too-big features). 3. We publicize this move in the automake (1.x) web pages, documentation, etc, inviting users and developers to try out the new automake 2.0 pre-alpha, and to send cricisims, suggestions, praise and ranting to the automake lists. 4. Time and user responses decide wether automake 2.0 will succeed or die out. WDYT? It seems very hard to be successful and break backwards-compatibility. Yes, but in the automake case we'd want to break backward-compatibility to offer better and/or more powerful APIs than the current ones -- whose limitations and complications are mostly due to the necessity of having to work with portable make. To quote the README from the (sadly dead) Quagmire: One general rule in Quagmire is that if there is a simple, generic GNU make way to do something, then we don't need special support for it. This leads to many differences from the way that Automake does things. For instance, in Automake there is a separate quux_CFLAGS variable for an aggregate. In Quagmire, you would instead use a target-specific CFLAGS assignment. When we introduced shell functions into Autoconf, and in general updated Autoconf/M4sh/libtool for relatively new shells (new = newer than Ultrix), it was successful exactly because no one noticed! Maybe a first step would be to rewrite many (most?) of the automake internals to take advantage of GNU make features, while retaining ~ 90% of backward compatibility; and only then start to improve the APIs and, where necessary/conveninent, breaking backward compatibility. Regards, Stefano
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On Tuesday 22 November 2011, Paolo Bonzini wrote: On 11/22/2011 01:13 PM, Stefano Lattarini wrote: When we introduced shell functions into Autoconf, and in general updated Autoconf/M4sh/libtool for relatively new shells (new = newer than Ultrix), it was successful exactly because no one noticed! Maybe a first step would be to rewrite many (most?) of the automake internals to take advantage of GNU make features, while retaining ~ 90% of backward compatibility; and only then start to improve the APIs and, where necessary/conveninent, breaking backward compatibility. I agree. However, I am not sure of the benefit except for Automake maintainability. Which IMHO would be a killer benefit :-) But now that I think about it, a GNU make-based rewrite might also offer better extensibility (if we get the APIs right, that is), and that would be a *great* improvement over the current situation, and one which would benefit the whole user base (not only the maintainers). One old (and fortunately :) never realized) idea I had was to rewrite Automake in m4 as an extension of Autoconf. The main benefit would be introspectability of the build process via m4 traces. This would indeed be much simpler if you could base the implementation on GNU Make features from the beginning; but Automake is a very mature program with a mature codebase. Regards, Stefano
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
At the risk of repeating myself from the last time this question came up, let me selfishly say as a NTP maintainer that I do not look forward to NTP configure failing with a message indicating GNU make is required and could not be located. I have no appreciation for how much simpler and easier to maintain Automake might become with a shift from targetting portable make to requiring GNU make. I've never maintained Makefile or Makefile.am files in a GNU-make-only project. I do find it is sometimes easier to track down problems affecting both GNU make and more traditional implementations using a traditional make as the verbose debug output of GNU make is so much longer due to more implicit rules. It would be my inclination to stay with older Automake as long as feasible if newer Automake drops support for traditional make. Harlan Stenn, who initially converted the NTP code to use Autoconf and Automake, likely has a different perspective which might well matter more than mine. Cheers, Dave Hart
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On 11/22/2011 04:35 PM, Stefano Lattarini wrote: 1. Automake 2 turns out to be a failure, it gets abandoned, and Automake 1 becomes again the center of all our developement efforts. No problem for you, since you're still using this older automake. 2. Automake 2 is a success, and we drop support for Automake 1. At this point, it shouldn't be too big a pain for you to convert to the new automake (a good documentation about incompatibilities between, and/or transition from, automake 1 and 2 should exist at this point). Also, assuming that many other packages are using automake 2 by now, and thus requiring GNU make, it should be much more acceptable for the NTP build system to do the same. A half transition happens, which turns out to be a nightmare until Automake 2 adoption actually becomes common. This is exactly what happen between Automake 1.5 and Automake 1.8, each of which introduced many backwards-incompatible features, and between Autoconf 2.50 and some time later. Paolo
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On Tuesday 22 November 2011, Paolo Bonzini wrote: On 11/22/2011 04:35 PM, Stefano Lattarini wrote: 1. Automake 2 turns out to be a failure, it gets abandoned, and Automake 1 becomes again the center of all our developement efforts. No problem for you, since you're still using this older automake. 2. Automake 2 is a success, and we drop support for Automake 1. At this point, it shouldn't be too big a pain for you to convert to the new automake (a good documentation about incompatibilities between, and/or transition from, automake 1 and 2 should exist at this point). Also, assuming that many other packages are using automake 2 by now, and thus requiring GNU make, it should be much more acceptable for the NTP build system to do the same. A half transition happens, which turns out to be a nightmare until Automake 2 adoption actually becomes common. This is exactly what happen between Automake 1.5 and Automake 1.8, each of which introduced many backwards-incompatible features, and between Autoconf 2.50 and some time later. Hmm, good point... maybe the safest way to prevent this is to make it clear that this would be a *new project*, that while inspired to automake and sprouting from its codebase, has different goals and assumptions and APIs? In particular, a new name might be warranted... What about AutoMire? It's not already taken (just make a web search), it gives some due credit to Quagmire (from which I'd like to steal as much as I can :-), and will allow us to retain the `AM_' namespace. Regards, Stefano
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
Hi Ralf. On Tuesday 22 November 2011, Ralf Corsepius wrote: On 11/22/2011 04:50 PM, Bob Friesenhahn wrote: On Tue, 22 Nov 2011, Stefano Lattarini wrote: Which IMHO would be a killer benefit :-) But now that I think about it, a GNU make-based rewrite might also offer better extensibility (if we get the APIs right, that is), and that would be a *great* improvement over the current situation, and one which would benefit the whole user base (not only the maintainers). Only Automake maintainers (a diminishingly-small percentage of the total user base) care about how easy it is to maintain Automake. The users care about how easy and reliable it is to build the software. It would be useful to enumerate the user-visible benefits if Automake can depend on using GNU make. It's hard for me to imagine any, because keeping Makefile.am's free of any proprietary make constructs (comprising gmake's) had been automake's job. Not exactly; automake job's has been double-fold: 1. As you say, helping in keeping Makefile.am's free of proprietary make constructs, *but only if the maintainer asks for it* (`-Wportability' or `--gnu'). 2. Offering well-tested and feature-rich implementation of common targets and checks -- while trying hard to remain compatible with portable make. That said, apart from the fact that each generation of automake maintainers at one point in his automake-carriere comes up with switch to gmake, my feel is automake must not use gmake because (in theory) there should not be any to use gmake. I don't understand what you're trying to convey here, sorry. Another question is if GNU make is really good enough to warrant this sort of change. Good point - gmake has a long history of hickups :-) Care to elaborate on this? That sounds like something I should really know more about. Thanks, Stefano
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On Tue, 22 Nov 2011, Ralf Corsepius wrote: Another question is if GNU make is really good enough to warrant this sort of change. Good point - gmake has a long history of hickups :-) My question was not meant to imply that GNU make is riddled with bugs. My question is if deciding to move to a tool which optimizes 30+ year old build concepts is a good idea. Automake is a good build system because it provides a simple syntax by which the developer can specify his intention. Any analysis of the build for large projects will show that timestamp-based 'make' and recursion lead to huge losses in build performance and build integrity. Build dependencies and knowledge of the current build state are not adequately handled by timestamp-based 'make', even if it is GNU make. Software builds will only grow larger. It is time for some innovative thinking. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On Tuesday 22 November 2011, Ralf Corsepius wrote: That said, apart from the fact that each generation of automake maintainers at one point in his automake-carriere comes up with switch to gmake, This to me is the real point. I feel history repeats. I guess that's the sad fate of humanity ;-) my feel is automake must not use gmake because (in theory) there should not be any to use gmake. I don't understand what you're trying to convey here, sorry. Sorry, fedora's broken thunderbird had corrupted my sentence: Let me try to rephrase it: If automake so far has been able to achieve its job, by not using gmake proprietary constructs in its Makefile.ins, then there should not be any need for automake to _now_ start using gmake-constructs in Makefile.ins. Or simpler: So far, automake has not been using gmake, so why should it now start doing so? Because IMHO the cost/benefit ratio of using portable make only has become higher and higer -- not because the cost of writing portable Makefiles has increased, rather because the the benefits of doing so have stadily decreased over time, thanks to the rise of GNU/Linux and, considerably less, of Cygwin (rise which has had as a consequence that fact that their versions of the standard tools have become more widespread and easily available). Another question is if GNU make is really good enough to warrant this sort of change. Good point - gmake has a long history of hickups :-) Care to elaborate on this? Difficult to answer for me, because I am using automake with gmake (i.e. my works rely upon the subset of make-constructs automake uses) and do not exploit gmake. But I recall there had been massively broken gmake releases and releases with major functional changes, which had broken a lot. This sounds scary. Still, I'd like to see for such a serious statement more solid proof and references than hearsay and vague memories. As an aside, you (involountarily?) raised an ortoghonal, interesting point: the fact that many projects uses automake-generated Makefiles, which relies only on a tiny subset of the GNU make features, implies that the more advanced and thus potentially tricky of those features are much less tested real-wordly-wise than they could be. Having Automake-generated Makefiles require GNU make could probably help, in the long run, to improve the quality of GNU make itself. Regards, Stefano
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
Hi Bob. On Tuesday 22 November 2011, Bob Friesenhahn wrote: On Tue, 22 Nov 2011, Ralf Corsepius wrote: Another question is if GNU make is really good enough to warrant this sort of change. Good point - gmake has a long history of hickups :-) My question was not meant to imply that GNU make is riddled with bugs. My question is if deciding to move to a tool which optimizes 30+ year old build concepts is a good idea. Automake is a good build system because it provides a simple syntax by which the developer can specify his intention. Any analysis of the build for large projects will show that timestamp-based 'make' and recursion lead to huge losses in build performance and build integrity. Build dependencies and knowledge of the current build state are not adequately handled by timestamp-based 'make', even if it is GNU make. Software builds will only grow larger. It is time for some innovative thinking. I have to drop the ball on this, because I don't feel qualified nor motivated enough to follow you on this road (especially not qualified enough). Also, if a non-negligible percentage of peoples have problems accepting even just GNU make as a requirement over plain vendor make, I'm not sure how your theoretical new-generation build tools would be greeted in practice. All of this is not to belittle your idea, that should be clear -- it's just that IMHO it sounds quite ortoghonal to the ongoing discussion. Regards, Stefano
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On Tue, 22 Nov 2011, Stefano Lattarini wrote: Or simpler: So far, automake has not been using gmake, so why should it now start doing so? Because IMHO the cost/benefit ratio of using portable make only has become higher and higer -- not because the cost of writing portable Makefiles has increased, rather because the the benefits of doing so have stadily decreased over time, thanks to the rise of GNU/Linux and, considerably less, of Cygwin (rise which has had as a consequence that fact that their versions of the standard tools have become more widespread and easily available). Empires come and go. Regardless, there is no reason for other perfectly good systems to throw in the towel just yet. The problem that Automake still needs to solve is enable the easy construction of reliable build systems for large projects with many build products and source files occupying a large directory tree. Relying on GNU make may make accomplishing this easier, but it does not solve the Talking Heads How Did I Get Here problem we are in now. The GNU build system has ended up where it is today based on refinements to approaches and build tools invented in the late '70s and early '80s. Even autoconf dates from the early '90s. A build system which requires one or more processes to entirely orchestrate the build order and rules based on mere file existence and simple timestamps is fundamentally broken. Currently observed problems will only grow worse. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On Tue, 22 Nov 2011, Ralf Corsepius wrote: Build dependencies and knowledge of the current build state are not adequately handled by timestamp-based 'make', even if it is GNU make. I do not agree with this claim - More precisely, I think, timestamp based handling is the only viable approach. Make does not currently write anything to the filesystem to record how the current build products were produced. It has no database. Timestamps do not indicate how current build products were produced. Automake does arrange for the compiler to write .d files which at least indicate which header files were used to build associated object files. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On 11/22/2011 04:50 PM, Bob Friesenhahn wrote: On Tue, 22 Nov 2011, Stefano Lattarini wrote: Which IMHO would be a killer benefit :-) But now that I think about it, a GNU make-based rewrite might also offer better extensibility (if we get the APIs right, that is), and that would be a *great* improvement over the current situation, and one which would benefit the whole user base (not only the maintainers). Only Automake maintainers (a diminishingly-small percentage of the total user base) care about how easy it is to maintain Automake. The users care about how easy and reliable it is to build the software. It would be useful to enumerate the user-visible benefits if Automake can depend on using GNU make. It's hard for me to imagine any, because keeping Makefile.am's free of any proprietary make constructs (comprising gmake's) had been automake's job. That said, apart from the fact that each generation of automake maintainers at one point in his automake-carriere comes up with switch to gmake, my feel is automake must not use gmake because (in theory) there should not be any to use gmake. Another question is if GNU make is really good enough to warrant this sort of change. Good point - gmake has a long history of hickups :-) Ralf
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On 22 November 2011 15:50, Bob Friesenhahn bfrie...@simple.dallas.tx.us wrote: It would be useful to enumerate the user-visible benefits if Automake can depend on using GNU make. Otherwise it is difficult to do a cost/benefit analysis from the user perspective. Can you and others please brainstorm a list of benefits (e.g. eliminate external 'compile' and 'depcomp' scripts with the benefit of better ...)? Having to avoid GNU make features is a real pain for developers: I miss various string-handling functions and modern suffix rules. Having to compile Makefile.am into a portable Makefile makes debugging problems harder (with two extra places to look: Makefile.in and Makefile). It also complicates automake, as it ends up reimplementing features of GNU Make. -- http://rrt.sc3d.org
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On 11/22/2011 06:04 PM, Stefano Lattarini wrote: Hi Ralf. On Tuesday 22 November 2011, Ralf Corsepius wrote: On 11/22/2011 04:50 PM, Bob Friesenhahn wrote: On Tue, 22 Nov 2011, Stefano Lattarini wrote: Which IMHO would be a killer benefit :-) But now that I think about it, a GNU make-based rewrite might also offer better extensibility (if we get the APIs right, that is), and that would be a *great* improvement over the current situation, and one which would benefit the whole user base (not only the maintainers). Only Automake maintainers (a diminishingly-small percentage of the total user base) care about how easy it is to maintain Automake. The users care about how easy and reliable it is to build the software. It would be useful to enumerate the user-visible benefits if Automake can depend on using GNU make. It's hard for me to imagine any, because keeping Makefile.am's free of any proprietary make constructs (comprising gmake's) had been automake's job. Not exactly; automake job's has been double-fold: 1. As you say, helping in keeping Makefile.am's free of proprietary make constructs, *but only if the maintainer asks for it* (`-Wportability' or `--gnu'). 2. Offering well-tested and feature-rich implementation of common targets and checks -- while trying hard to remain compatible with portable make. Mostly agreed, except that IIRC, 1. originally was different: Automake once switched to allow non-portable constructs, because the majority of automake's users is using linux+gmake underneath and does not care about portable make. That said, apart from the fact that each generation of automake maintainers at one point in his automake-carriere comes up with switch to gmake, This to me is the real point. I feel history repeats. my feel is automake must not use gmake because (in theory) there should not be any to use gmake. I don't understand what you're trying to convey here, sorry. Sorry, fedora's broken thunderbird had corrupted my sentence: Let me try to rephrase it: If automake so far has been able to achieve its job, by not using gmake proprietary constructs in its Makefile.ins, then there should not be any need for automake to _now_ start using gmake-constructs in Makefile.ins. Or simpler: So far, automake has not been using gmake, so why should it now start doing so? Another question is if GNU make is really good enough to warrant this sort of change. Good point - gmake has a long history of hickups :-) Care to elaborate on this? Difficult to answer for me, because I am using automake with gmake (i.e. my works rely upon the subset of make-constructs automake uses) and do not exploit gmake. But I recall there had been massively broken gmake releases and releases with major functional changes, which had broken a lot. Ralf
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On 11/22/2011 06:47 PM, Bob Friesenhahn wrote: On Tue, 22 Nov 2011, Ralf Corsepius wrote: Another question is if GNU make is really good enough to warrant this sort of change. Good point - gmake has a long history of hickups :-) My question was not meant to imply that GNU make is riddled with bugs. OK, but is gmake stable enough or is gmake just another moving target just like many other GNU-programs? My question is if deciding to move to a tool which optimizes 30+ year old build concepts is a good idea. Well, one may call this 30 year old build concept rock-solid. Automake is a good build system because it provides a simple syntax by which the developer can specify his intention. Agreed, ... as a by-product you also buy-in a large amount of portability. A by-product many people seem to forget about because they get it for free. Any analysis of the build for large projects will show that timestamp-based 'make' and recursion lead to huge losses in build performance and build integrity. Build dependencies and knowledge of the current build state are not adequately handled by timestamp-based 'make', even if it is GNU make. I do not agree with this claim - More precisely, I think, timestamp based handling is the only viable approach. That said, I agree insofar as automake has efficiency problems due to the ballast it carries around to support decade old OSes. Software builds will only grow larger. It is time for some innovative thinking. There are plenty of alternative build-tools, each of them likely once believed to have the ultimate solution. Ralf
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
I probably still do the lion's share of updates to the Makefile.am's in NTP, at least for nontrivial changes. But I'm not sure that matters for this discussion. (And Dave Hart has done a few of the tricky ones, too.) NTP started using autoconf as its detection needs are ... nontrivial, and started using automake as I wanted a way to have a single copy of the source code with subdir build trees. Metaconfig (the beast used by Perl that I maintained for its version 2 lifetime) did not have this capability (and as I understand it, still doesn't). NTP pretty much runs everywhere. It was not that long ago that NTP dropped support for KR C compilers, and at that point required ANSI C (and I'm not sure if it's C89/C90 or C99, but nobody has complained so I haven't looked harder). Now that commodity x86 boxes are so pervasive there is no longer a compelling reason to support things like ancient sparc, mips, hp, ... boxes. My goal is to make sure that people can easily build NTP. Toward that end I want to minimize the number of extra tools that might need to be installed. I would not want to require GCC, for example. We don't require perl, but if it is there we use it. We do not require yacc-lex/bison-flex or GNU autogen for building. But if a developer wants to changes certain files, those tools will be needed. If there is a compelling reason to upgrade from current automake we'll do it. Some things I'd like to see would include easy non-recursive Makefiles (that would let folks easily build any list of given particular programs), and a means to integrate NTP into a larger build environment. More generally, I'd love to see a widely-used framework that would make it easy to create a build from a number of different separate packages. But I don't know what friction we'll be living with in the future, or what friction we have now that will either get worse or go away. Live and learn, if we're lucky... H
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
Ralf wrote: If automake so far has been able to achieve its job, by not using gmake proprietary constructs in its Makefile.ins, then there should not be any need for automake to _now_ start using gmake-constructs in Makefile.ins. I agree, there is probably no *need*. Or simpler: So far, automake has not been using gmake, so why should it now start doing so? - compelling, significant performance gains - compelling, significant improvement in inter-package builds are the two most obvious cases to me. H
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On 11/22/2011 8:50 AM, Bob Friesenhahn wrote: It would be useful to enumerate the user-visible benefits if Automake can depend on using GNU make. - For me, the biggest potential benefit isn't to Automake, but to Autoconf: if it were reimplemented in terms of GNU make, you'd then get parallel ./configure for free. (e.g. gmake -j config) One way to do that would be for each autoconf test to write out a file containing the results of the test, so make's dependency graph walker could figure out which tests can be run independently of each other and correctly order the rest. I'm envisioning that you'd have a subdirectory (e.g. accache) with one file in it per configure test, essentially replacing config.cache. That in turn means that this hypothetical Autoconf 3: Return of the Code Jedi would give you test result caching for free. Keep in mind, we aren't far from seeing 8-core *laptops*. My simplest autotools based projects take about 5 seconds to reconfigure on my fastest box. It would be nice if I could get that down below a second. - Reduced build overhead. GNU make can do many things internally that standard make(1) needs to shell out to do. For example, given a list of *.cpp names, generate *.o: OBJECTS=$(SOURCES:.cpp=.o) You might think this is a bad example in the context of this discussion, since Automake can do that conversion itself, since it makes you give it the list of source files. However, I expect that compressing the same effect into a single line would save time due to the lower I/O and parsing overheads. These overheads can stack up. One case that's so easy to see it would be a straw-man if it weren't for the fact that so many people use it, is Cygwin. The autotools are *lloooww* on Cygwin due to the lack of a native fork() type facility in Windows. A lot of that overhead would go away if the generated Makefiles didn't have to shell out as often to get things done, because they could use internal GNU make facilities instead. Even on a fast modern Unixy box, you can see measurable overheads due to the autotools. You've probably watched some source-based package system at work before. (e.g. FreeBSD ports, Gentoo Linux, Fink, MacPorts, Homebrew...) Wouldn't you say the autotools are responsible for ~5% of the time spent in such systems? With OSes that ship binary packages instead, well, *someone* has to build them. I seem to recall that full-featured modern OSes like RHEL take days, maybe weeks to completely rebuild from source. Outside that rarefied world, you have more common things like continuous integration servers. If someone knocks 5% down to 4%, they'd save the world hours or days per year per machine. There are probably dozens of other ways to reduce the size, parse time, and I/O overhead of generated Makefiles, once the autotools can count on having GNU make features. Some follow: - Build commands could use the $^ or $+ automatic variable instead of an explicit repetition of the prereq list. - Some variables could use GNU make's simply-expanded variable syntax (FOO := value) to avoid having to reparse and recalculate the value on each use. This can give big savings, particularly when the value is calculated by an external shell script and the value can be counted on not to change during a given build. - You can save one shell command in this common pattern in recursive Makefiles by use of GNU make's nonstandard -C flag: sometarget: cd subdir ; $(MAKE) sometarget - GNU make's text transformation functions can substitute for many common uses of sed, awk and handrolled shell code: https://www.gnu.org/s/hello/manual/make/Functions.html#Functions Again, this is not necessarily just of value to those developing the autotools. Fewer fork()s mean the autotools run faster, which can add up to user-visible build time savings in many cases. - GNU make makes it easier to do things like specify the source files for a build with wildcards, rather than the current method where you have to list them separately. I've read emails on this list a few times where people tried to justify this current requirement, but they've been so unconvincing I don't even remember the justifications. It would be much nicer for us Automake users if we could say something like this: myprog_SOURCES=myprog/%.cpp myprog/%.h Not only is that shorter than actually listing all the files, it means that whenever I add a new module to myprog/, it gets built and linked in automatically. - GNU make's more powerful '%' form of build rules would let Automake implement some things as easy defaults. For example, there's the common wish for object files to go somewhere other than the default. Automake offers subdir-objects and separate builddirs as two ways to achieve this, but I think it would be nicer if Automake generated something like this by default instead: .objs/%.o:
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On 11/22/2011 10:33 AM, Ralf Corsepius wrote: So far, automake has not been using gmake, so why should it now start doing so? Because gmake is all but ubiquitous, and has been so for a decade. The only exception I can think of is the BSDs, which still stubbornly stick to BSD make, apparently for political reasons. I think we'd see a change there PDQ if the autotools started requiring GNU make.
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
Warren wrote: So far, automake has not been using gmake, so why should it now start doing so? Because gmake is all but ubiquitous, and has been so for a decade. The only exception I can think of is the BSDs, which still stubbornly stick to BSD make, apparently for political reasons. I think we'd see a change there PDQ if the autotools started requiring GNU make. Just because you *can* be a gorilla doesn't mean you *should* be a gorilla. The BSDs have their good reasons to want to avoid GPL'd code, especially GPL3. H
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On 11/22/2011 11:18 AM, Ralf Corsepius wrote: is gmake stable enough or is gmake just another moving target just like many other GNU-programs? If you go back just three versions, you've gone back in time ~9 years. That is about the length of time that GNU make has been the default implementation of make everywhere but the BSDs, so if this hypothetical rewrite happened, I think you could safely depend only on what's available in GNU make 3.80. The past two releases have had some backwards-incompatible changes, but they're all by way of fixing obscure corner cases that are easy to stay away from. Looking at the changes, I can't say I'd want them to have kept the old behavior just for compatibility sake. For example, see the Pattern search changes section of this article: http://www.electric-cloud.com/blog/2010/08/03/gnu-make-3-82-is-out/ As long as this hypothetical rewrite used GNU make in standard ways, I don't see that it's likely to get caught by future changes. I don't think I've ever had a GNUmakefile fail to work after upgrading GNU make.
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On Tue, 22 Nov 2011, Warren Young wrote: Besides, why should BSD purity get to hold back the Autotools? If the distrowatch.com stats are to be believed, *BSD's market share is under 1% that of Linux, which itself is only about 1% of the overall market of machines the Autotools can reasonably be used on. Further reduce that by the percentage of BSD boxes that have not yet had gmake installed after installation; 10% maybe? We're probably talking about a set of boxes comprising 0.001% of the market. (10% of 1% of 1%.) I agree that there can be massive savings in effort if we only support the 99 percenters and only support one common OS running only on the common i686 platform. It should not matter that the 1% of developers who are developing under some OS other than GNU/Linux produce vastly more than 1% of the development innovations because they are still actively using source code and doing interesting development work on interesting hardware. At least 99% of GNU/Linux users (the 99 percenters) only use binary distributions and never look at source code so there is hardly any value to the source code being free any more and we should just tear up that manifesto and sign it all over to a nice big umbrella corporation who will agree to provide us with our binary distributions. If we can just accept that GNU/Linux of a particular recent vintage (e.g. Ubuntu 10.10) is the approved target than most of the configure script tests can be removed and everything will be so much easier. Bob P.S. I choose to be a 1 percenter. -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On 11/22/2011 7:46 PM, Bob Friesenhahn wrote: P.S. I choose to be a 1 percenter. 1% I'm fine with. It's 0.001% I'm talking about. Do we let OpenVMS drive Autotools' direction, too? OS/2? BeOS? Ooops, nope. GNU make probably shipped on those platforms, too. :)
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
Warren wrote: On 11/22/2011 6:02 PM, Harlan Stenn wrote: The BSDs have their good reasons to want to avoid GPL'd code, especially GPL3. Besides, why should BSD purity get to hold back the Autotools? So GNU/Linux purity is fine but BSD purity is not? If the distrowatch.com stats are to be believed, *BSD's market share is under 1% that of Linux, which itself is only about 1% of the overall market of machines the Autotools can reasonably be used on. Further reduce that by the percentage of BSD boxes that have not yet had gmake installed after installation; 10% maybe? We're probably talking about a set of boxes comprising 0.001% of the market. (10% of 1% of 1%.) The hyperconservative autotools do drop backwards compatibility for marginalized systems occasionally. I seem to recall that some Ultrix compatibility hacks were dropped recently, for example. Not that I'm comparing modern BSDs to Ultrix. They BSDs probably couldn't be killed off at this point even if one wanted to. Still, to cater to the limitations of systems commanding something on the order 0.001% of the market seems a *bit* obsessive. There are %s and there are numbers. And we all know what they say about statistics. I'm refraining, mightily and successfully, from escalating. At least so far. H
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On Tue, 2011-11-22 at 19:50 -0700, Warren Young wrote: On 11/22/2011 7:46 PM, Bob Friesenhahn wrote: P.S. I choose to be a 1 percenter. Do we let OpenVMS drive Autotools' direction, too? OS/2? BeOS? Ooops, nope. GNU make probably shipped on those platforms, too. :) As a matter of fact GNU make DOES still support those platforms. The OpenVMS port is actually quite active. And Amiga, as well, if you can believe that. Anyway the code is still there and the last time I asked, people were still using it and didn't want it deleted. Cheers! :-) -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist
Re: [gnu-prog-discuss] Could automake-generated Makefiles required GNU make?
On Tue, 2011-11-22 at 18:33 +0100, Ralf Corsepius wrote: But I recall there had been massively broken gmake releases and releases with major functional changes, which had broken a lot. I don't believe that this is so. There have been changes which broke some makefiles here and there, and sometimes popular ones (for example there was a small change which broke two rules in the Linux kernel makefiles, which caused some annoyance) but massively broken is not accurate. Somewhere (lost it now) someone mentioned needing make to build GNU make (bootstrapping). In fact, GNU make has always provided a build.sh shell script that can be used if you don't already have any version of make. This script doesn't attempt to do build avoidance, of course, but for a program the size of GNU make this is not so important... and anyway, once you've built it the first time you'll have a make you can use for future attempts. -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist
