Re: configure speedup proposal: add --assume-c99 and --assume-posix2008 flags
Dear Rhys Ulerich, On Fri, 6 Jun 2014 10:14:41 -0500, Rhys Ulerich wrote: Granted, configure time does add up if one builds 10s of packages in sequence. Is there some reason configure caches are unsuitable to gain speedup there? I see almost a six-fold configure improvement on grep-2.20 when its working from a config.cache. That brings configure to 140% of parallel compilation, but we're talking an overall build like 6 seconds. That works when you repeatedly build the same package over and over again with the exact same toolchain and environment. My use case in Buildroot is very different: we build potentially several dozens of different software components, each time with a different toolchain/environment. So we're almost never rebuilding with the exact same configuration, and therefore using a cache doesn't really work for our use case. Best regards, Thomas -- Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com ___ Autoconf mailing list Autoconf@gnu.org https://lists.gnu.org/mailman/listinfo/autoconf
Re: configure speedup proposal: add --assume-c99 and --assume-posix2008 flags
My use case in Buildroot is very different: we build potentially several dozens of different software components, each time with a different toolchain/environment. So we're almost never rebuilding with the exact same configuration, and therefore using a cache doesn't really work for our use case. It sounds like there's already an appreciable amount of embarassingly parallel work in the environment as none of those builds should depend on any other. Non-parallel autoconf would hurt the latency of a single build but shouldn't cause much change in the throughput of builds one can achieve. My apologies if my experience doesn't line up with others-- my projects take 1-3 minutes to configure, somewhere between 10 minutes and an hour to build, and something like a month of straight computation on hundreds of nodes before I get an answer. Configure times just aren't my bottleneck. I'm not saying parallel configure wouldn't be nice, but that I keep seeing other ways to take advantage of caching and parallelism that don't require making a long-standing serial codebase parallel. - Rhys ___ Autoconf mailing list Autoconf@gnu.org https://lists.gnu.org/mailman/listinfo/autoconf
Re: configure speedup proposal: add --assume-c99 and --assume-posix2008 flags
Dear John Spencer, On Sun, 23 Mar 2014 01:24:54 +0100, John Spencer wrote: there are many configure scripts out there that still check for things that are standard since at least 10 years, and doing this extensively and over and over (people building software themselves usually build more than one package) consumes a lot of time (especially due to the non-parallel nature of configure scripts). As one of the core developer of Buildroot (http://buildroot.org), a tool that automates the process of cross-compiling a potentially significant number of software components to generate embedded Linux systems, I am also very concerned about the speed of configure scripts. On a full build, the time taken by configure scripts is very often around 25% of the overall build time, if not more. The suggestion of using a cache is indeed very interesting. We actually tried it a couple of years ago, with a global cache shared by all packages we build. However, it turned out that several packages use the same autoconf variable name to store the result of tests that are not exactly the same, causing a number of issues. The proposal in the thread to use a pre-defined cache with only a selected number of known-safe entries looks like a good idea. However, what seems like the most important problem to me is the non-parallel nature of autoconf tests. With CPUs going more and more multi-core, the build part of packages (which is usually highly parallel) is going to get faster and faster comparatively to the configure part of packages (which is going to get stuck to a slow speed due to the non-parallel nature of autoconf). Wouldn't it be time to think about moving autoconf to a more parallel logic, where N tests could run in parallel in sub-processes? Thanks, Thomas -- Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com ___ Autoconf mailing list Autoconf@gnu.org https://lists.gnu.org/mailman/listinfo/autoconf
Re: configure speedup proposal: add --assume-c99 and --assume-posix2008 flags
Thomas Petazzoni wrote: Wouldn't it be time to think about moving autoconf to a more parallel logic, where N tests could run in parallel in sub-processes? It's long been past time to do that. Some technical problems need to be solved, but the main problem is that nobody has volunteered to take on the task. ___ Autoconf mailing list Autoconf@gnu.org https://lists.gnu.org/mailman/listinfo/autoconf
Re: configure speedup proposal: add --assume-c99 and --assume-posix2008 flags
Wouldn't it be time to think about moving autoconf to a more parallel logic, where N tests could run in parallel in sub-processes? I only find the configure process to be frustratingly slow when run against compilers with license keys or with checks accessing network storage. Running those in parallel would do nothing but block on the license servers and beat up already sluggish file servers. - Rhys ___ Autoconf mailing list Autoconf@gnu.org https://lists.gnu.org/mailman/listinfo/autoconf
Re: configure speedup proposal: add --assume-c99 and --assume-posix2008 flags
Rhys Ulerich wrote: I only find the configure process to be frustratingly slow when run against compilers with license keys or with checks accessing network storage. That's not my experience at all. Take the bleeding-edge version of GNU grep, say, which I just now happened to be working on. On my platform (RHEL 6.5 x86-64, GCC 4.9.0, dual Xeon E5620), 'configure' takes 32 seconds real-time, whereas 'make -j8' takes 3 seconds. No doubt there are environments where compilation is slow and 'configure' is not the bottleneck, but these days it's veerrry common for 'configure' to be an unreasonably large part of the build process, and where parallelizing would help. ___ Autoconf mailing list Autoconf@gnu.org https://lists.gnu.org/mailman/listinfo/autoconf
Re: configure speedup proposal: add --assume-c99 and --assume-posix2008 flags
On Fri, Jun 6, 2014 at 9:48 AM, Paul Eggert egg...@cs.ucla.edu wrote: veerrry common for 'configure' to be an unreasonably large part of the build process But is an unreasonably large part of a reasonably long build process? You quote a from-scratch build with configure being an order of magnitude slower than parallel compilation. But one just 35 seconds long. Granted, configure time does add up if one builds 10s of packages in sequence. Is there some reason configure caches are unsuitable to gain speedup there? I see almost a six-fold configure improvement on grep-2.20 when its working from a config.cache. That brings configure to 140% of parallel compilation, but we're talking an overall build like 6 seconds. - Rhys ___ Autoconf mailing list Autoconf@gnu.org https://lists.gnu.org/mailman/listinfo/autoconf
Re: configure speedup proposal: add --assume-c99 and --assume-posix2008 flags
Rhys Ulerich wrote: Is there some reason configure caches are unsuitable to gain speedup there? I regularly build with experimental versions in which the 'configure' tests themselves differ from run to run, and where configure caches would give me incorrect results. So I never use caches. I have to wait for an unreasonably long period of time whenever I change anything that requires reconfiguration, and this happens quite frequently to me (often several times per day). one just 35 seconds long. It's a 35-second wait in an edit-build-run cycle, and that's wayy too long. I typically give up and work on something else for a while. The mental context-switching overhead is extremely annoying. I would greatly appreciate it if 'configure' ran faster. ___ Autoconf mailing list Autoconf@gnu.org https://lists.gnu.org/mailman/listinfo/autoconf
Re: configure speedup proposal: add --assume-c99 and --assume-posix2008 flags
I should probably get the configure and build times re-measured for NTP. Feel free to grab http://www.eecis.udel.edu/~ntp/ntp_spool/ntp4/ntp-dev/ntp-dev-4.2.7p444.tar.gz and see for yourself... it's a bit of a monster for something that's about 200k LOC. It would be great to see significant speedup in the configure time, as we have a growing number of buildbot targets and we plan to do both scratch builds as well as upgrade (with autoconf cache) builds as part of our build/test process. -- Harlan Stenn st...@ntp.org http://networktimefoundation.org - be a member! ___ Autoconf mailing list Autoconf@gnu.org https://lists.gnu.org/mailman/listinfo/autoconf
Re: configure speedup proposal: add --assume-c99 and --assume-posix2008 flags
On 03/23/2014 01:24 AM, John Spencer wrote: there are many configure scripts out there that still check for things that are standard since at least 10 years, and doing this extensively and over and over (people building software themselves usually build more than one package) consumes a lot of time (especially due to the non-parallel nature of configure scripts). often configure scripts take longer to finish than compiling the source code itself on decent machines with multiple cpu cores. having an option like --assume-c99 could provide a shortcut so all checks like - have stdint.h - have snprintf() - etc and for posix - sizeof int == 4 - have select() - etc These assumptions are unsafe, esp. on non-mainstream architectures/OSes, which only partially implement c99. E.g. it's not that uncommon to find toolchains for embedded platforms which implement most of C99, but with sizeof int != 4 or without select(). Ralf ___ Autoconf mailing list Autoconf@gnu.org https://lists.gnu.org/mailman/listinfo/autoconf
Re: configure speedup proposal: add --assume-c99 and --assume-posix2008 flags
Ralf Corsepius wrote: On 03/23/2014 01:24 AM, John Spencer wrote: and for posix - sizeof int == 4 - have select() - etc These assumptions are unsafe, esp. on non-mainstream architectures/OSes, which only partially implement c99. E.g. it's not that uncommon to find toolchains for embedded platforms which implement most of C99, but with sizeof int != 4 or without select(). well, noone forces you to use --assume-compliant-c99. as outlined in my previous responses, the usage is *optional* and *not default* and you would only use it on known non-broken platforms (i.e. those that implement C99 to the letter of the standard, for example linux + musl libc). ___ Autoconf mailing list Autoconf@gnu.org https://lists.gnu.org/mailman/listinfo/autoconf
Re: configure speedup proposal: add --assume-c99 and --assume-posix2008 flags
On Sun, 23 Mar 2014, John Spencer wrote: there are many configure scripts out there that still check for things that are standard since at least 10 years, and doing this extensively and over and over (people building software themselves usually build more than one package) consumes a lot of time (especially due to the non-parallel nature of configure scripts). often configure scripts take longer to finish than compiling the source code itself on decent machines with multiple cpu cores. Agreed. having an option like --assume-c99 could provide a shortcut so all checks I don't like any approach which hard-codes assumptions in Autoconf. Hard-coded assumptions can easily be wrong and configure scripts could be used ten years later without being re-generated with the latest rules. The Autoconf configuration cache mechanism (e.g. --config-cache) is very effective. In my test here, using --config-cache reduced repeated configure run time from 20 seconds to 5 seconds. By transcribing selected lines from config.cache to config.site, these cached values can be made the default for a system. Modern autoconf searches for both $prefix/share/config.site and $prefix/etc/config.site and will use whichever ones are found. E.g. egrep '^ac_cv_func_select' config.cache /usr/local/share/config.cache and in /usr/local/share/config.site . /usr/local/share/config.cache In fact, one could take a boilerplate configure.ac, use it to create a bare configure, and then use most of its config.cache to form config.site. With this approach, Autoconf can be optimized for a system with known behavior without placing brittle assumptions within Autoconf itself. A separate script could be used to create a reusable subset of config.cache based on a specification. BTFW that some software might not configure correctly given the presence of a config.site file. In my experience, a config.site file can cause harm when configuring GCC. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ Autoconf mailing list Autoconf@gnu.org https://lists.gnu.org/mailman/listinfo/autoconf
Re: configure speedup proposal: add --assume-c99 and --assume-posix2008 flags
John Spencer maillist-autoc...@barfooze.de writes: having an option like --assume-c99 could provide a shortcut so all checks like - have stdint.h - have snprintf() - etc These are un-alike, just to mention. A surprising number of platforms have an snprintf function that's broken. To test it properly, you need something akin to the gnulib snprintf check, and it's broken in enough places that you may not want to assume the result. Some of the problems with snprintf are also quite serious. For example, Solaris 9 (which I believe is otherwise C99-conforming) would return -1 when given NULL, 0 as the first arguments to snprintf, which makes it impossible to use snprintf safely in many programs. See: https://www.gnu.org/software/gnulib/manual/html_node/snprintf.html for more information about the portability mess. -- Russ Allbery (ea...@eyrie.org) http://www.eyrie.org/~eagle/ ___ Autoconf mailing list Autoconf@gnu.org https://lists.gnu.org/mailman/listinfo/autoconf
Re: configure speedup proposal: add --assume-c99 and --assume-posix2008 flags
Russ Allbery wrote: John Spencer maillist-autoc...@barfooze.de writes: having an option like --assume-c99 could provide a shortcut so all checks like - have stdint.h - have snprintf() - etc These are un-alike, just to mention. A surprising number of platforms have an snprintf function that's broken. To test it properly, you need something akin to the gnulib snprintf check, and it's broken in enough places that you may not want to assume the result. Some of the problems with snprintf are also quite serious. For example, Solaris 9 (which I believe is otherwise C99-conforming) would return -1 when given NULL, 0 as the first arguments to snprintf, which makes it impossible to use snprintf safely in many programs. See: https://www.gnu.org/software/gnulib/manual/html_node/snprintf.html for more information about the portability mess. point taken, although all the platforms described there are not exactly what i would call modern, my leaning would be more towards calling them broken, obsolete or simply undead. thinking_loud i even ask myself sometimes if it is still worth plastering software and configure scripts with checks and ifdefs for these broken platforms that probably cant even build the dependencies of modern software (these ifdefs are almost always completely unmaintained due to lack of testing and very likely to fail), and even if they still compile *and* work there, are probably not even used. because the handful of people using these platforms stick to their current versions anyway... speaking of it, it may be a good test to revive one of these zombie OS's trying to do a full build of the 500 most popular non-linux-specific packages in debian to proof that probably less than 10% will compile despite extensive platform-specific hacks and workarounds. carrying around all this legacy baggage is kinda like porting a 16bit DOS program to win 3.11, then to win95, 98, xp, 7, 8 and still keeping all the old 16bit code in #ifdefs without testing it ever. btw, to this day i've yet to see a library or program, that for example checks for unistd.h, and then uses HAVE_UNISTD_H consequently in all TUs, including optional usage of any of its exported functions. this probably worked once, when the initial autoconf support was written and tested, but the next developer that came along just used unistd.h without any conditionals... /thinking_loud note that my suggestion is not to default to this shortcut, it's meant to be used by users that know that their platform is fully compliant, so it wouldnt affect users of these systems. but maybe to make it more clear, the option should be called --assume-c99-compliance instead. --JS ___ Autoconf mailing list Autoconf@gnu.org https://lists.gnu.org/mailman/listinfo/autoconf
Re: configure speedup proposal: add --assume-c99 and --assume-posix2008 flags
Bob Friesenhahn wrote: On Sun, 23 Mar 2014, John Spencer wrote: there are many configure scripts out there that still check for things that are standard since at least 10 years, and doing this extensively and over and over (people building software themselves usually build more than one package) consumes a lot of time (especially due to the non-parallel nature of configure scripts). often configure scripts take longer to finish than compiling the source code itself on decent machines with multiple cpu cores. Agreed. having an option like --assume-c99 could provide a shortcut so all checks I don't like any approach which hard-codes assumptions in Autoconf. Hard-coded assumptions can easily be wrong and configure scripts could be used ten years later without being re-generated with the latest rules. The Autoconf configuration cache mechanism (e.g. --config-cache) is very effective. In my test here, using --config-cache reduced repeated configure run time from 20 seconds to 5 seconds. By transcribing selected lines from config.cache to config.site, these cached values can be made the default for a system. Modern autoconf searches for both $prefix/share/config.site and $prefix/etc/config.site and will use whichever ones are found. yes, i currently use an approach that copies a custom config.cache into the builddir. however this implicates a maintenance burden, and i heard some voices criticizing that approach as to unreliable. using a config.site in place of my manual approach is interesting and may make it look a little bit more legitimate, however it's essentially the same. what i'm trying to achieve is that there's some kind of official upstream stategy (like currently the config.site usage) that could be used/consumed easily (i.e. without having to manually maintain the cachefile). E.g. egrep '^ac_cv_func_select' config.cache /usr/local/share/config.cache and in /usr/local/share/config.site . /usr/local/share/config.cache In fact, one could take a boilerplate configure.ac, use it to create a bare configure, and then use most of its config.cache to form config.site. this is a good idea. maybe autoconf could ship a config.site generator script which tests the libc capabilities in regard to c99 and posix 2008, and generates the cache for usage in a systemwide install? testing anything else like for example whether ncurses.h is installed is imo bogus, as the user could easily remove ncurses-dev from his box after having run through the .site-generator script. otoh it's unlikely that his libc implementation removes functionality that it already had. despite that, the script should definitely display a warning that it should be rerun whenever the libc was updated. having this script available as part of autoconf would make it easily accessible to distribution maintainers, so everyone could benefit. With this approach, Autoconf can be optimized for a system with known behavior without placing brittle assumptions within Autoconf itself. A separate script could be used to create a reusable subset of config.cache based on a specification. BTFW that some software might not configure correctly given the presence of a config.site file. In my experience, a config.site file can cause harm when configuring GCC. i didnt experience this sort of problem with gcc, but i had some cases where the information in my handtuned config.cache was outdated and caused build failures. --JS ___ Autoconf mailing list Autoconf@gnu.org https://lists.gnu.org/mailman/listinfo/autoconf
Re: configure speedup proposal: add --assume-c99 and --assume-posix2008 flags
On Sun, Mar 23, 2014 at 12:24 AM, John Spencer maillist-autoc...@barfooze.de wrote: there are many configure scripts out there that still check for things that are standard since at least 10 years, and doing this extensively and over and over (people building software themselves usually build more than one package) consumes a lot of time (especially due to the non-parallel nature of configure scripts). often configure scripts take longer to finish than compiling the source code itself on decent machines with multiple cpu cores. having an option like --assume-c99 could provide a shortcut so all checks like - have stdint.h - have snprintf() - etc and for posix - sizeof int == 4 - have select() - etc could be skipped and assumed to be always true. If changing the configure scripts in question is an option, developers could simply require C99 with AC_PROG_CC_C99 if they rely on many characteristics of C99. autoscan suggests tests; some of these might be unnecessary or included in broader tests. ___ Autoconf mailing list Autoconf@gnu.org https://lists.gnu.org/mailman/listinfo/autoconf
Re: configure speedup proposal: add --assume-c99 and --assume-posix2008 flags
On Sun, 23 Mar 2014, Bob Friesenhahn wrote: On Sun, 23 Mar 2014, John Spencer wrote: having an option like --assume-c99 could provide a shortcut so all checks I don't like any approach which hard-codes assumptions in Autoconf. Hard-coded assumptions can easily be wrong and configure scripts could be used ten years later without being re-generated with the latest rules. Agreed. An example is xz-5.0.5. Configure errors out complaining of no c99 compiler but after fooling configure the package builds fine using a c89 compiler. -- Tim RiceMultitalents t...@multitalents.net ___ Autoconf mailing list Autoconf@gnu.org https://lists.gnu.org/mailman/listinfo/autoconf
Re: configure speedup proposal: add --assume-c99 and --assume-posix2008 flags
On Sun, 23 Mar 2014, John Spencer wrote: i even ask myself sometimes if it is still worth plastering software and configure scripts with checks and ifdefs for these broken platforms that probably cant even build the dependencies of modern software (these ifdefs are almost always completely unmaintained due to lack of testing and very likely to fail), and even if they still compile *and* work there, are probably not even used. because the handful of people using these platforms stick to their current versions anyway... It seems likely that you are not a user of one of these platforms and therefore not really qualified to characterize them. Most of them are not broken platforms since they work just fine. People who still use old systems do so because newer operating system versions don't run on (or run well on) their hardware. It is up to package developers if they care to ensure that their software runs on something other than a very recent OS version. speaking of it, it may be a good test to revive one of these zombie OS's trying to do a full build of the 500 most popular non-linux-specific packages in debian to proof that probably less than 10% will compile despite extensive platform-specific hacks and workarounds. carrying around all this legacy baggage is kinda like porting a 16bit DOS program to win 3.11, then to win95, 98, xp, 7, 8 and still keeping all the old 16bit code in #ifdefs without testing it ever. You are blowing the magnitude of the legacy baggage way out of proportion. In fact, there is really not much left compared to what there used to be. Please take care because most of the software you are using was developed on one of these legacy systems and worked just fine when it was developed (and most still does now). Many people here developed those systems that you describe as broken now. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ Autoconf mailing list Autoconf@gnu.org https://lists.gnu.org/mailman/listinfo/autoconf
configure speedup proposal: add --assume-c99 and --assume-posix2008 flags
there are many configure scripts out there that still check for things that are standard since at least 10 years, and doing this extensively and over and over (people building software themselves usually build more than one package) consumes a lot of time (especially due to the non-parallel nature of configure scripts). often configure scripts take longer to finish than compiling the source code itself on decent machines with multiple cpu cores. having an option like --assume-c99 could provide a shortcut so all checks like - have stdint.h - have snprintf() - etc and for posix - sizeof int == 4 - have select() - etc could be skipped and assumed to be always true. only functions which are known to be non-conforming on some platforms (like the broken strerror_r() semantics of GLIBC which conflict with POSIX when certain CPPFLAGS are used), and special platform specific things like linux extensions needed to be checked separately using this mechanism. to me, this sounds like a simple way to make configure runs at least 50% faster on most modern platforms, while not impacting legacy systems negatively at all. having configure run much faster could also lead to a better public perception of autoconf in general. as for how to implement, maybe these flags could just set the appropriate autoconf cache variables to true. regards, --JS ___ Autoconf mailing list Autoconf@gnu.org https://lists.gnu.org/mailman/listinfo/autoconf
