[PATCH] libtool: minimise forks per invocation on cygwin and mingw.
The recently pushed series of patches included the controversial
introduction of an additional 3 forks per invocation, which might
add a minute or two of wall-clock time to giant builds on windows.
By assuming that windows will run shell scripts on some shell with
all the modern optional features that libtool wants, this patch
eliminates even those 3 new forks.
Okay to push?
* build-aux/general.m4sh (lt_HAVE_PLUSEQ_OP, lt_HAVE_ARITH_OP)
(lt_HAVE_XSI_OPS) [cygwin, mingw]: Set these without a test on
the assumption that a modern shell (i.e. bash) will be used to
run libtool and libtoolize.
Signed-off-by: Gary V. Vaughan g...@gnu.org
---
build-aux/general.m4sh | 13 +
1 files changed, 13 insertions(+), 0 deletions(-)
diff --git a/build-aux/general.m4sh b/build-aux/general.m4sh
index 252b2d2..2ac6238 100644
--- a/build-aux/general.m4sh
+++ b/build-aux/general.m4sh
@@ -56,6 +56,7 @@ test ${ECHO+set} = set || ECHO=${as_echo-'printf %s\n'}
: ${SED=@SED@}
: ${SHELL=${CONFIG_SHELL-/bin/sh}}
: ${Xsed=$SED -e 1s/^X//}
+: ${host=@host_triplet@}
# Global variables:
EXIT_SUCCESS=0
@@ -74,6 +75,18 @@ dirname='s|/[^/]*$||'
basename='s|^.*/||'
+# Forks are unreasonably slow under Windows, so we assume that, for at
+# least cygwin and mingw, /bin/sh is bash, and save at least 3 forks per
+# invocation:
+case $host in
+ *cygwin* | *mingw*)
+test -n $lt_HAVE_PLUSEQ_OP || lt_HAVE_PLUSEQ_OP=yes
+test -n $lt_HAVE_ARITH_OP || lt_HAVE_ARITH_OP=yes
+test -n $lt_HAVE_XSI_OPS || lt_HAVE_XSI_OPS=yes
+;;
+esac
+
+
# lt_HAVE_PLUSEQ_OP
# Can be empty, in which case the shell is probed, yes if += is useable
# or anything else if += does not work.
--
1.7.8
Cheers,
--
Gary V. Vaughan (gary AT gnu DOT org)
Re: [PATCH] libtool: minimise forks per invocation on cygwin and mingw.
On 12/08/2011 03:21 AM, Gary V. Vaughan wrote:
The recently pushed series of patches included the controversial
introduction of an additional 3 forks per invocation, which might
add a minute or two of wall-clock time to giant builds on windows.
By assuming that windows will run shell scripts on some shell with
all the modern optional features that libtool wants, this patch
eliminates even those 3 new forks.
Okay to push?
I'm a bit reluctant to do this via a host check;
+# Forks are unreasonably slow under Windows, so we assume that, for at
+# least cygwin and mingw, /bin/sh is bash, and save at least 3 forks per
+# invocation:
+case $host in
+ *cygwin* | *mingw*)
Instead of doing it this way, I'd almost rather see:
if test ${BASH_VERSION+set} = set; then
although if cygwin ever follows debian's lead of using dash for faster
/bin/sh, I'm not sure if there is a reliable forkless way to detect dash.
--
Eric Blake ebl...@redhat.com+1-919-301-3266
Libvirt virtualization library http://libvirt.org
signature.asc
Description: OpenPGP digital signature
Re: [PATCH] libtool: minimise forks per invocation on cygwin and mingw.
Hi Peter, On 8 Dec 2011, at 20:40, Peter O'Gorman wrote: On 12/08/2011 04:21 AM, Gary V. Vaughan wrote: The recently pushed series of patches included the controversial introduction of an additional 3 forks per invocation, which might add a minute or two of wall-clock time to giant builds on windows. By assuming that windows will run shell scripts on some shell with all the modern optional features that libtool wants, this patch eliminates even those 3 new forks. Okay to push? The whole reason these checks were done in configure and not in libtool was to avoid these forks, and the slow down involved. We did a lot of work at speeding up libtool so that non-portable tools such as dolt would not be necessary, including moving large chunks of compile mode script closer the the top of libtool so that it would execute faster. Ack. However since I wrote all the hairy M4 code to do the configure time replacements, I've changed my mind about how well they work in the grand scheme of things... The usual mode of integrating libtool is to have your project's configure build a libtool script, which you then call as a compiler wrapper. With the exception of projects with a huge number of tiny files to compile, I seriously doubt that 3 additional forks per invocation would be noticable as it stands now, and after looking back through the replacement code I recently expunged, we probably save a few hundred forks from sed and perl invocations at configure time, which would mean you don't get any slow down overall in a './configure; make; make install' sequence until you've invoked the libtool script at least several dozen times. Also, I'm trying hard to make hacking on libtool much less infuriating to attract more outside patches and development. Having configure switch out function definitions is an odd thing to do, and libtool is already hard enough to understand without having to worry about having configure rewriting some of libtools functions... hopefully it doesn't make any mistakes ;) Any additional forks will slow down the script and should be avoided on all platforms. Agreed. And using a BASH_VERSION test will get us a good way there. I'll also work on a follow up patch to set the new lt_HAVE_XSI_OPS and friends using the existing machinery for injecting configure time variables into libtool, so I can also throw out the last couple of M4 replacement macros. Simply removing $(SHELL) from the LIBTOOL variable should fix the bug that this is attempting to fix, without slowing down libtool. But then we go back to having bug reports about libtool not getting an execute bit. I'm not in favour of making that change. Cheers, -- Gary V. Vaughan (gary AT gnu DOT org)
Re: [PATCH] libtool: minimise forks per invocation on cygwin and mingw.
Hi Eric,
On 8 Dec 2011, at 19:56, Eric Blake wrote:
On 12/08/2011 03:21 AM, Gary V. Vaughan wrote:
The recently pushed series of patches included the controversial
introduction of an additional 3 forks per invocation, which might
add a minute or two of wall-clock time to giant builds on windows.
By assuming that windows will run shell scripts on some shell with
all the modern optional features that libtool wants, this patch
eliminates even those 3 new forks.
Okay to push?
I'm a bit reluctant to do this via a host check;
+# Forks are unreasonably slow under Windows, so we assume that, for at
+# least cygwin and mingw, /bin/sh is bash, and save at least 3 forks per
+# invocation:
+case $host in
+ *cygwin* | *mingw*)
Instead of doing it this way, I'd almost rather see:
if test ${BASH_VERSION+set} = set; then
Face palm! Absolutely, that is far more sensible. Assuming we decide
to push this patch, I'll do it that way and ditch the host check. Thanks!
although if cygwin ever follows debian's lead of using dash for faster
/bin/sh, I'm not sure if there is a reliable forkless way to detect dash.
I think we can worry about that later, if it ever happens.
Cheers,
--
Gary V. Vaughan (gary AT gnu DOT org)
Re: [PATCH] libtool: minimise forks per invocation on cygwin and mingw.
On Thu, 8 Dec 2011, Gary V. Vaughan wrote:
Instead of doing it this way, I'd almost rather see:
if test ${BASH_VERSION+set} = set; then
Face palm! Absolutely, that is far more sensible. Assuming we decide
to push this patch, I'll do it that way and ditch the host check. Thanks!
Is the value of this variable inheritable by subordinate shells, or is
it an internal shell variable which would never be exported to
subordinate shells?
Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: [PATCH] libtool: minimise forks per invocation on cygwin and mingw.
On 12/08/2011 08:04 AM, Bob Friesenhahn wrote:
On Thu, 8 Dec 2011, Gary V. Vaughan wrote:
Instead of doing it this way, I'd almost rather see:
if test ${BASH_VERSION+set} = set; then
Face palm! Absolutely, that is far more sensible. Assuming we decide
to push this patch, I'll do it that way and ditch the host check.
Thanks!
Is the value of this variable inheritable by subordinate shells, or is
it an internal shell variable which would never be exported to
subordinate shells?
By default, bash sets up $BASH_VERSION as an internal variable, and not
exported. It is unwise for users (or scripts) to export BASH_VERSION to
child processes.
--
Eric Blake ebl...@redhat.com+1-919-301-3266
Libvirt virtualization library http://libvirt.org
signature.asc
Description: OpenPGP digital signature
Re: [PATCH] libtool: minimise forks per invocation on cygwin and mingw.
On 12/8/2011 5:21 AM, Gary V. Vaughan wrote: The recently pushed series of patches included the controversial introduction of an additional 3 forks per invocation, which might add a minute or two of wall-clock time to giant builds on windows. By assuming that windows will run shell scripts on some shell with all the modern optional features that libtool wants, this patch eliminates even those 3 new forks. Okay to push? Has anybody done a comparison between: cygwin + libtool + dash/posh (e.g. small, fast shell -- without XSI) cygwin + libtool + bash (e.g. big bloated slow shell -- with XSI) to see which is better? -- Chuck
Re: [PATCH] libtool: minimise forks per invocation on cygwin and mingw.
On Thu, 8 Dec 2011, Peter O'Gorman wrote: Any additional forks will slow down the script and should be avoided on all platforms. I definitely agree with that. Besides the Windows problem, it does not seem like fork performance improves linearly from adding processor cores so it is important to minimize forks so that parallel compiles don't hit an bottleneck. Simply removing $(SHELL) from the LIBTOOL variable should fix the bug that this is attempting to fix, without slowing down libtool. Will this approach work ok if the Cygwin/MinGW user is using something like zsh or dash rather than bash? Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: [PATCH] libtool: minimise forks per invocation on cygwin and mingw.
On 12/08/2011 08:29 AM, Charles Wilson wrote:
On 12/8/2011 5:21 AM, Gary V. Vaughan wrote:
The recently pushed series of patches included the controversial
introduction of an additional 3 forks per invocation, which might
add a minute or two of wall-clock time to giant builds on windows.
By assuming that windows will run shell scripts on some shell with
all the modern optional features that libtool wants, this patch
eliminates even those 3 new forks.
Okay to push?
Has anybody done a comparison between:
cygwin + libtool + dash/posh (e.g. small, fast shell -- without XSI)
Umm, dash has XSI features (where XSI features covers things like
${var##prefix}). It is only shells like Solaris /bin/sh that lack this
mandatory POSIX feature. Meanwhile, libtool is using more than just XSI
extensions; for example, it is probing for bash's += variable append
extension. I'm not sure how much difference += makes (especially since
it is not shaving on forks, but is reducing O(n^2) malloc behavior for
large piece-wise constructions), but do know that XSI variable usage
definitely shaves a lot of forkes.
As for actual timing comparisons, I have not done any recently.
--
Eric Blake ebl...@redhat.com+1-919-301-3266
Libvirt virtualization library http://libvirt.org
signature.asc
Description: OpenPGP digital signature
Re: [PATCH] libtool: minimise forks per invocation on cygwin and mingw.
On 12/8/2011 11:22 AM, Eric Blake wrote:
On 12/08/2011 08:29 AM, Charles Wilson wrote:
cygwin + libtool + dash/posh (e.g. small, fast shell -- without XSI)
Umm, dash has XSI features (where XSI features covers things like
${var##prefix}). ... Meanwhile, libtool is using more than just XSI
extensions; for example, it is probing for bash's += variable append
extension.
Oh, I didn't realize. I was primarily thinking about += -- I thought it
was one of the XSI extensions, not a bash-specific thing, and I knew
dash did not support it.
--
Chuck
Re: [PATCH] libtool: minimise forks per invocation on cygwin and mingw.
On 12/08/2011 09:29 AM, Charles Wilson wrote:
Has anybody done a comparison between:
cygwin + libtool + dash/posh (e.g. small, fast shell -- without XSI)
cygwin + libtool + bash (e.g. big bloated slow shell -- with XSI)
to see which is better?
Because I installed mingw32 yesterday on my rarely used windows vista
home basic VM, I thought I'd do my compile an empty file 100 times test
on it.
With todays git, bash:
time (for x in {1..100}; do lt_HAVE_XSI_OPS=yes lt_HAVE_PLUSEQ_OP=yes
lt_HAVE_ARITH_OP=yes ./libtool --mode=compile --tag=CC gcc -c -o a.lo
a.c; done)
real0m58.766s
user0m5.315s
sys 0m31.981s
real0m54.413s
user0m5.380s
sys 0m31.143s
With dash
time (for x in {1..100}; do lt_HAVE_XSI_OPS=yes lt_HAVE_PLUSEQ_OP=no
lt_HAVE_ARITH_OP=yes dash ./libtool --mode=compile --tag=CC gcc -c -o
a.lo a.c; done)
real0m32.089s
user0m1.657s
sys 0m9.330s
real0m29.905s
user0m1.637s
sys 0m9.125s
I think dash might be the winner.
Peter
