Re: [PATCH 3/4] maint: pick XSI funcs at runtime, not configure time.
On 28 Nov 2011, at 21:44, Bob Friesenhahn wrote: > On Mon, 28 Nov 2011, Peter Rosin wrote: >> >> My typical use case is "mid-sized" at a magnitude or so larger, and >> even there with a fork rate of approx 10-15 Hz as I'm seeing, it wouldn't >> be too harsh with a couple of extra forks - a minutes or so on the >> wall clock time. But it would really add to the pain on some >> (hypothetical?) large project with thousands of libtool invocations. >> That's all I'm saying, but *I* am not building any of those... > > Is Windows "fork" performance observed to improve linearly as processor cores > are added, or does it maintain pretty much a fixed rate? If it does not > improve linearly as processor cores are added, then the extra forks will > severely impact available performance of parallel builds. > > I have become used to seeing substantial speedup with 'make -j 4' on a > Windows system with four cores. I just pushed the series containing this patch, but refactored to add only 3 additional forks per invocation, and with a patch to follow which might be good enough to eliminate even those 3 forks on windows machines. Cheers, -- Gary V. Vaughan (gary AT gnu DOT org)
Re: [PATCH 3/4] maint: pick XSI funcs at runtime, not configure time.
On Mon, 28 Nov 2011, Peter Rosin wrote: My typical use case is "mid-sized" at a magnitude or so larger, and even there with a fork rate of approx 10-15 Hz as I'm seeing, it wouldn't be too harsh with a couple of extra forks - a minutes or so on the wall clock time. But it would really add to the pain on some (hypothetical?) large project with thousands of libtool invocations. That's all I'm saying, but *I* am not building any of those... Is Windows "fork" performance observed to improve linearly as processor cores are added, or does it maintain pretty much a fixed rate? If it does not improve linearly as processor cores are added, then the extra forks will severely impact available performance of parallel builds. I have become used to seeing substantial speedup with 'make -j 4' on a Windows system with four cores. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: [PATCH 3/4] maint: pick XSI funcs at runtime, not configure time.
Hi Peter, On 28 Nov 2011, at 18:05, Peter Rosin wrote: > Gary V. Vaughan skrev 2011-11-28 10:20: >> On 28 Nov 2011, at 15:48, Peter Rosin wrote: >>> Gary V. Vaughan skrev 2011-11-25 09:57: Determine, on a function by function basis, what XSI features are available in the shell that is actually running the script, rather than the one that was picked at configure time by the re-execution engine. >>> >>> Doesn't this mean that the libtool script will do a number of >>> extra forks each time it is run? >> >> Well, yes and no. There will be an extra 4 or 8 evals (depending >> on whether the XSI function is chosen) as libtool starts up when > > That's the evals in sub-shells, right? Those with this pattern: > > if $_use_fast_funcs && (eval ...) 2>/dev/null > then > eval '... > > or this: > > if (eval ...) 2>/dev/null > then > eval '... Yes, that's them. >> [[snip]] >>> If that's the case, I would like >>> to know how that impacts the execution time on Cygwin/MinGW... >> >> I don't have access to Windows so I can't actually measure it, but >> I doubt that the difference will be noticeable for even a large >> project, and quite possibly a small project like libltdl that only >> runs libtool a hand full of times will actually be (imperceptibly) >> faster. > > I think you are underestimating the Cygwin fork pain. > > BTW, libltdl runs libtool more like 20 times, but I agree it's very > small and the number of forks will not matter much. > > My typical use case is "mid-sized" at a magnitude or so larger, and > even there with a fork rate of approx 10-15 Hz as I'm seeing, it wouldn't > be too harsh with a couple of extra forks - a minutes or so on the > wall clock time. But it would really add to the pain on some > (hypothetical?) large project with thousands of libtool invocations. > That's all I'm saying, but *I* am not building any of those... Maybe I should switch the sense of the _use_fast_funcs, so that the pattern becomes: if $libtool_force_fast_funcs || (eval ...) 2>/dev/null then eval '... And then recommend that cygwin users always use an XSI and += capable shell, with libtool_force_fast_funcs=: in the environment to cut out the forks? > Hmm, looking closer, shouldn't _use_fast_funcs have some kind of lt > prefix? Good point. I'll add a prefix before pushing. Thanks. Cheers, -- Gary V. Vaughan (gary AT gnu DOT org)
Re: [PATCH 3/4] maint: pick XSI funcs at runtime, not configure time.
Gary V. Vaughan skrev 2011-11-28 10:20: > Hi Peter, > > On 28 Nov 2011, at 15:48, Peter Rosin wrote: >> Gary V. Vaughan skrev 2011-11-25 09:57: >>> Determine, on a function by function basis, what XSI features >>> are available in the shell that is actually running the script, >>> rather than the one that was picked at configure time by the >>> re-execution engine. >> >> Doesn't this mean that the libtool script will do a number of >> extra forks each time it is run? > > Well, yes and no. There will be an extra 4 or 8 evals (depending > on whether the XSI function is chosen) as libtool starts up when That's the evals in sub-shells, right? Those with this pattern: if $_use_fast_funcs && (eval ...) 2>/dev/null then eval '... or this: if (eval ...) 2>/dev/null then eval '... > compared to 2.4.2, but the fast_funcs will each save a fork or > two on every invocation compared to if libtool configure found a non- > XSI shell when libtool was installed. However, don't forget that > almost always, libtool is generated by the configure script of the > project that uses it, and the loss of seds, evals, and file renames > at configure time to splice in the fast_funcs most likely means > you'll be at a net win on total execution time for at least a couple > of dozen libtool invocations. > > And let's not forget this actually fixes a bug... we've had reports > of configure re-execing with a good shell (say /opt/bin/bash) and > substituting in the XSI fast_funcs, but the libtool script doesn't > have all the slow re-exec machinery and runs with /bin/sh which then > chokes on the XSI extensions spliced in to libtool, unless the > caller manually sets SHELL and CONFIG_SHELL in their environment > before calling make (which is a bad idea for other reasons). > >> If that's the case, I would like >> to know how that impacts the execution time on Cygwin/MinGW... > > I don't have access to Windows so I can't actually measure it, but > I doubt that the difference will be noticeable for even a large > project, and quite possibly a small project like libltdl that only > runs libtool a hand full of times will actually be (imperceptibly) > faster. I think you are underestimating the Cygwin fork pain. BTW, libltdl runs libtool more like 20 times, but I agree it's very small and the number of forks will not matter much. My typical use case is "mid-sized" at a magnitude or so larger, and even there with a fork rate of approx 10-15 Hz as I'm seeing, it wouldn't be too harsh with a couple of extra forks - a minutes or so on the wall clock time. But it would really add to the pain on some (hypothetical?) large project with thousands of libtool invocations. That's all I'm saying, but *I* am not building any of those... Hmm, looking closer, shouldn't _use_fast_funcs have some kind of lt prefix? Cheers, Peter
Re: [PATCH 3/4] maint: pick XSI funcs at runtime, not configure time.
Hi Peter, On 28 Nov 2011, at 15:48, Peter Rosin wrote: > Gary V. Vaughan skrev 2011-11-25 09:57: >> Determine, on a function by function basis, what XSI features >> are available in the shell that is actually running the script, >> rather than the one that was picked at configure time by the >> re-execution engine. > > Doesn't this mean that the libtool script will do a number of > extra forks each time it is run? Well, yes and no. There will be an extra 4 or 8 evals (depending on whether the XSI function is chosen) as libtool starts up when compared to 2.4.2, but the fast_funcs will each save a fork or two on every invocation compared to if libtool configure found a non- XSI shell when libtool was installed. However, don't forget that almost always, libtool is generated by the configure script of the project that uses it, and the loss of seds, evals, and file renames at configure time to splice in the fast_funcs most likely means you'll be at a net win on total execution time for at least a couple of dozen libtool invocations. And let's not forget this actually fixes a bug... we've had reports of configure re-execing with a good shell (say /opt/bin/bash) and substituting in the XSI fast_funcs, but the libtool script doesn't have all the slow re-exec machinery and runs with /bin/sh which then chokes on the XSI extensions spliced in to libtool, unless the caller manually sets SHELL and CONFIG_SHELL in their environment before calling make (which is a bad idea for other reasons). > If that's the case, I would like > to know how that impacts the execution time on Cygwin/MinGW... I don't have access to Windows so I can't actually measure it, but I doubt that the difference will be noticeable for even a large project, and quite possibly a small project like libltdl that only runs libtool a hand full of times will actually be (imperceptibly) faster. Cheers, -- Gary V. Vaughan (gary AT gnu DOT org)
Re: [PATCH 3/4] maint: pick XSI funcs at runtime, not configure time.
Gary V. Vaughan skrev 2011-11-25 09:57: > Determine, on a function by function basis, what XSI features > are available in the shell that is actually running the script, > rather than the one that was picked at configure time by the > re-execution engine. Doesn't this mean that the libtool script will do a number of extra forks each time it is run? If that's the case, I would like to know how that impacts the execution time on Cygwin/MinGW... Cheers, Peter
[PATCH 3/4] maint: pick XSI funcs at runtime, not configure time.
Determine, on a function by function basis, what XSI features
are available in the shell that is actually running the script,
rather than the one that was picked at configure time by the
re-execution engine.
* m4/libtool.m4 (_LT_PROG_FUNCTION_REPLACE)
(_LT_PROG_REPLACE_SHELLFNS): Remove.
(_LT_CHECK_SHELL_FEATURES): Remove tests for XSI and += support.
* build-aux/general.m4sh (func_append, func_append_quoted)
(func_arith, func_len, func_basename, func_dirname)
(func_dirname_and_basename, func_stripname): List both enhanced
and fallback implementations, and choose one based on whether
the enhanced features required by the fastest version are
available.
* build-aux/getopt.m4sh (func_split_short_opt)
(func_split_long_opt): Ditto.
* build-aux/ltmain.m4sh (func_lo2o, func_xform): Ditto.
* tests/getopt-m4sh.at: Add tests for enhanced and fallback
implementations of func_arith, func_len, func_stripname and
func_dirname_and_basename.
* NEWS: Updated.
Signed-off-by: Gary V. Vaughan
---
NEWS |4 +
build-aux/general.m4sh | 162 ---
build-aux/getopt.m4sh | 38 ++--
build-aux/ltmain.m4sh | 33 --
m4/libtool.m4 | 117 +--
tests/getopt-m4sh.at | 116 --
6 files changed, 256 insertions(+), 214 deletions(-)
diff --git a/NEWS b/NEWS
index 0479a34..ecd4fa1 100644
--- a/NEWS
+++ b/NEWS
@@ -49,6 +49,10 @@ NEWS - list of user-visible changes between releases of GNU
Libtool
mode is selected; `--standalone' never worked, and is no longer
accepted.
- Libtool and libtoolize no longer choke on paths with a comma in them.
+ - In the case where $SHELL does not have the same enhanced features
+(e.g. the ability to parse `var+=append') as $CONFIG_SHELL, libtool
+will now correctly fallback to using only vanilla shell features
+instead of failing with a parse at startup.
** Important incompatible changes:
diff --git a/build-aux/general.m4sh b/build-aux/general.m4sh
index eeda222..648315d 100644
--- a/build-aux/general.m4sh
+++ b/build-aux/general.m4sh
@@ -73,63 +73,114 @@ IFS=" $lt_nl"
dirname='s|/[^/]*$||'
basename='s|^.*/||'
-
-# func_append var value
-# Append VALUE to the end of shell variable VAR.
-func_append ()
-{
+: ${_use_fast_funcs=":"}
+
+if $_use_fast_funcs && (eval 'x=a; x+=" b"; test "a b" = "$x"') 2>/dev/null
+then
+ # func_append var value
+ # Append VALUE to the end of shell variable VAR.
+ eval 'func_append ()
+ {
+eval "$1+=\$2"
+ }'
+
+ # func_append_quoted var value
+ # Quote VALUE and append to the end of shell variable VAR, separated
+ # by a space.
+ eval 'func_append_quoted ()
+ {
+func_quote_for_eval "$2"
+eval "$1+=\\ \$func_quote_for_eval_result"
+ }'
+else
+ func_append ()
+ {
eval "$1=\$$1\$2"
-} # func_append may be replaced by extended shell implementation
+ }
-
-# func_append_quoted var value
-# Quote VALUE and append to the end of shell variable VAR, separated
-# by a space.
-func_append_quoted ()
-{
+ func_append_quoted ()
+ {
func_quote_for_eval "$2"
eval "$1=\$$1\\ \$func_quote_for_eval_result"
-} # func_append_quoted may be replaced by extended shell implementation
+ }
+fi
# func_arith arithmetic-term...
-func_arith ()
-{
+if $_use_fast_funcs && (eval 'x=$(( 1 + 1 )); test 2 = "$x"') 2>/dev/null
+then
+ eval 'func_arith ()
+ {
+func_arith_result=$(( $* ))
+ }'
+else
+ func_arith ()
+ {
func_arith_result=`expr "$@"`
-} # func_arith may be replaced by extended shell implementation
+ }
+fi
# func_len string
# STRING may not start with a hyphen.
-func_len ()
-{
+if $_use_fast_funcs && (eval 'x=12; test 2 = "${#x}"') 2>/dev/null
+then
+ eval 'func_len ()
+ {
+func_len_result=${#1}
+ }'
+else
+ func_len ()
+ {
func_len_result=`expr "$1" : ".*" 2>/dev/null || echo $max_cmd_len`
-} # func_len may be replaced by extended shell implementation
+ }
+fi
-# func_dirname file append nondir_replacement
-# Compute the dirname of FILE. If nonempty, add APPEND to the result,
-# otherwise set result to NONDIR_REPLACEMENT.
-func_dirname ()
-{
-func_dirname_result=`$ECHO "$1" | $SED "$dirname"`
-if test "X$func_dirname_result" = "X$1"; then
- func_dirname_result=$3
-else
- func_append func_dirname_result $2
-fi
-} # func_dirname may be replaced by extended shell implementation
+if $_use_fast_funcs \
+&& (eval 'x=a/x/b; a=${x%/*}; b=${x##*/}; test ba/x = "$a$b"') 2>/dev/null
+then
+ # If this shell supports suffix pattern removal, then use it to avoid
+ # forking. Hide the definitions single quotes in case the shell chokes
+ # on unsupported syntax...
+
+ _b='func_basename_result=${1##*/}'
+ _d='case $1 in
+*/*) func_dirname_result=${1%/*}$2 ;;
+* ) func_dirname_result=$3;;
+ esac'
+
+else
+ # ...otherwise fall back to using sed.
+
+ _b='func_basename_result=
