compile mode, XSI shell: do not fork for $xform
OK to apply?
Thanks,
Ralf
2008-04-10 Ralf Wildenhues [EMAIL PROTECTED]
* libltdl/m4/libtool.m4 (_LT_PROG_XSI_SHELLFNS) func_xform:
New function.
* libltdl/config/ltmain.m4sh (func_mode_compile): Use it to
avoid two forks with XSI shells.
Index: libltdl/config/ltmain.m4sh
===
RCS file: /cvsroot/libtool/libtool/libltdl/config/ltmain.m4sh,v
retrieving revision 1.102
diff -u -r1.102 ltmain.m4sh
--- libltdl/config/ltmain.m4sh 16 Mar 2008 08:34:25 - 1.102
+++ libltdl/config/ltmain.m4sh 9 Apr 2008 23:20:40 -
@@ -1361,27 +1361,16 @@
# Recognize several different file suffixes.
# If the user specifies -o file.o, it is replaced with file.lo
-xform='[cCFSifmso]'
case $libobj in
-*.ada) xform=ada ;;
-*.adb) xform=adb ;;
-*.ads) xform=ads ;;
-*.asm) xform=asm ;;
-*.c++) xform=c++ ;;
-*.cc) xform=cc ;;
-*.ii) xform=ii ;;
-*.class) xform=class ;;
-*.cpp) xform=cpp ;;
-*.cxx) xform=cxx ;;
-*.[fF][09]?) xform='[fF][09].' ;;
-*.for) xform=for ;;
-*.java) xform=java ;;
-*.obj) xform=obj ;;
-*.sx) xform=sx ;;
+*.[cCFSifmso] | \
+*.ada | *.adb | *.ads | *.asm | \
+*.c++ | *.cc | *.ii | *.class | *.cpp | *.cxx | \
+*.[fF][09]? | *.for | *.java | *.obj | *.sx)
+ func_xform $libobj
+ libobj=$func_xform_result
+ ;;
esac
-libobj=`$ECHO X$libobj | $Xsed -e s/\.$xform$/.lo/`
-
case $libobj in
*.lo) func_lo2o $libobj; obj=$func_lo2o_result ;;
*)
Index: libltdl/m4/libtool.m4
===
RCS file: /cvsroot/libtool/libtool/libltdl/m4/libtool.m4,v
retrieving revision 1.146
diff -u -r1.146 libtool.m4
--- libltdl/m4/libtool.m4 1 Apr 2008 18:23:19 - 1.146
+++ libltdl/m4/libtool.m4 9 Apr 2008 23:20:42 -
@@ -7098,6 +7098,7 @@
[case $xsi_shell in
yes)
cat \_LT_EOF $cfgfile
+
# func_dirname file append nondir_replacement
# Compute the dirname of FILE. If nonempty, add APPEND to the result,
# otherwise set result to NONDIR_REPLACEMENT.
@@ -7165,10 +7166,18 @@
*)func_lo2o_result=${1} ;;
esac
}
+
+# func_xform libobj-or-source
+func_xform ()
+{
+ func_xform_result=${1%.*}.lo
+}
+
_LT_EOF
;;
*) # Bourne compatible functions.
cat \_LT_EOF $cfgfile
+
# func_dirname file append nondir_replacement
# Compute the dirname of FILE. If nonempty, add APPEND to the result,
# otherwise set result to NONDIR_REPLACEMENT.
@@ -7245,6 +7254,12 @@
{
func_lo2o_result=`$ECHO X${1} | $Xsed -e $lo2o`
}
+
+# func_xform libobj-or-source
+func_xform ()
+{
+ func_xform_result=`$ECHO X${1} | $Xsed -e s/\.[[^.]]*/.lo/`
+}
_LT_EOF
esac
@@ -7269,6 +7284,7 @@
{
eval $[1]=\$$[1]\$[2]
}
+
_LT_EOF
;;
esac
Speed up progname, progpath setting
OK to apply?
What about eliminating (when progdir='.')
curwd=`cd . pwd`
Do we know when (with which shells) this can be replaced by $PWD
portably?
Thanks,
Ralf
2008-04-10 Ralf Wildenhues [EMAIL PROTECTED]
* libltdl/m4/libtool.m4 (_LT_PROG_XSI_SHELLFNS): Move non-XSI
definition of func_dirname_and_basename ...
* libltdl/config/general.m4sh (func_dirname_and_basename):
... here, to use it for the other scripts that use general.m4sh
(and as portable fallback for libtool). Add marker for early
insertion of generated shell functions (into libtool, not the
other scripts) here.
Use func_dirname_and_basename to compute progname, progpath.
* libltdl/config/ltmain.m4sh (func_mode_help): Remove marker
for insertion of generated shell functions.
* tests/sh.test: Adjust function definition test to not match
function call.
Index: libltdl/config/general.m4sh
===
RCS file: /cvsroot/libtool/libtool/libltdl/config/general.m4sh,v
retrieving revision 1.10
diff -u -r1.10 general.m4sh
--- libltdl/config/general.m4sh 9 Mar 2008 14:12:10 - 1.10
+++ libltdl/config/general.m4sh 9 Apr 2008 23:24:13 -
@@ -61,6 +61,32 @@
dirname=s,/[^/]*$,,
basename=s,^.*/,,
+# func_dirname_and_basename file append nondir_replacement
+# perform func_basename and func_dirname in a single function
+# call:
+# dirname: Compute the dirname of FILE. If nonempty,
+# add APPEND to the result, otherwise set result
+# to NONDIR_REPLACEMENT.
+# value returned in $func_dirname_result
+# basename: Compute filename of FILE.
+# value retuned in $func_basename_result
+# Implementation must be kept synchronized with func_dirname
+# and func_basename. For efficiency, we do not delegate to
+# those functions but instead duplicate the functionality here.
+func_dirname_and_basename ()
+{
+ # Extract subdirectory from the argument.
+ func_dirname_result=`$ECHO X${1} | $Xsed -e $dirname`
+ if test X$func_dirname_result = X${1}; then
+func_dirname_result=${3}
+ else
+func_dirname_result=$func_dirname_result${2}
+ fi
+ func_basename_result=`$ECHO X${1} | $Xsed -e $basename`
+}
+
+# Generated shell functions inserted here.
+
# Work around backward compatibility issue on IRIX 6.5. On IRIX 6.4+, sh
# is ksh but when the shell is invoked as sh and the current value of
# the _XPG environment variable is not equal to 1 (one), the special
@@ -71,13 +97,17 @@
# The name of this program:
# In the unlikely event $progname began with a '-', it would play havoc with
# func_echo (imagine progname=-n), so we prepend ./ in that case:
-progname=`$ECHO X$progpath | $Xsed -e $basename -e 's,^-,./-,'`
+func_dirname_and_basename $progpath
+progname=$func_basename_result
+case $progname in
+ -*) progname=./$progname ;;
+esac
# Make sure we have an absolute path for reexecution:
case $progpath in
[\\/]*|[A-Za-z]:\\*) ;;
*[\\/]*)
- progdir=`$ECHO X$progpath | $Xsed -e $dirname`
+ progdir=$func_dirname_result
progdir=`cd $progdir pwd`
progpath=$progdir/$progname
;;
Index: libltdl/config/ltmain.m4sh
===
RCS file: /cvsroot/libtool/libtool/libltdl/config/ltmain.m4sh,v
retrieving revision 1.102
diff -u -r1.102 ltmain.m4sh
--- libltdl/config/ltmain.m4sh 16 Mar 2008 08:34:25 - 1.102
+++ libltdl/config/ltmain.m4sh 9 Apr 2008 23:24:15 -
@@ -432,9 +432,6 @@
exit $?
}
-# Generated shell functions inserted here.
-
-
# Parse options once, thoroughly. This comes as soon as possible in
# the script to make things like `libtool --version' happen quickly.
{
Index: libltdl/m4/libtool.m4
===
RCS file: /cvsroot/libtool/libtool/libltdl/m4/libtool.m4,v
retrieving revision 1.146
diff -u -r1.146 libtool.m4
--- libltdl/m4/libtool.m4 1 Apr 2008 18:23:19 - 1.146
+++ libltdl/m4/libtool.m4 9 Apr 2008 23:24:17 -
@@ -7189,29 +7189,9 @@
func_basename_result=`$ECHO X${1} | $Xsed -e $basename`
}
-# func_dirname_and_basename file append nondir_replacement
-# perform func_basename and func_dirname in a single function
-# call:
-# dirname: Compute the dirname of FILE. If nonempty,
-# add APPEND to the result, otherwise set result
-# to NONDIR_REPLACEMENT.
-# value returned in $func_dirname_result
-# basename: Compute filename of FILE.
-# value retuned in $func_basename_result
-# Implementation must be kept synchronized with func_dirname
-# and func_basename. For efficiency, we do not delegate to
-# those functions but instead duplicate the functionality here.
-func_dirname_and_basename ()
-{
- # Extract subdirectory from the argument.
- func_dirname_result=`$ECHO X${1} | $Xsed -e $dirname`
- if
No need for object file removal before compilation
OK to apply? I fail to see why it's necessary to remove these files.
If they existed from a previous compile, they are outdated, so what.
But I may be overlooking something here.
Thanks,
Ralf
2008-04-10 Ralf Wildenhues [EMAIL PROTECTED]
* libltdl/config/ltmain.m4sh (func_mode_compile): No need to
remove old output files before compilation.
Index: libltdl/config/ltmain.m4sh
===
RCS file: /cvsroot/libtool/libtool/libltdl/config/ltmain.m4sh,v
retrieving revision 1.102
diff -u -r1.102 ltmain.m4sh
--- libltdl/config/ltmain.m4sh 16 Mar 2008 08:34:25 - 1.102
+++ libltdl/config/ltmain.m4sh 9 Apr 2008 23:33:43 -
@@ -1437,7 +1423,6 @@
removelist=$lobj $libobj ${libobj}T
fi
-$opt_dry_run || $RM $removelist
trap $opt_dry_run || $RM $removelist; exit $EXIT_FAILURE 1 2 15
# On Cygwin there's no real PIC flag so we must build both object types
@@ -1568,7 +1553,6 @@
# Suppress compiler output if we already did a PIC compilation.
command=$command$suppress_output
- $opt_dry_run || $RM $obj $output_obj
func_show_eval_locale $command \
'$opt_dry_run || $RM $removelist; exit $EXIT_FAILURE'
Re: compile mode, XSI shell: do not fork for $xform
Ralf Wildenhues wrote:
OK to apply?
+# func_xform libobj-or-source
+func_xform ()
+{
+ func_xform_result=`$ECHO X${1} | $Xsed -e s/\.[[^.]]*/.lo/`
+}
_LT_EOF
esac
Did you test this sed expression? Maybe I am testing it wrong?
$ echo ada.ada.ada | sed -e s/\.[^.]*/.lo/
ada.lo.ada
Peter
--
Peter O'Gorman
http://pogma.com
Re: compile mode, XSI shell: do not fork for $xform
Peter O'Gorman wrote:
Ralf Wildenhues wrote:
OK to apply?
+# func_xform libobj-or-source
+func_xform ()
+{
+ func_xform_result=`$ECHO X${1} | $Xsed -e s/\.[[^.]]*/.lo/`
+}
_LT_EOF
esac
Did you test this sed expression? Maybe I am testing it wrong?
$ echo ada.ada.ada | sed -e s/\.[^.]*/.lo/
ada.lo.ada
A $ helps:
echo ada.ada.ada | sed -e s/\.[^.]*$/.lo/
ada.ada.lo
It's ok with this changed.
Peter
--
Peter O'Gorman
http://pogma.com
Re: Speed up progname, progpath setting
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 According to Peter O'Gorman on 4/9/2008 10:19 PM: | What about eliminating (when progdir='.') | curwd=`cd . pwd` | | Do we know when (with which shells) this can be replaced by $PWD | portably? | | The ones that are not buggy and comply with the standards? :) (I don't | know, maybe all of them). To some degree, it depends on whether you are okay with $PWD being logical on platforms where pwd behaves like the posix 'pwd -P'; however, since autoconf documents that you can't reliably expect either 'pwd -L' or 'pwd - -P', you have to deal with both anyway. But I personally don't know how likely you are to hit a shell that forgets to assign PWD. - -- Don't work too hard, make some time for fun as well! Eric Blake [EMAIL PROTECTED] -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Cygwin) Comment: Public key at home.comcast.net/~ericblake/eblake.gpg Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkf9l6QACgkQ84KuGfSFAYB1cQCfbSg9VOLCutPYN3N64PrYgSWg fF8AnRbBdPGsdSZrCA9xhro679+xuVku =Ahjz -END PGP SIGNATURE-
Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half
Hello Josh, can we limit followups to a subset of this impressive array of mailing lists? Say, to libtool@gnu.org? That would be readable at http://thread.gmane.org/gmane.linux.debian.devel.general/126905. Thanks. * Josh Triplett wrote on Wed, Apr 09, 2008 at 12:34:18PM CEST: Libtool knows how to handle libraries for umpteen different systems, including many ancient systems that have terrible shared library support. It has some extensive shell script logic to figure out how to build libraries for your system, and how to compile objects that go in those libraries. This logic does an amazingly impressive job of coping with adverse conditions. However, this logic all lives in an ~8500 line, ~250kB shell script, which runs *every single time you compile a source file*. This does not do wonders for performance. Curious: can you please state which Libtool version you timed against, and if not 2.2.x, redo timing against 2.2.2? Not that I expect wonders, but I expect something better than what you measured. Thanks, Ralf ___ http://lists.gnu.org/mailman/listinfo/libtool
Announcing Dolt, a drop-in Libtool replacement which cuts build times in half
Many packages use GNU autotools (automake and autoconf) to build, to the point that ./configure make represents one of the most common build procedures for Free Software packages. Libraries using autotools typically use GNU Libtool, partly because it works on almost any system and partly because autotools makes it difficult to do otherwise. Packages which use these libraries sometimes use libtool as well. Yet for many of these libraries and other packages, more than half of the build time goes into running the libtool shell script. Libtool knows how to handle libraries for umpteen different systems, including many ancient systems that have terrible shared library support. It has some extensive shell script logic to figure out how to build libraries for your system, and how to compile objects that go in those libraries. This logic does an amazingly impressive job of coping with adverse conditions. However, this logic all lives in an ~8500 line, ~250kB shell script, which runs *every single time you compile a source file*. This does not do wonders for performance. Meanwhile, modern systems such as GNU/Linux have reasonable library mechanisms, and need relatively little of the machinery in libtool. On these common systems, it would significantly improve build times to avoid running that libtool machinery for every compilation. Thus, I wrote Dolt, a drop-in replacement for libtool's compilation mode. Dolt runs any necessary system-specific or configuration-specific logic as part of configure, writes out a simple shell script doltcompile[1], and substitutes it for libtool in the automake variables LTCOMPILE and LTCXXCOMPILE. If you use automake, autoconf, and libtool, then using Dolt just requires two steps: 1) add DOLT after the call to LT_INIT, AC_PATH_LIBTOOL, or AM_PATH_LIBTOOL in your configure.ac or configure.in script, and 2) append dolt.m4 to your project's acinclude.m4. For any system Dolt does not support, it will transparently fall back to libtool. dolt.m4 takes up less than 4kB; it writes out a minimal doltcompile script which never forks except to run the compiler or to mkdir the .libs directory if it doesn't already exist. I have tested it with various projects, and benchmarked[2] its performance against the same projects using only libtool. Results: kdelibs without dolt: 8m6.115s kdelibs with dolt:3m50.065s gtk+-2.0 without dolt: 2m31.825s gtk+-2.0 with dolt:1m33.858s libx11 without dolt: 1m50.163s libx11 with dolt:0m53.417s libxml2 without dolt: 0m25.722s libxml2 with dolt:0m19.576s dbus without dolt: 0m20.062s dbus with dolt:0m8.940s I have attached a snapshot of dolt.m4 for convenience. You can also obtain the current version of Dolt from Git with: git clone git://svcs.cs.pdx.edu/git/dolt or download a snapshot tarball from http://svcs.cs.pdx.edu/gitweb?p=dolt.git;a=snapshot;h=master;sf=tgz. Please try Dolt with your project, and see if you get comparable performance improvements. If you want to make Dolt replace libtool on your system, feel free to send me a patch to dolt.m4; just remember the basic tenet that any logic must run at configure time, not build time. You can figure out what compiler flags libtool uses by running touch dummy.c libtool --mode=compile gcc -c dummy.c -o dummy.lo; that will print two compiler command lines, one for the shared object and one for the static object. Future directions: * Support GNU/Linux on architectures other than x86 and x86-64. I think most will work with exactly the same compiler flags, but I didn't want to add any architecture I couldn't test. * Support other systems. * Possibly try to run libtool on a dummy source file at configure time to figure out the necessary flags to use when building library objects, but that seems error-prone. * Replace libtool --mode=link. * Replace libtool --mode=install. * Optionally stop installing .la files. * Make dolt.m4's output of doltcompile cleaner. - Josh Triplett [1] doltcompile stands for do ltcompile; the alternate reading dolt compile led to the name dolt. [2] General testing methodology: * Run ./configure make make clean, to make sure it builds and to get everything cached. * Get the before time: time make /dev/null 21 * Remove and re-extract the source. * Add dolt.m4 to acinclude.m4 and add DOLT to configure.in or configure.ac. * autoreconf -v -f -i ./configure make make clean, to make sure it still builds and to get everything cached again. * Get the after time: time make /dev/null 21 dnl dolt, a replacement for libtool dnl Copyright © 2007-2008 Josh Triplett [EMAIL PROTECTED] dnl Copying and distribution of this file, with or without modification, dnl are permitted in any medium without royalty provided the copyright dnl notice and this notice are preserved. dnl dnl To use dolt, invoke the DOLT macro immediately after the libtool macros. dnl Optionally, copy this file into acinclude.m4, to avoid the need to have it dnl
Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half
Josh Triplett [EMAIL PROTECTED] - Wed, 09 Apr 2008 03:34:18 -0700 Thus, I wrote Dolt, a drop-in replacement for libtool's compilation mode. Excellent, impressive :))) ___ http://lists.gnu.org/mailman/listinfo/libtool
Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half
Ralf Wildenhues Ralf.Wildenhues at gmx.de writes: Curious: can you please state which Libtool version you timed against, and if not 2.2.x, redo timing against 2.2.2? Not that I expect wonders, but I expect something better than what you measured. I just tested this out on my laptop with Evolution Data Server: EDS with libtool 1.5.26-3 real4m35.934s user3m38.786s sys 0m52.855s EDS with libtool 2.1 (2.1a+cvs1.2525+20071016-1) real4m10.648s user3m37.994s sys 0m26.578s EDS with dolt real3m40.974s user3m4.448s sys 0m31.406s I realise this isn't libtool 2.2, so I'm willing to believe there are further speedups. Ross ___ http://lists.gnu.org/mailman/listinfo/libtool
Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half
Ralf Wildenhues wrote: can we limit followups to a subset of this impressive array of mailing lists? Say, to libtool@gnu.org? That would be readable at http://thread.gmane.org/gmane.linux.debian.devel.general/126905. Thanks. Sure. I assumed that anyone wanting to respond would do so to the mailing list they got it through if appropriate; for instance, I had a followup on just debian-devel. * Josh Triplett wrote on Wed, Apr 09, 2008 at 12:34:18PM CEST: Libtool knows how to handle libraries for umpteen different systems, including many ancient systems that have terrible shared library support. It has some extensive shell script logic to figure out how to build libraries for your system, and how to compile objects that go in those libraries. This logic does an amazingly impressive job of coping with adverse conditions. However, this logic all lives in an ~8500 line, ~250kB shell script, which runs *every single time you compile a source file*. This does not do wonders for performance. Curious: can you please state which Libtool version you timed against, and if not 2.2.x, redo timing against 2.2.2? Not that I expect wonders, but I expect something better than what you measured. I tested against 1.5.26. I'll give 2.2.2 a shot and see what I find. However, when I looked at 2.2.2, it still seems to have a multi-thousand-line shell script; do you just expect the benefit to come from the new shell-specific optimizations? - Josh Triplett signature.asc Description: OpenPGP digital signature ___ http://lists.gnu.org/mailman/listinfo/libtool
Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half
Josh Triplett wrote: I tested against 1.5.26. I'll give 2.2.2 a shot and see what I find. However, when I looked at 2.2.2, it still seems to have a multi-thousand-line shell script; do you just expect the benefit to come from the new shell-specific optimizations? Hi Josh, There are speedups for link mode, I don't think you will see much of an improvement for compile mode over 1.5.26, but I do not have numbers to back that up. Splitting off libtool compile might be worthwhile, but there should be no need for: AC_MSG_CHECKING([if libtool sucks]) AC_MSG_RESULT([yup, it does]) :-) Or to restrict the shell, compiler and OS to bash, gcc and GNU/linux. It should be possible to have a portable (at least a more portable than this) ltcompile script. Are you willing to work on libtool to improve its performance for compile mode? If so, we can send you copyright assignment forms etc. Peter -- Peter O'Gorman http://pogma.com ___ http://lists.gnu.org/mailman/listinfo/libtool
Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half
* Josh Triplett wrote on Wed, Apr 09, 2008 at 06:02:36PM CEST: Ralf Wildenhues wrote: Curious: can you please state which Libtool version you timed against, and if not 2.2.x, redo timing against 2.2.2? Not that I expect wonders, but I expect something better than what you measured. I tested against 1.5.26. I'll give 2.2.2 a shot and see what I find. However, when I looked at 2.2.2, it still seems to have a multi-thousand-line shell script; do you just expect the benefit to come from the new shell-specific optimizations? Partly, yes. And Ross showed that we already got half the way, and yes, the compile mode improvements are the biggest: http://thread.gmane.org/gmane.linux.debian.devel.general/126905/focus=9125 Libtool 2.2.2 in compile mode uses only 6 forks in compile mode if your shell has XSI features. Another thing to keep in mind is to put func_mode_compile early in the script, and outside of conditionals, so that the shell doesn't even have to parse anything below that. That gave the biggest speedup on GNU/Linux (where forks are relatively cheap). http://thread.gmane.org/gmane.comp.gnu.libtool.patches/7230 That message also shows timings for a separated libtool-link script, beyond which I then saw no further performance improvements. I guess I could try to dig out that old patch (which was never applied). Also, I see two more low-hanging fruits at the moment. Will follow up with patches later tonight or tomorrow night. Cheers, Ralf ___ http://lists.gnu.org/mailman/listinfo/libtool
Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half
Ralf Wildenhues wrote: That gave the biggest speedup on GNU/Linux (where forks are relatively cheap). http://thread.gmane.org/gmane.comp.gnu.libtool.patches/7230 This entire message just goes to prove that I do not have a good memory. I had completely forgotten that you sped up compile mode too. Peter -- Peter O'Gorman http://pogma.com ___ http://lists.gnu.org/mailman/listinfo/libtool
Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half
On Wed, 9 Apr 2008, Ross Burton wrote: I realise this isn't libtool 2.2, so I'm willing to believe there are further speedups. What I notice from your timings is that libtool 2.1 takes much less system time than libtool 1.5.26-3 but the user time is quite similar. The real time is surely most important, and it seems that the real time is improved with libtool 2.1 but not very dramatically. The mystery as to where the time is actually going is why I have suggested several times that there should be an effort to test libtool under a DTrace-enhanced shell (shell with a DTrace provider) and perform an analysis of what is being done and where time is being spent using dtrace. Even without a DTrace-enhanced shell, it is possible to see where time is going in a script using DTrace. A DTrace-enhanced shell is going into OpenSolaris. See http://blogs.sun.com/tpenta/entry/psarc_2008_008_dtrace_provider;. Quite often when I profile a program I find that something quite unexpected is hogging the time and the fix is quite trivial. Perhaps we will find this with libtool as well. For those of you who feel offended that I suggest using a tool which is only currently available under Solaris and OS-X Leopard, I find it to be no more insulting than a suggestion to run software under valgrind, which is only available under Linux (and sometimes on FreeBSD). Bob == Bob Friesenhahn [EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ http://lists.gnu.org/mailman/listinfo/libtool
Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half
On Wednesday 09 April 2008 03:34:18 am Josh Triplett wrote: Meanwhile, modern systems such as GNU/Linux have reasonable library mechanisms, and need relatively little of the machinery in libtool. On these common systems, it would significantly improve build times to avoid running that libtool machinery for every compilation. You keep using the plural, when your script specifically checks for a singular case. It's not a replacement for libtool until it can support the overwhelming majority of modern systems, and not just GNU/Linux. FreeBSD, NetBSD, OpenBSD, OpenSolaris, Darwin. -- David Johnson ___ http://lists.gnu.org/mailman/listinfo/libtool
Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half
On Wed, 9 Apr 2008, Josh Triplett wrote: I tested against 1.5.26. I'll give 2.2.2 a shot and see what I find. However, when I looked at 2.2.2, it still seems to have a multi-thousand-line shell script; do you just expect the benefit to come from the new shell-specific optimizations? Since shell scripts are not compiled, the size of a shell script has very little to do with its execution time. Bob == Bob Friesenhahn [EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ http://lists.gnu.org/mailman/listinfo/libtool
Re: supportof the cegcc compiler
On Fri, 14 Mar 2008, Bob Friesenhahn wrote: On Fri, 14 Mar 2008, Vincent Torri wrote: it's not my patch but here is what I've found: http://lists.pld-linux.org/mailman/pipermail/pld-cvs-commit/Week-of-Mon-20070226/144657.html and it's not in the libtool patch list. Should all patches be sent there ? It seems that most of this patch is just adding cegcc* to existing case statements. Do you think that the patch looks good ? I'm porting more and more libs, and creating the dll by hand is quite bothering. So i'm planning to works a bit on how to support cegcc in libtool. The problem is that, for me, libtool is some kind of black voodoo magic. So i don't know if it is reasonnable or not (i didn't write that patch) thank you Vincent ___ http://lists.gnu.org/mailman/listinfo/libtool
Re: supportof the cegcc compiler
On Wed, 9 Apr 2008, Vincent Torri wrote: It seems that most of this patch is just adding cegcc* to existing case statements. Do you think that the patch looks good ? I'm porting more and more libs, and creating the dll by hand is quite bothering. So i'm planning to works a bit on how to support cegcc in libtool. The problem is that, for me, libtool is some kind of black voodoo magic. So i don't know if it is reasonnable or not (i didn't write that patch) Not knowing anything in particular about 'cegcc', I don't have any philosophical or technical issues with the patch except that it should be submitted by its author so we know who wrote it and that they intended it to be contributed to GNU libtool. It is also useful to have a volunteer who is willing to test libtool to make sure that it works (and continues to work) for this target. I suggest that you find out who wrote this patch and convince them to submit it as an official libtool patch against libtool 2.X. Bob == Bob Friesenhahn [EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ http://lists.gnu.org/mailman/listinfo/libtool
Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half
Hi Ross, * Ross Burton wrote on Wed, Apr 09, 2008 at 03:44:40PM CEST: EDS with libtool 1.5.26-3 real 4m35.934s EDS with libtool 2.1 (2.1a+cvs1.2525+20071016-1) real 4m10.648s EDS with dolt real 3m40.974s OK, so we're already half-way there, and the speedup in Libtool 2.2 benefits not only with modern bash and on GNU/Linux, but all XSI-compliant shells, on any system. I realise this isn't libtool 2.2, so I'm willing to believe there are further speedups. No. But 2.2.4 will have further speedups, if the patches I just posted on libtool-patches are accepted. Cheers, Ralf ___ http://lists.gnu.org/mailman/listinfo/libtool
Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 According to Bob Friesenhahn on 4/9/2008 11:15 AM: | On Wed, 9 Apr 2008, Josh Triplett wrote: | | I tested against 1.5.26. I'll give 2.2.2 a shot and see what I find. | However, when I looked at 2.2.2, it still seems to have a | multi-thousand-line shell script; do you just expect the benefit to | come from the new shell-specific optimizations? | | Since shell scripts are not compiled, the size of a shell script has | very little to do with its execution time. On the other hand, recent improvements in autoconf 2.62 proved that we were able to speed up testsuite performance by more than 10% by merely refactoring Autotest output to avoid shell parsing of code that would not be executed. In other words, the time the shell spends on parsing its input, whether or not that input is executed, is not trivial. - -- Don't work too hard, make some time for fun as well! Eric Blake [EMAIL PROTECTED] -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Cygwin) Comment: Public key at home.comcast.net/~ericblake/eblake.gpg Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkf9WpAACgkQ84KuGfSFAYBXRACgwWkVTjy5WMdo8CHkVYcXg6fa cO0An2v7JSz2Ss9lcnsVmNal3msA+fl7 =4zFK -END PGP SIGNATURE- ___ http://lists.gnu.org/mailman/listinfo/libtool
Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half
On Wed, 9 Apr 2008, Eric Blake wrote: | Since shell scripts are not compiled, the size of a shell script has | very little to do with its execution time. On the other hand, recent improvements in autoconf 2.62 proved that we were able to speed up testsuite performance by more than 10% by merely refactoring Autotest output to avoid shell parsing of code that would not be executed. In other words, the time the shell spends on parsing its input, whether or not that input is executed, is not trivial. This is true. The script could be 10MB in size but if only the first five lines are ever executed the execution time should be similar to a similar five line script. This means that parts which are executed often should be up front. If there is a long list of shell functions which appear up front, then all those shell functions need to be parsed and remembered before the real executable code is executed. There may be value to extracting these shell functions into separate files which are sourced only when needed. This way the overhead of parsing linking code is not encountered while compiling, and vice-versa. Since there is surely overhead associated with opening and sourcing a file the chunk of code should be pretty large. Unfortunately, more files gives people more to complain about since a single large file is less offensive than many smaller files. If the build-time overhead associated with libtool 2.2 (vs Dolt) is really only 13% or so, someone needs to make the call as to what is an acceptable level of runtime overhead before Dolt can be disposed of an unecessary idea. Is the acceptable overhead 5%, 7%, 10%? What is the magic number? Bob == Bob Friesenhahn [EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ http://lists.gnu.org/mailman/listinfo/libtool
Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half
Eric Blake wrote:
According to Bob Friesenhahn on 4/9/2008 11:15 AM:
| On Wed, 9 Apr 2008, Josh Triplett wrote:
|
| I tested against 1.5.26. I'll give 2.2.2 a shot and see what I find.
| However, when I looked at 2.2.2, it still seems to have a
| multi-thousand-line shell script; do you just expect the benefit to
| come from the new shell-specific optimizations?
|
| Since shell scripts are not compiled, the size of a shell script has
| very little to do with its execution time.
On the other hand, recent improvements in autoconf 2.62 proved that we
were able to speed up testsuite performance by more than 10% by merely
refactoring Autotest output to avoid shell parsing of code that would not
be executed. In other words, the time the shell spends on parsing its
input, whether or not that input is executed, is not trivial.
Just for fun, lets compare shells at parsing useless code.
(using bash)
$ for y in {1..100}; do echo func_notused${y} () { parse.sh; for x
in {1..1}; do echo foo parse.sh; done; echo '}' parse.sh;
done; echo 'echo Done' parse.sh
On linux:
$ time bash parse.sh
Done
real0m4.567s
user0m3.970s
sys 0m0.188s
$ time dash parse.sh
Done
real0m1.421s
user0m1.242s
sys 0m0.096s
$ time zsh parse.sh
Done
real0m1.635s
user0m1.293s
sys 0m0.161s
On mac os x:
$ time zsh parse.sh
Done
real0m1.429s
user0m1.176s
sys 0m0.193s
$ time bash parse.sh
Done
real0m4.921s
user0m4.706s
sys 0m0.215s
$ time ksh parse.sh
Done
real5m31.311s ***
user5m29.284s
sys 0m1.876s
I know that libtool has not yet reached a million+ lines of useless
shell functions though :)
Peter
--
Peter O'Gorman
http://pogma.com
___
http://lists.gnu.org/mailman/listinfo/libtool
Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 According to Bob Friesenhahn on 4/9/2008 7:01 PM: | There may be value to extracting these shell functions into separate | files which are sourced only when needed. This way the overhead of | parsing linking code is not encountered while compiling, and | vice-versa. Since there is surely overhead associated with opening and | sourcing a file the chunk of code should be pretty large. In fact, that's what autotest does in autoconf 2.62 - it distributes a single file with all the tests in the tail of the file, where the shell script at the head of the file parses the command line options, uses that to drive an awk script to parse out the line numbers where the tests of interest live, then uses the line number cache to drive a sed script to parse out each test into a separate file for sourcing by the shell only when it is needeed. As crazy as this multi-process action sounds, it was hands down faster than letting the shell parse all the test functions when running a subset of the testsuite, without too much penalty when running the entire testsuite. - -- Don't work too hard, make some time for fun as well! Eric Blake [EMAIL PROTECTED] -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Cygwin) Comment: Public key at home.comcast.net/~ericblake/eblake.gpg Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkf9iLUACgkQ84KuGfSFAYAcIwCeNuLsON7gZ0XLG8k+U3cXydeT 2REAnjZOLd81d7Kw5Hdh3FH/PkhFbWPs =LLpS -END PGP SIGNATURE- ___ http://lists.gnu.org/mailman/listinfo/libtool
Re: Announcing Dolt, a drop-in Libtool replacement
Bob Friesenhahn wrote: On Wed, 9 Apr 2008, Eric Blake wrote: | Since shell scripts are not compiled, the size of a shell script has | very little to do with its execution time. On the other hand, recent improvements in autoconf 2.62 proved that we were able to speed up testsuite performance by more than 10% by merely refactoring Autotest output to avoid shell parsing of code that would not be executed. In other words, the time the shell spends on parsing its input, whether or not that input is executed, is not trivial. This is true. The script could be 10MB in size but if only the first five lines are ever executed the execution time should be similar to a similar five line script. This means that parts which are executed often should be up front. If there is a long list of shell functions which appear up front, then all those shell functions need to be parsed and remembered before the real executable code is executed. There may be value to extracting these shell functions into separate files which are sourced only when needed. This way the overhead of parsing linking code is not encountered while compiling, and vice-versa. Since there is surely overhead associated with opening and sourcing a file the chunk of code should be pretty large. Unfortunately, more files gives people more to complain about since a single large file is less offensive than many smaller files. Once upon a time, long, long ago and far, far away, I started futzing with a C wrapper around lots of this stuff. The point being to keep all the text in one place (avoiding a gazillion files) and having the shell parse only the text relevant to not just the mode, but also whatever else could easily be determined in a binary and passed on to a shell. It saved a lot of time. You did have to compile the thing once at the beginning, but that gets lost in the noise. The difference was huge. Real Life got in the way tho Hope this works out better :) Cheers -Bruce ___ http://lists.gnu.org/mailman/listinfo/libtool
Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half
Bob Friesenhahn wrote: On Wed, 9 Apr 2008, Eric Blake wrote: | Since shell scripts are not compiled, the size of a shell script has | very little to do with its execution time. On the other hand, recent improvements in autoconf 2.62 proved that we were able to speed up testsuite performance by more than 10% by merely refactoring Autotest output to avoid shell parsing of code that would not be executed. In other words, the time the shell spends on parsing its input, whether or not that input is executed, is not trivial. This is true. The script could be 10MB in size but if only the first five lines are ever executed the execution time should be similar to a similar five line script. This means that parts which are executed often should be up front. If there is a long list of shell functions which appear up front, then all those shell functions need to be parsed and remembered before the real executable code is executed. This is a list off shell functions that appear in the generated libtool script on my linux system (one of Ralf's patches is applied). Yes, we could probably move these around some to get func_mode_compile closer to the top. func_dirname_and_basename () func_dirname () func_basename () func_dirname_and_basename () func_stripname () func_opt_split () func_lo2o () func_append () func_echo () func_verbose () func_error () func_warning () func_fatal_error () func_fatal_help () func_grep () func_mkdir_p () func_mktempdir () func_quote_for_eval () func_quote_for_expand () func_show_eval () func_show_eval_locale () func_version () func_usage () func_help () func_missing_arg () func_fatal_configuration () func_config () func_features () func_enable_tag () func_mode_help () func_check_version_match () func_lalib_p () func_lalib_unsafe_p () func_ltwrapper_script_p () func_ltwrapper_executable_p () func_ltwrapper_scriptname () func_ltwrapper_p () func_execute_cmds () func_source () func_win32_libid () func_infer_tag () func_generate_dlsyms () func_extract_an_archive () func_extract_archives () func_write_libtool_object () func_mode_compile () func_mode_execute () func_mode_finish () func_mode_install () func_emit_wrapper () func_emit_cwrapperexe_src () func_mode_link () func_mode_uninstall () Peter -- Peter O'Gorman http://pogma.com ___ http://lists.gnu.org/mailman/listinfo/libtool
Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half
On Wed, 9 Apr 2008, Peter O'Gorman wrote:
(using bash)
$ for y in {1..100}; do echo func_notused${y} () { parse.sh; for x
in {1..1}; do echo foo parse.sh; done; echo '}' parse.sh;
done; echo 'echo Done' parse.sh
It seems that the slowest possible shell is selected by default.
Maybe that is bad?
On Solaris 10 (Opteron)
% ptime bash parse.sh
Done
real3.247
user2.923
sys 0.322
% ptime zsh parse.sh
Done
real0.991
user0.877
sys 0.112
% ptime ksh parse.sh
Done
real1.063
user0.922
sys 0.141
On Solaris 10 (SPARC)
% ptime bash parse.sh
Done
real 10.417
user 10.128
sys 0.283
% ptime zsh parse.sh
Done
real3.045
user2.817
sys 0.227
% ptime ksh parse.sh
Done
real3.432
user3.255
sys 0.175
On FreeBSD 7.0 (Intel Xeon)
% time bash parse.sh
Done
bash parse.sh 5.63s user 0.18s system 99% cpu 5.808 total
% time zsh parse.sh
Done
zsh parse.sh 2.02s user 0.36s system 99% cpu 2.379 total
On Mac OS-X Leopard (G5)
% time bash parse.sh
Done
bash parse.sh 4.46s user 0.41s system 99% cpu 4.891 total
% time zsh parse.sh
Done
zsh parse.sh 1.35s user 0.39s system 83% cpu 2.074 total
% time ksh parse.sh
[ runs forever apparently ]
==
Bob Friesenhahn
[EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
___
http://lists.gnu.org/mailman/listinfo/libtool
Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half
Bob Friesenhahn wrote:
On Wed, 9 Apr 2008, Peter O'Gorman wrote:
(using bash)
$ for y in {1..100}; do echo func_notused${y} () { parse.sh; for x
in {1..1}; do echo foo parse.sh; done; echo '}' parse.sh;
done; echo 'echo Done' parse.sh
It seems that the slowest possible shell is selected by default. Maybe
that is bad?
Only if this test is actually a good benchmark (which is doubtful) :-)
Peter
--
Peter O'Gorman
http://pogma.com
___
http://lists.gnu.org/mailman/listinfo/libtool
Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half
On Wed, 9 Apr 2008, Peter O'Gorman wrote: This is a list off shell functions that appear in the generated libtool script on my linux system (one of Ralf's patches is applied). Yes, we could probably move these around some to get func_mode_compile closer to the top. The important thing is not how close func_mode_compile() is to the top, but how close the code which invokes it is to the top. Bob == Bob Friesenhahn [EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ http://lists.gnu.org/mailman/listinfo/libtool
