Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Richard Purdie]

Josh Triplett  freedesktop.org> writes:
> Thus, I wrote Dolt, a drop-in replacement for libtool's compilation
> mode.  Dolt runs any necessary system-specific or
> configuration-specific logic as part of configure, writes out a simple
> shell script "doltcompile"[1], and substitutes it for libtool in the
> automake variables LTCOMPILE and LTCXXCOMPILE.  If you use automake,
> autoconf, and libtool, then using Dolt just requires two steps:
> 
> 1) add "DOLT" after the call to LT_INIT, AC_PATH_LIBTOOL, or
>AM_PATH_LIBTOOL in your configure.ac or configure.in script, and
> 2) append dolt.m4 to your project's acinclude.m4.
> 
> For any system Dolt does not support, it will transparently fall back
> to libtool.

I work with Poky (pokylinux.org) and OpenEmbedded (openembedded.org) which are
cross compiling build systems and speed improvements like this are extremely
useful to us. Those systems also provide an interesting testing ground
where we can expose changes like this to a wide variety of source code.

I found this last part doesn't hold true, dolt does not fall back to
libtool transparently, the reason being the AC_SUBST causes LTCOMPILE to
become "", even if that section of the if block isn't called. I couldn't
find a nice way to fix this since AC_SUBST operates at reautoconf time,
not at configure and you only know if the system is compatible at
configure time.

Poky/OE use an old libtool (1.5.10) since we have various hacks we had
to make to get libtool to support cross compiling sanely[1] and
everytime we've tried to upgrade, something goes wrong and we've never
had time to debug the newer versions.

[1] Are there any plans to support sysroots with libtool?

In the interests of experimentation I hacked dolt into our libtool
recipe, inserting it into libtool.m4 in AC_PROG_LIBTOOL so I didn't have
to change any apps. I enabled it for arm linux targets and removed the
broken fallback code. The test build I used as a benchmark builds a
cross compiling toolchain and then from that a complete PDA style system
including X and some GTK+ apps with all the parts in between, many of
which use libtool.

Before the dolt change this image took 108 minutes, afterwards it took
96 minutes so an 11% reduction in time which is certainly beneficial!

Cheers,

Richard



___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Ralf Wildenhues]

* Tor Lillqvist wrote on Thu, Apr 10, 2008 at 09:54:35AM CEST:
> It speeds up builds on Win32 very
> nicely. http://libtool-cache.sourceforge.net/

I don't remember well; was libtool-cache more than marginally beneficial
on non-w32 systems as well?

> Maybe some libtool-cache-like functionality could even be integrated
> into libtool itself in the future?

Same issue as with shcomp: the more general/portable/fast, the better,
provided there are no functionality regressions.

Thanks,
Ralf


___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Gary V. Vaughan]

On 10 Apr 2008, at 02:18, Peter O'Gorman wrote:

Ralf Wildenhues wrote:

* Bob Friesenhahn wrote on Thu, Apr 10, 2008 at 07:51:57AM CEST:

On Wed, 9 Apr 2008, Peter O'Gorman wrote:
This is a list off shell functions that appear in the generated  
libtool
script on my linux system (one of Ralf's patches is applied).  
Yes, we
could probably move these around some to get func_mode_compile  
closer to

the top.
The important thing is not how close func_mode_compile() is to the  
top,

but how close the code which invokes it is to the top.


Most of these optimizations are already done in 2.2.2.  The code
invoking func_mode_compile is right after its definition.

Wrt. moving functions, func_win32_libid, func_generate_dlsyms, and
func_extract_an_archive, func_extract_archives can still be moved
after func_mode_compile.  That gets another 10% overhead reduction.
My patches from last night get about the same amount.


func_mode_help can also be easily moved, I am testing a patch  
currently

that moves all of these functions to after mode_compile.



Pity that the following spews garbage :-(

] awk '/^func_/ { inside=$1; next; }
/func_/ { match($0, "func_[a-zA-Z0-9_]*"); print inside,  
substr($0,RSTART,RLENGTH); }

' +release/libtool | tsort -q
func_mode_uninstall
func_emit_cwrapperexe_src
func_mode_link
func_mode_install
func_emit_wrapper
func_emit_wrapper_arg1
func_mode_finish
func_mode_execute
func_mode_compile
func_extract_archives
func_write_libtool_object
func_extract_archives_result
func_generate_dlsyms
func_extract_an_archive
func_ltwrapper_p
func_execute_cmds
func_source
func_win32_libid
func_infer_tag
func_ltwrapper_scriptname
func_ltwrapper_scriptname_result
func_ltwrapper_script_p
func_ltwrapper_executable_p
func_ltwrapper_exec_suffix
func_append
func_check_version_match
func_mode_help
func_lalib_p
func_lalib_unsafe_p
func_basename
func_dirname_and_basename
func_basename_result
func_dirname
func_stripname
func_dirname_result
func_opt_split
func_stripname_result
func_lo2o
func_opt_split_arg
func_opt_split_opt
func_lo2o_result
func_enable_tag
func_fatal_configuration
func_config
func_features
func_missing_arg
func_show_eval_locale
func_version
func_usage
func_help
func_show_eval
func_mktempdir
func_quote_for_eval
func_quote_for_expand
func_quote_for_eval_result
func_quote_for_eval_unquoted_result
func_quote_for_expand_result
func_mkdir_p
func_fatal_help
func_grep
func_fatal_error
func_verbose
func_error
func_echo
func_warning

Cheers,
Gary
--
  ())_.  Email me: [EMAIL PROTECTED]
  ( '/   Read my blog: http://blog.azazil.net
  / )= ...and my book: http://sources.redhat.com/autobook
`(_~)_






PGP.sig
Description: This is a digitally signed message part
___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Tor Lillqvist]

Let me just point out another, old but apparently not well known, more
portable imrovement on libtool, libtool-cache. libtool-cache doesn't
replace libtool, but as can be guessed from its name, it just avoids
running libtool if the actual shell commands that would be executed by
libtool can be known beforehand. I have been using libtool-cache with
good success for several years. It speeds up builds on Win32 very
nicely. http://libtool-cache.sourceforge.net/

Maybe some libtool-cache-like functionality could even be integrated
into libtool itself in the future?

--tml


___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Peter O'Gorman]

Ralf Wildenhues wrote:
> * Bob Friesenhahn wrote on Thu, Apr 10, 2008 at 07:51:57AM CEST:
>> On Wed, 9 Apr 2008, Peter O'Gorman wrote:
>>> This is a list off shell functions that appear in the generated libtool
>>> script on my linux system (one of Ralf's patches is applied). Yes, we
>>> could probably move these around some to get func_mode_compile closer to
>>> the top.
>> The important thing is not how close func_mode_compile() is to the top, 
>> but how close the code which invokes it is to the top.
> 
> Most of these optimizations are already done in 2.2.2.  The code
> invoking func_mode_compile is right after its definition.
> 
> Wrt. moving functions, func_win32_libid, func_generate_dlsyms, and
> func_extract_an_archive, func_extract_archives can still be moved
> after func_mode_compile.  That gets another 10% overhead reduction.
> My patches from last night get about the same amount.

func_mode_help can also be easily moved, I am testing a patch currently
that moves all of these functions to after mode_compile.

Peter
-- 
Peter O'Gorman
http://pogma.com


___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Ralf Wildenhues]

* Bob Friesenhahn wrote on Thu, Apr 10, 2008 at 07:51:57AM CEST:
> On Wed, 9 Apr 2008, Peter O'Gorman wrote:
>> This is a list off shell functions that appear in the generated libtool
>> script on my linux system (one of Ralf's patches is applied). Yes, we
>> could probably move these around some to get func_mode_compile closer to
>> the top.
>
> The important thing is not how close func_mode_compile() is to the top, 
> but how close the code which invokes it is to the top.

Most of these optimizations are already done in 2.2.2.  The code
invoking func_mode_compile is right after its definition.

Wrt. moving functions, func_win32_libid, func_generate_dlsyms, and
func_extract_an_archive, func_extract_archives can still be moved
after func_mode_compile.  That gets another 10% overhead reduction.
My patches from last night get about the same amount.

More tonight.
Cheers,
Ralf


___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Bob Friesenhahn]

On Wed, 9 Apr 2008, Peter O'Gorman wrote:

This is a list off shell functions that appear in the generated libtool
script on my linux system (one of Ralf's patches is applied). Yes, we
could probably move these around some to get func_mode_compile closer to
the top.


The important thing is not how close func_mode_compile() is to the 
top, but how close the code which invokes it is to the top.


Bob
==
Bob Friesenhahn
[EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/



___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Peter O'Gorman]

Bob Friesenhahn wrote:
> On Wed, 9 Apr 2008, Peter O'Gorman wrote:
>>
>> (using bash)
>> $ for y in {1..100}; do echo "func_notused${y} () {" >> parse.sh; for x
>> in {1..1}; do echo foo >> parse.sh; done; echo '}' >> parse.sh;
>> done; echo 'echo Done' >> parse.sh
>>
> 
> It seems that the slowest possible shell is selected by default. Maybe
> that is bad?

Only if this test is actually a good benchmark (which is doubtful) :-)

Peter
-- 
Peter O'Gorman
http://pogma.com


___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Bob Friesenhahn]

On Wed, 9 Apr 2008, Peter O'Gorman wrote:


(using bash)
$ for y in {1..100}; do echo "func_notused${y} () {" >> parse.sh; for x
in {1..1}; do echo foo >> parse.sh; done; echo '}' >> parse.sh;
done; echo 'echo Done' >> parse.sh



It seems that the slowest possible shell is selected by default. 
Maybe that is bad?


On Solaris 10 (Opteron)

% ptime bash parse.sh
Done

real3.247
user2.923
sys 0.322

% ptime zsh parse.sh
Done

real0.991
user0.877
sys 0.112

% ptime ksh parse.sh
Done

real1.063
user0.922
sys 0.141

On Solaris 10 (SPARC)

% ptime bash parse.sh
Done

real   10.417
user   10.128
sys 0.283

% ptime zsh parse.sh
Done

real3.045
user2.817
sys 0.227

% ptime ksh parse.sh
Done

real3.432
user3.255
sys 0.175

On FreeBSD 7.0 (Intel Xeon)

% time bash parse.sh
Done
bash parse.sh  5.63s user 0.18s system 99% cpu 5.808 total

% time zsh parse.sh
Done
zsh parse.sh  2.02s user 0.36s system 99% cpu 2.379 total

On Mac OS-X Leopard (G5)

% time bash parse.sh
Done
bash parse.sh  4.46s user 0.41s system 99% cpu 4.891 total

% time zsh parse.sh
Done
zsh parse.sh  1.35s user 0.39s system 83% cpu 2.074 total

% time ksh parse.sh
[ runs forever apparently ]

==
Bob Friesenhahn
[EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/



___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Peter O'Gorman]

Bob Friesenhahn wrote:
> On Wed, 9 Apr 2008, Eric Blake wrote:
>> | Since shell scripts are not compiled, the size of a shell script has
>> | very little to do with its execution time.
>>
>> On the other hand, recent improvements in autoconf 2.62 proved that we
>> were able to speed up testsuite performance by more than 10% by merely
>> refactoring Autotest output to avoid shell parsing of code that would not
>> be executed.  In other words, the time the shell spends on parsing its
>> input, whether or not that input is executed, is not trivial.
> 
> This is true.  The script could be 10MB in size but if only the first
> five lines are ever executed the execution time should be similar to a
> similar five line script.  This means that parts which are executed
> often should be up front.
> 
> If there is a long list of shell functions which appear up front, then
> all those shell functions need to be parsed and remembered before the
> real executable code is executed.

This is a list off shell functions that appear in the generated libtool
script on my linux system (one of Ralf's patches is applied). Yes, we
could probably move these around some to get func_mode_compile closer to
the top.

func_dirname_and_basename ()
func_dirname ()
func_basename ()
func_dirname_and_basename ()
func_stripname ()
func_opt_split ()
func_lo2o ()
func_append ()
func_echo ()
func_verbose ()
func_error ()
func_warning ()
func_fatal_error ()
func_fatal_help ()
func_grep ()
func_mkdir_p ()
func_mktempdir ()
func_quote_for_eval ()
func_quote_for_expand ()
func_show_eval ()
func_show_eval_locale ()
func_version ()
func_usage ()
func_help ()
func_missing_arg ()
func_fatal_configuration ()
func_config ()
func_features ()
func_enable_tag ()
func_mode_help ()
func_check_version_match ()
func_lalib_p ()
func_lalib_unsafe_p ()
func_ltwrapper_script_p ()
func_ltwrapper_executable_p ()
func_ltwrapper_scriptname ()
func_ltwrapper_p ()
func_execute_cmds ()
func_source ()
func_win32_libid ()
func_infer_tag ()
func_generate_dlsyms ()
func_extract_an_archive ()
func_extract_archives ()
func_write_libtool_object ()
func_mode_compile ()
func_mode_execute ()
func_mode_finish ()
func_mode_install ()
func_emit_wrapper ()
func_emit_cwrapperexe_src ()
func_mode_link ()
func_mode_uninstall ()

Peter
-- 
Peter O'Gorman
http://pogma.com


___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Eric Blake]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

According to Bob Friesenhahn on 4/9/2008 7:01 PM:
| There may be value to extracting these shell functions into separate
| files which are sourced only when needed.  This way the overhead of
| parsing "linking" code is not encountered while compiling, and
| vice-versa.  Since there is surely overhead associated with opening and
| sourcing a file the chunk of code should be pretty large.

In fact, that's what autotest does in autoconf 2.62 - it distributes a
single file with all the tests in the tail of the file, where the shell
script at the head of the file parses the command line options, uses that
to drive an awk script to parse out the line numbers where the tests of
interest live, then uses the line number cache to drive a sed script to
parse out each test into a separate file for sourcing by the shell only
when it is needeed.  As crazy as this multi-process action sounds, it was
hands down faster than letting the shell parse all the test functions when
running a subset of the testsuite, without too much penalty when running
the entire testsuite.

- --
Don't work too hard, make some time for fun as well!

Eric Blake [EMAIL PROTECTED]
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkf9iLUACgkQ84KuGfSFAYAcIwCeNuLsON7gZ0XLG8k+U3cXydeT
2REAnjZOLd81d7Kw5Hdh3FH/PkhFbWPs
=LLpS
-END PGP SIGNATURE-


___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Peter O'Gorman]

Eric Blake wrote:
> According to Bob Friesenhahn on 4/9/2008 11:15 AM:
> | On Wed, 9 Apr 2008, Josh Triplett wrote:
> |>
> |> I tested against 1.5.26.  I'll give 2.2.2 a shot and see what I find.
> |> However, when I looked at 2.2.2, it still seems to have a
> |> multi-thousand-line shell script; do you just expect the benefit to
> |> come from the new shell-specific optimizations?
> |
> | Since shell scripts are not compiled, the size of a shell script has
> | very little to do with its execution time.
> 
> On the other hand, recent improvements in autoconf 2.62 proved that we
> were able to speed up testsuite performance by more than 10% by merely
> refactoring Autotest output to avoid shell parsing of code that would not
> be executed.  In other words, the time the shell spends on parsing its
> input, whether or not that input is executed, is not trivial.
> 

Just for fun, lets compare shells at parsing useless code.

(using bash)
$ for y in {1..100}; do echo "func_notused${y} () {" >> parse.sh; for x
in {1..1}; do echo foo >> parse.sh; done; echo '}' >> parse.sh;
done; echo 'echo Done' >> parse.sh

On linux:
$ time bash parse.sh
Done

real0m4.567s
user0m3.970s
sys 0m0.188s

$ time dash parse.sh
Done

real0m1.421s
user0m1.242s
sys 0m0.096s

$ time zsh parse.sh
Done

real0m1.635s
user0m1.293s
sys 0m0.161s

On mac os x:
$ time zsh parse.sh
Done

real0m1.429s
user0m1.176s
sys 0m0.193s

$ time bash parse.sh
Done

real0m4.921s
user0m4.706s
sys 0m0.215s

$ time ksh parse.sh
Done

real5m31.311s ***
user5m29.284s
sys 0m1.876s

I know that libtool has not yet reached a million+ lines of useless
shell functions though :)

Peter
-- 
Peter O'Gorman
http://pogma.com


___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Bob Friesenhahn]

On Wed, 9 Apr 2008, Eric Blake wrote:

| Since shell scripts are not compiled, the size of a shell script has
| very little to do with its execution time.

On the other hand, recent improvements in autoconf 2.62 proved that we
were able to speed up testsuite performance by more than 10% by merely
refactoring Autotest output to avoid shell parsing of code that would not
be executed.  In other words, the time the shell spends on parsing its
input, whether or not that input is executed, is not trivial.


This is true.  The script could be 10MB in size but if only the first 
five lines are ever executed the execution time should be similar to a 
similar five line script.  This means that parts which are executed 
often should be up front.


If there is a long list of shell functions which appear up front, then 
all those shell functions need to be parsed and remembered before the 
real executable code is executed.


There may be value to extracting these shell functions into separate 
files which are sourced only when needed.  This way the overhead of 
parsing "linking" code is not encountered while compiling, and 
vice-versa.  Since there is surely overhead associated with opening 
and sourcing a file the chunk of code should be pretty large.


Unfortunately, more files gives people more to complain about since a 
single large file is less offensive than many smaller files.


If the build-time overhead associated with libtool 2.2 (vs Dolt) is 
really only 13% or so, someone needs to make the call as to what is an 
acceptable level of runtime overhead before Dolt can be disposed of an 
unecessary idea.  Is the acceptable overhead 5%, 7%, 10%?  What is the 
magic number?


Bob
==
Bob Friesenhahn
[EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/



___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Eric Blake]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

According to Bob Friesenhahn on 4/9/2008 11:15 AM:
| On Wed, 9 Apr 2008, Josh Triplett wrote:
|>
|> I tested against 1.5.26.  I'll give 2.2.2 a shot and see what I find.
|> However, when I looked at 2.2.2, it still seems to have a
|> multi-thousand-line shell script; do you just expect the benefit to
|> come from the new shell-specific optimizations?
|
| Since shell scripts are not compiled, the size of a shell script has
| very little to do with its execution time.

On the other hand, recent improvements in autoconf 2.62 proved that we
were able to speed up testsuite performance by more than 10% by merely
refactoring Autotest output to avoid shell parsing of code that would not
be executed.  In other words, the time the shell spends on parsing its
input, whether or not that input is executed, is not trivial.

- --
Don't work too hard, make some time for fun as well!

Eric Blake [EMAIL PROTECTED]
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkf9WpAACgkQ84KuGfSFAYBXRACgwWkVTjy5WMdo8CHkVYcXg6fa
cO0An2v7JSz2Ss9lcnsVmNal3msA+fl7
=4zFK
-END PGP SIGNATURE-


___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Ralf Wildenhues]

Hi Ross,

* Ross Burton wrote on Wed, Apr 09, 2008 at 03:44:40PM CEST:
>
> EDS with libtool 1.5.26-3
> 
> real  4m35.934s

> EDS with libtool 2.1 (2.1a+cvs1.2525+20071016-1)
> 
> real  4m10.648s

> EDS with dolt
> 
> real  3m40.974s

OK, so we're already half-way there, and the speedup in Libtool 2.2
benefits not only with modern bash and on GNU/Linux, but all
XSI-compliant shells, on any system.

> I realise this isn't libtool 2.2, so I'm willing to believe there are
> further speedups.

No.  But 2.2.4 will have further speedups, if the patches I just posted
on libtool-patches are accepted.

Cheers,
Ralf


___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[David Johnson]

On Wednesday 09 April 2008 03:34:18 am Josh Triplett wrote:
> Meanwhile, modern systems such as GNU/Linux have reasonable library
> mechanisms, and need relatively little of the machinery in libtool.
> On these common systems, it would significantly improve build times to
> avoid running that libtool machinery for every compilation.

You keep using the plural, when your script specifically checks for a singular 
case.

It's not a replacement for libtool until it can support the overwhelming 
majority of modern systems, and not just GNU/Linux. FreeBSD, NetBSD, OpenBSD, 
OpenSolaris, Darwin.

-- 
David Johnson


___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Bob Friesenhahn]

On Wed, 9 Apr 2008, Josh Triplett wrote:


I tested against 1.5.26.  I'll give 2.2.2 a shot and see what I find.
However, when I looked at 2.2.2, it still seems to have a
multi-thousand-line shell script; do you just expect the benefit to
come from the new shell-specific optimizations?


Since shell scripts are not compiled, the size of a shell script has 
very little to do with its execution time.


Bob
==
Bob Friesenhahn
[EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/



___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Bob Friesenhahn]

On Wed, 9 Apr 2008, Ross Burton wrote:


I realise this isn't libtool 2.2, so I'm willing to believe there are further
speedups.


What I notice from your timings is that libtool 2.1 takes much less 
system time than libtool 1.5.26-3 but the user time is quite similar. 
The real time is surely most important, and it seems that the real 
time is improved with libtool 2.1 but not very dramatically.


The mystery as to where the time is actually going is why I have 
suggested several times that there should be an effort to test libtool 
under a DTrace-enhanced shell (shell with a DTrace provider) and 
perform an analysis of what is being done and where time is being 
spent using dtrace.  Even without a DTrace-enhanced shell, it is 
possible to see where time is going in a script using DTrace.


A DTrace-enhanced shell is going into OpenSolaris.  See 
"http://blogs.sun.com/tpenta/entry/psarc_2008_008_dtrace_provider";.


Quite often when I profile a program I find that something quite 
unexpected is hogging the time and the fix is quite trivial.  Perhaps 
we will find this with libtool as well.


For those of you who feel offended that I suggest using a tool which 
is only currently available under Solaris and OS-X Leopard, I find it 
to be no more insulting than a suggestion to run software under 
valgrind, which is only available under Linux (and sometimes on 
FreeBSD).


Bob
==
Bob Friesenhahn
[EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/



___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Peter O'Gorman]

Ralf Wildenhues wrote:
That gave the biggest speedup on
> GNU/Linux (where forks are relatively cheap).
> 

This entire message just goes to prove that I do not have a good memory.
I had completely forgotten that you sped up compile mode too.

Peter
-- 
Peter O'Gorman
http://pogma.com


___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Ralf Wildenhues]

* Josh Triplett wrote on Wed, Apr 09, 2008 at 06:02:36PM CEST:
> Ralf Wildenhues wrote:
> > 
> > Curious: can you please state which Libtool version you timed against,
> > and if not 2.2.x, redo timing against 2.2.2?  Not that I expect wonders,
> > but I expect something better than what you measured.
> 
> I tested against 1.5.26.  I'll give 2.2.2 a shot and see what I find.
> However, when I looked at 2.2.2, it still seems to have a
> multi-thousand-line shell script; do you just expect the benefit to
> come from the new shell-specific optimizations?

Partly, yes.  And Ross showed that we already got half the way, and yes,
the compile mode improvements are the biggest:

Libtool 2.2.2 in compile mode uses only 6 forks in compile mode if your
shell has XSI features.

Another thing to keep in mind is to put func_mode_compile early in the
script, and outside of conditionals, so that the shell doesn't even have
to parse anything below that.  That gave the biggest speedup on
GNU/Linux (where forks are relatively cheap).


That message also shows timings for a separated libtool-link script,
beyond which I then saw no further performance improvements.  I guess
I could try to dig out that old patch (which was never applied).


Also, I see two more low-hanging fruits at the moment.  Will follow up
with patches later tonight or tomorrow night.

Cheers,
Ralf


___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Peter O'Gorman]

Josh Triplett wrote:

> I tested against 1.5.26.  I'll give 2.2.2 a shot and see what I find.
> However, when I looked at 2.2.2, it still seems to have a
> multi-thousand-line shell script; do you just expect the benefit to
> come from the new shell-specific optimizations?
>

Hi Josh,

There are speedups for link mode, I don't think you will see much of an
improvement for compile mode over 1.5.26, but I do not have numbers to
back that up.

Splitting off libtool compile might be worthwhile, but there should be
no need for:

AC_MSG_CHECKING([if libtool sucks])
AC_MSG_RESULT([yup, it does])

:-)

Or to restrict the shell, compiler and OS to bash, gcc and GNU/linux. It
should be possible to have a portable (at least a more portable than
this) ltcompile script.

Are you willing to work on libtool to improve its performance for
compile mode? If so, we can send you copyright assignment forms etc.

Peter
-- 
Peter O'Gorman
http://pogma.com


___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Josh Triplett]

Ralf Wildenhues wrote:
> can we limit followups to a subset of this impressive array of mailing
> lists?  Say, to ?  That would be readable at
> .
> Thanks.

Sure.  I assumed that anyone wanting to respond would do so to the
mailing list they got it through if appropriate; for instance, I had a
followup on just debian-devel.

> * Josh Triplett wrote on Wed, Apr 09, 2008 at 12:34:18PM CEST:
>> Libtool knows how to handle libraries for umpteen different systems,
>> including many ancient systems that have terrible shared library
>> support.  It has some extensive shell script logic to figure out how
>> to build libraries for your system, and how to compile objects that go
>> in those libraries.  This logic does an amazingly impressive job of
>> coping with adverse conditions.  However, this logic all lives in an
>> ~8500 line, ~250kB shell script, which runs *every single time you
>> compile a source file*.
>>
>> This does not do wonders for performance.
> 
> Curious: can you please state which Libtool version you timed against,
> and if not 2.2.x, redo timing against 2.2.2?  Not that I expect wonders,
> but I expect something better than what you measured.

I tested against 1.5.26.  I'll give 2.2.2 a shot and see what I find.
However, when I looked at 2.2.2, it still seems to have a
multi-thousand-line shell script; do you just expect the benefit to
come from the new shell-specific optimizations?

- Josh Triplett



signature.asc
Description: OpenPGP digital signature
___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Ross Burton]

Ralf Wildenhues  gmx.de> writes:
> Curious: can you please state which Libtool version you timed against,
> and if not 2.2.x, redo timing against 2.2.2?  Not that I expect wonders,
> but I expect something better than what you measured.

I just tested this out on my laptop with Evolution Data Server:

EDS with libtool 1.5.26-3

real4m35.934s
user3m38.786s
sys 0m52.855s

EDS with libtool 2.1 (2.1a+cvs1.2525+20071016-1)

real4m10.648s
user3m37.994s
sys 0m26.578s

EDS with dolt

real3m40.974s
user3m4.448s
sys 0m31.406s

I realise this isn't libtool 2.2, so I'm willing to believe there are further
speedups.

Ross



___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Michel BRIAND]

Josh Triplett <[EMAIL PROTECTED]> - Wed, 09 Apr 2008 03:34:18 -0700

>Thus, I wrote Dolt, a drop-in replacement for libtool's compilation
>mode.

Excellent, impressive :)))


___
http://lists.gnu.org/mailman/listinfo/libtool

Re: Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Ralf Wildenhues]

Hello Josh,

can we limit followups to a subset of this impressive array of mailing
lists?  Say, to ?  That would be readable at
.
Thanks.

* Josh Triplett wrote on Wed, Apr 09, 2008 at 12:34:18PM CEST:
> Libtool knows how to handle libraries for umpteen different systems,
> including many ancient systems that have terrible shared library
> support.  It has some extensive shell script logic to figure out how
> to build libraries for your system, and how to compile objects that go
> in those libraries.  This logic does an amazingly impressive job of
> coping with adverse conditions.  However, this logic all lives in an
> ~8500 line, ~250kB shell script, which runs *every single time you
> compile a source file*.
> 
> This does not do wonders for performance.

Curious: can you please state which Libtool version you timed against,
and if not 2.2.x, redo timing against 2.2.2?  Not that I expect wonders,
but I expect something better than what you measured.

Thanks,
Ralf



___
http://lists.gnu.org/mailman/listinfo/libtool

Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Josh Triplett]

Many packages use GNU autotools (automake and autoconf) to build, to
the point that "./configure && make" represents one of the most common
build procedures for Free Software packages.  Libraries using
autotools typically use GNU Libtool, partly because it works on almost
any system and partly because autotools makes it difficult to do
otherwise.  Packages which use these libraries sometimes use libtool
as well.

Yet for many of these libraries and other packages, more than half of
the build time goes into running the libtool shell script.

Libtool knows how to handle libraries for umpteen different systems,
including many ancient systems that have terrible shared library
support.  It has some extensive shell script logic to figure out how
to build libraries for your system, and how to compile objects that go
in those libraries.  This logic does an amazingly impressive job of
coping with adverse conditions.  However, this logic all lives in an
~8500 line, ~250kB shell script, which runs *every single time you
compile a source file*.

This does not do wonders for performance.

Meanwhile, modern systems such as GNU/Linux have reasonable library
mechanisms, and need relatively little of the machinery in libtool.
On these common systems, it would significantly improve build times to
avoid running that libtool machinery for every compilation.

Thus, I wrote Dolt, a drop-in replacement for libtool's compilation
mode.  Dolt runs any necessary system-specific or
configuration-specific logic as part of configure, writes out a simple
shell script "doltcompile"[1], and substitutes it for libtool in the
automake variables LTCOMPILE and LTCXXCOMPILE.  If you use automake,
autoconf, and libtool, then using Dolt just requires two steps:

1) add "DOLT" after the call to LT_INIT, AC_PATH_LIBTOOL, or
   AM_PATH_LIBTOOL in your configure.ac or configure.in script, and
2) append dolt.m4 to your project's acinclude.m4.

For any system Dolt does not support, it will transparently fall back
to libtool.

dolt.m4 takes up less than 4kB; it writes out a minimal doltcompile
script which never forks except to run the compiler or to mkdir the
.libs directory if it doesn't already exist.  I have tested it with
various projects, and benchmarked[2] its performance against the same
projects using only libtool.  Results:

kdelibs without dolt: 8m6.115s
kdelibs with dolt:3m50.065s

gtk+-2.0 without dolt: 2m31.825s
gtk+-2.0 with dolt:1m33.858s

libx11 without dolt: 1m50.163s
libx11 with dolt:0m53.417s

libxml2 without dolt: 0m25.722s
libxml2 with dolt:0m19.576s

dbus without dolt: 0m20.062s
dbus with dolt:0m8.940s

I have attached a snapshot of dolt.m4 for convenience.  You can also
obtain the current version of Dolt from Git with:

git clone git://svcs.cs.pdx.edu/git/dolt

or download a snapshot tarball from
.

Please try Dolt with your project, and see if you get comparable
performance improvements.  If you want to make Dolt replace libtool on
your system, feel free to send me a patch to dolt.m4; just remember
the basic tenet that any logic must run at configure time, not build
time.  You can figure out what compiler flags libtool uses by running
"touch dummy.c && libtool --mode=compile gcc -c dummy.c -o dummy.lo";
that will print two compiler command lines, one for the shared object
and one for the static object.

Future directions:
* Support GNU/Linux on architectures other than x86 and x86-64.  I
  think most will work with exactly the same compiler flags, but I
  didn't want to add any architecture I couldn't test.
* Support other systems.
* Possibly try to run libtool on a dummy source file at configure time
  to figure out the necessary flags to use when building library
  objects, but that seems error-prone.
* Replace libtool --mode=link.
* Replace libtool --mode=install.
* Optionally stop installing .la files.
* Make dolt.m4's output of doltcompile cleaner.

- Josh Triplett

[1] "doltcompile" stands for "do ltcompile"; the alternate reading
"dolt compile" led to the name "dolt".

[2] General testing methodology:
* Run ./configure && make && make clean, to make sure it builds and to get
  everything cached.
* Get the "before" time: time make >/dev/null 2>&1
* Remove and re-extract the source.
* Add dolt.m4 to acinclude.m4 and add DOLT to configure.in or configure.ac.
* autoreconf -v -f -i && ./configure && make && make clean, to
  make sure it still builds and to get everything cached again.
* Get the "after" time: time make >/dev/null 2>&1
dnl dolt, a replacement for libtool
dnl Copyright © 2007-2008 Josh Triplett <[EMAIL PROTECTED]>
dnl Copying and distribution of this file, with or without modification,
dnl are permitted in any medium without royalty provided the copyright
dnl notice and this notice are preserved.
dnl
dnl To use dolt, invoke the DOLT macro immediately after the libtool macros.
dnl Optionally, copy this file into acinclud