Re: Performance issue of libtool-2.4.4
Great, I verified that we nearly get the speed back:
When build xz:
libtool-2.4.5 libtool-2.4.2
bash:14s13s
dash:12s11s
// Robert
On 02/10/2015 06:35 PM, Richard Purdie wrote:
On Mon, 2015-02-09 at 23:36 +, Richard Purdie wrote:
On Mon, 2015-02-09 at 13:05 +, Richard Purdie wrote:
In an effort to get to the bottom of this I made a git bisection, timing
the performance of building xz with make -j1 using each different
libtool.
The issues come down to this commit:
http://git.savannah.gnu.org/cgit/libtool.git/commit/?id=0a42997c6032b9550a009a271552e811bfbcc430
libtool: rewritten over funclib.sh instead of general.m4sh.
Before that, I get a time of about 20s, after it, 39s. If I cherry-pick
in the fix in master mentioned above, I get 27s.
So whilst things are better (thanks!), the above change is still causing
a regression in the performance somewhere else. Any ideas what else in
that rather large change may be causing this?
To further narrow this down, of the changes in the above commit, the
problem appears to be in the changes to the option parsing code. I've
included the diff below which if I apply on top of the above, I get the
speed back. I've left the func_split_short_opt/func_split_long_opt code
in there but that is worth a tiny part of the speed, the issues are
around the addition of the func_options call.
As yet I don't know enough about the code in question to know why this
is an issue but traces of libtool show a lot more looping in code to do
with argument parsing and quoting.
To be more specific, if I take my "good" libtool and add:
func_options_prep ${1+"$@"}
it slows the build down by 0.5s on a 21s build. If I look at
func_options_prep and comment out the line:
func_run_hooks func_options_prep ${1+"$@"}
I get the 0.5s back.
In func_run_hooks, if I comment:
func_quote_for_eval ${1+"$@"}
func_run_hooks_result=$func_quote_for_eval_result
I get the 0.5s back. The issue is all the quoting of the various return
values through all this looping. It doesn't appear to be hitting the
printf/sed in func_quote_for_eval which would be an obvious slow path,
its just the shear number of loops run through with the commandline
arguments. The change adds a number of calls to func_run_hooks, not just
the single test case I have above and all combined, it slows things down
significantly.
So is there a way we can change things so its not calling
func_quote_for_eval all the time with all the looping that entails?
Cheers,
Richard
___
https://lists.gnu.org/mailman/listinfo/libtool
libtool and sysroot
I noted that --with-sysroot=yes calls(=yes seems undocumented?) lt_sysroot=`$CC --print-sysroot 2>/dev/null` Should not $CFLAGS be included here too as gcc may have its sysroot changed with --sysroot? Jocke ___ https://lists.gnu.org/mailman/listinfo/libtool
Re: Performance issue of libtool-2.4.4
On Tue, 2015-02-10 at 10:53 +, Gary V. Vaughan wrote:
> On Feb 10, 2015, at 10:35 AM, Richard Purdie
> wrote:
> > On Mon, 2015-02-09 at 23:36 +, Richard Purdie wrote:
> >> On Mon, 2015-02-09 at 13:05 +, Richard Purdie wrote:
> >>> In an effort to get to the bottom of this I made a git bisection, timing
> >>> the performance of building xz with make -j1 using each different
> >>> libtool.
> >>>
> >>> The issues come down to this commit:
> >>>
> >>> http://git.savannah.gnu.org/cgit/libtool.git/commit/?id=0a42997c6032b9550a009a271552e811bfbcc430
> >>>
> >>> libtool: rewritten over funclib.sh instead of general.m4sh.
> >>>
> >>> Before that, I get a time of about 20s, after it, 39s. If I cherry-pick
> >>> in the fix in master mentioned above, I get 27s.
> >>>
> >>> So whilst things are better (thanks!), the above change is still causing
> >>> a regression in the performance somewhere else. Any ideas what else in
> >>> that rather large change may be causing this?
> >>
> >> To further narrow this down, of the changes in the above commit, the
> >> problem appears to be in the changes to the option parsing code. I've
> >> included the diff below which if I apply on top of the above, I get the
> >> speed back. I've left the func_split_short_opt/func_split_long_opt code
> >> in there but that is worth a tiny part of the speed, the issues are
> >> around the addition of the func_options call.
> >>
> >> As yet I don't know enough about the code in question to know why this
> >> is an issue but traces of libtool show a lot more looping in code to do
> >> with argument parsing and quoting.
> >
> > To be more specific, if I take my "good" libtool and add:
> >
> > func_options_prep ${1+"$@"}
> >
> > it slows the build down by 0.5s on a 21s build. If I look at
> > func_options_prep and comment out the line:
> >
> >func_run_hooks func_options_prep ${1+"$@"}
> >
> > I get the 0.5s back.
> >
> > In func_run_hooks, if I comment:
> >
> >func_quote_for_eval ${1+"$@"}
> >func_run_hooks_result=$func_quote_for_eval_result
> >
> > I get the 0.5s back. The issue is all the quoting of the various return
> > values through all this looping. It doesn't appear to be hitting the
> > printf/sed in func_quote_for_eval which would be an obvious slow path,
> > its just the shear number of loops run through with the commandline
> > arguments. The change adds a number of calls to func_run_hooks, not just
> > the single test case I have above and all combined, it slows things down
> > significantly.
> >
> > So is there a way we can change things so its not calling
> > func_quote_for_eval all the time with all the looping that entails?
>
> One possibility would be to add a post-processing script that rewrites
> the hook functions used by func_options into a a single top-level blob of
> sequential shell code. I imagine that adding some carefully chosen comment
> strings would make extracting the right parts in the right order relatively
> straight forward... it might even be that inlining func_options_prep and
> any hook functions it calls would be enough?
Thanks for the background info on this. I understand the need to change
and improve the software so I'm not proposing reverting, I do think
there has to be some way to get some of the speed here back though.
I have a bit of a pressing need to have a things performing as they were
and I'd prefer to stay on libtool 2.4.5 than revert back to 2.4.2 so I
cooked up the patch I've included below. This basically manually unrolls
the key problematic parts, cut and paste from options-parser. With this
applied to master, I have a build time of 22s compared to 20s or 21s
(need to go and retest as I'm forgetting numbers) with 2.4.2.
For now I'll probably merge something like this into the Yocto
Project/OpenEmbedded, the exact way to solve this longer term is TBD.
Cheers,
Richard
diff --git a/build-aux/ltmain.in b/build-aux/ltmain.in
index d5cf07a..0f54303 100644
--- a/build-aux/ltmain.in
+++ b/build-aux/ltmain.in
@@ -342,11 +342,15 @@ _LT_EOF
# libtool_options_prep [ARG]...
# -
# Preparation for options parsed by libtool.
-libtool_options_prep ()
-{
+#libtool_options_prep ()
+#{
$debug_mode
# Option defaults:
+opt_verbose=false
+opt_warning_types=
+
+# Option defaults:
opt_config=false
opt_dlopen=
opt_dry_run=false
@@ -382,19 +386,14 @@ libtool_options_prep ()
shift; set dummy --mode uninstall ${1+"$@"}; shift
;;
esac
-
-# Pass back the list of options.
-func_quote_for_eval ${1+"$@"}
-libtool_options_prep_result=$func_quote_for_eval_result
-}
-func_add_hook func_options_prep libtool_options_prep
+#}
# libtool_parse_options [ARG]...
# -
# Provide handling for libtool specific options.
-libtool_parse_options ()
-{
+#libtool_parse_options ()
+#{
$debug_cmd
# Perform our own loop to consume as many options as
Re: Performance issue of libtool-2.4.4
Hi Richard,
On Feb 10, 2015, at 10:35 AM, Richard Purdie
wrote:
> On Mon, 2015-02-09 at 23:36 +, Richard Purdie wrote:
>> On Mon, 2015-02-09 at 13:05 +, Richard Purdie wrote:
>>> In an effort to get to the bottom of this I made a git bisection, timing
>>> the performance of building xz with make -j1 using each different
>>> libtool.
>>>
>>> The issues come down to this commit:
>>>
>>> http://git.savannah.gnu.org/cgit/libtool.git/commit/?id=0a42997c6032b9550a009a271552e811bfbcc430
>>>
>>> libtool: rewritten over funclib.sh instead of general.m4sh.
>>>
>>> Before that, I get a time of about 20s, after it, 39s. If I cherry-pick
>>> in the fix in master mentioned above, I get 27s.
>>>
>>> So whilst things are better (thanks!), the above change is still causing
>>> a regression in the performance somewhere else. Any ideas what else in
>>> that rather large change may be causing this?
>>
>> To further narrow this down, of the changes in the above commit, the
>> problem appears to be in the changes to the option parsing code. I've
>> included the diff below which if I apply on top of the above, I get the
>> speed back. I've left the func_split_short_opt/func_split_long_opt code
>> in there but that is worth a tiny part of the speed, the issues are
>> around the addition of the func_options call.
>>
>> As yet I don't know enough about the code in question to know why this
>> is an issue but traces of libtool show a lot more looping in code to do
>> with argument parsing and quoting.
>
> To be more specific, if I take my "good" libtool and add:
>
> func_options_prep ${1+"$@"}
>
> it slows the build down by 0.5s on a 21s build. If I look at
> func_options_prep and comment out the line:
>
>func_run_hooks func_options_prep ${1+"$@"}
>
> I get the 0.5s back.
>
> In func_run_hooks, if I comment:
>
>func_quote_for_eval ${1+"$@"}
>func_run_hooks_result=$func_quote_for_eval_result
>
> I get the 0.5s back. The issue is all the quoting of the various return
> values through all this looping. It doesn't appear to be hitting the
> printf/sed in func_quote_for_eval which would be an obvious slow path,
> its just the shear number of loops run through with the commandline
> arguments. The change adds a number of calls to func_run_hooks, not just
> the single test case I have above and all combined, it slows things down
> significantly.
>
> So is there a way we can change things so its not calling
> func_quote_for_eval all the time with all the looping that entails?
One possibility would be to add a post-processing script that rewrites
the hook functions used by func_options into a a single top-level blob of
sequential shell code. I imagine that adding some carefully chosen comment
strings would make extracting the right parts in the right order relatively
straight forward... it might even be that inlining func_options_prep and
any hook functions it calls would be enough?
Cheers,
--
Gary V. Vaughan (gary AT gnu DOT org)
___
https://lists.gnu.org/mailman/listinfo/libtool
Re: Performance issue of libtool-2.4.4
Hi Richard, > On Feb 9, 2015, at 11:36 PM, Richard Purdie > wrote: > > On Mon, 2015-02-09 at 13:05 +, Richard Purdie wrote: >> In an effort to get to the bottom of this I made a git bisection, timing >> the performance of building xz with make -j1 using each different >> libtool. >> >> The issues come down to this commit: >> >> http://git.savannah.gnu.org/cgit/libtool.git/commit/?id=0a42997c6032b9550a009a271552e811bfbcc430 >> >> libtool: rewritten over funclib.sh instead of general.m4sh. >> >> Before that, I get a time of about 20s, after it, 39s. If I cherry-pick >> in the fix in master mentioned above, I get 27s. >> >> So whilst things are better (thanks!), the above change is still causing >> a regression in the performance somewhere else. Any ideas what else in >> that rather large change may be causing this? > > To further narrow this down, of the changes in the above commit, the > problem appears to be in the changes to the option parsing code. I've > included the diff below which if I apply on top of the above, I get the > speed back. I've left the func_split_short_opt/func_split_long_opt code > in there but that is worth a tiny part of the speed, the issues are > around the addition of the func_options call. Thanks for the analysis and patch. > As yet I don't know enough about the code in question to know why this > is an issue but traces of libtool show a lot more looping in code to do > with argument parsing and quoting. For background, one of the larger changes between 2.4.2 and 2.4.3 was to move from separately maintained options code in raw shell in libtoolize and generated by m4 in libtool, to the unified generalized shell options parser from bootstrap, kept in gl/build-aux/options-parser. The benefits of this include time saved in maintenance of only one options parsing implementation over three separate ones (I maintain bootstrap too), and a lot more flexibility in the shell function hooking code -- ultimately, this will enable splitting libltdl into a separate project, which can inject ltdl specific code into the already installed libtoolize script. I was aware this would result in a somewhat slower libtool, though I'm surprised that it is as large a difference as you have timed. I'd be very happy to find a way to patch gl/build-aux/option-parser or its clients to regain some of that speed, but I'm extremely reluctant to revert to the hokey mishmash of m4 and shell that 2.4.2 was using purely for the sake of speed -- it might be that for extremely large projects you'll want to stick with 2.4.2 until we find a way to restore the necessary speed in some future release? Cheers, -- Gary V. Vaughan (gary AT gnu DOT org) ___ https://lists.gnu.org/mailman/listinfo/libtool
Re: Performance issue of libtool-2.4.4
On Mon, 2015-02-09 at 23:36 +, Richard Purdie wrote:
> On Mon, 2015-02-09 at 13:05 +, Richard Purdie wrote:
> > In an effort to get to the bottom of this I made a git bisection, timing
> > the performance of building xz with make -j1 using each different
> > libtool.
> >
> > The issues come down to this commit:
> >
> > http://git.savannah.gnu.org/cgit/libtool.git/commit/?id=0a42997c6032b9550a009a271552e811bfbcc430
> >
> > libtool: rewritten over funclib.sh instead of general.m4sh.
> >
> > Before that, I get a time of about 20s, after it, 39s. If I cherry-pick
> > in the fix in master mentioned above, I get 27s.
> >
> > So whilst things are better (thanks!), the above change is still causing
> > a regression in the performance somewhere else. Any ideas what else in
> > that rather large change may be causing this?
>
> To further narrow this down, of the changes in the above commit, the
> problem appears to be in the changes to the option parsing code. I've
> included the diff below which if I apply on top of the above, I get the
> speed back. I've left the func_split_short_opt/func_split_long_opt code
> in there but that is worth a tiny part of the speed, the issues are
> around the addition of the func_options call.
>
> As yet I don't know enough about the code in question to know why this
> is an issue but traces of libtool show a lot more looping in code to do
> with argument parsing and quoting.
To be more specific, if I take my "good" libtool and add:
func_options_prep ${1+"$@"}
it slows the build down by 0.5s on a 21s build. If I look at
func_options_prep and comment out the line:
func_run_hooks func_options_prep ${1+"$@"}
I get the 0.5s back.
In func_run_hooks, if I comment:
func_quote_for_eval ${1+"$@"}
func_run_hooks_result=$func_quote_for_eval_result
I get the 0.5s back. The issue is all the quoting of the various return
values through all this looping. It doesn't appear to be hitting the
printf/sed in func_quote_for_eval which would be an obvious slow path,
its just the shear number of loops run through with the commandline
arguments. The change adds a number of calls to func_run_hooks, not just
the single test case I have above and all combined, it slows things down
significantly.
So is there a way we can change things so its not calling
func_quote_for_eval all the time with all the looping that entails?
Cheers,
Richard
___
https://lists.gnu.org/mailman/listinfo/libtool
