Re: [PATCH] Re: libtool-2.4.2 is fine but libtool-2.4.6 very slow.
On Wednesday 23 of September 2015 16:09:49 Eric Blake wrote: > On 09/23/2015 03:37 PM, Pavel Raiskup wrote: > > > > >>From 5e8a4c5173f1aa0786e8eba15fb07bfe04b83862 Mon Sep 17 00:00:00 2001 > > From: Pavel Raiskup> > Date: Fri, 18 Sep 2015 23:17:07 +0200 > > Subject: [PATCH] libtoolize: fix infinite recursion in m4 > > > > Some projects use this construct in configure.ac: > > > > m4_define([version], m4_include([version]) > > Missing a ) > > The faulty package used: > > m4_define([version], m4_include(version)) Thanks for catching the ')' typo! > and the infinite recursion occurred _because_ the usage was underquoted. > But as you wrote things here, you have sufficient quoting that you > won't trigger infinite recursion. > You need to drop the second [] for > this to be an accurate representation of the failure. I intentionally added the '[ ]' quotes there as that is something which would look like correct usage, but it still goes to infinite recursion. The problem is that while defining 'version': m4_define([version], m4_include([version])) The expansion is like 'version => m4_include(version)'. Then, usage of 'version' goes to infinite recursion. > > pkg_version=version > > > > When the m4_include builtin is undefined (as was done in > > libtoolize and extract-trace scripts), the call to this 'version' > > macro gone to infinite recursion (until ENOMEM). So rather > > s/gone to infinite/enters an infinite/ > > > re-define all potentially dangerous macros by empty strings, > > suggested by Eric Blake. > > > > While we are on it, merge the macro-"blacklist" with similar list > > implemented in gettext, except the 'm4_esyscmd'. It's kept > > s/except the/except for/ > > > defined because we already trace AC_INIT macro for package > > version, while it is often specified by > > m4_esyscmd(git-version-gen). Similarly to m4_include, m4_esyscmd > > might be opt-in-blacklisted in future. > > > > References: > > http://lists.gnu.org/archive/html/libtool/2015-09/msg0.html > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=764580 > > > > With the commit message touched up, it looks okay to me. I'll fix the typos, thanks! Pavel ___ https://lists.gnu.org/mailman/listinfo/libtool
Re: [PATCH] Re: libtool-2.4.2 is fine but libtool-2.4.6 very slow.
On Wednesday 23 of September 2015 16:09:49 Eric Blake wrote: > On 09/23/2015 03:37 PM, Pavel Raiskup wrote: > > > > >>From 5e8a4c5173f1aa0786e8eba15fb07bfe04b83862 Mon Sep 17 00:00:00 2001 > > From: Pavel Raiskup> > Date: Fri, 18 Sep 2015 23:17:07 +0200 > > Subject: [PATCH] libtoolize: fix infinite recursion in m4 > > > > Some projects use this construct in configure.ac: > > > > m4_define([version], m4_include([version]) > > Missing a ) > > The faulty package used: > > m4_define([version], m4_include(version)) Thanks for catching the ')' typo! > and the infinite recursion occurred _because_ the usage was underquoted. > But as you wrote things here, you have sufficient quoting that you > won't trigger infinite recursion. > You need to drop the second [] for > this to be an accurate representation of the failure. I intentionally added the '[ ]' quotes there as that is something which would look like correct usage, but it still goes to infinite recursion. The problem is that while defining 'version': m4_define([version], m4_include([version])) The expansion is like 'version => m4_include(version)'. Then, usage of 'version' goes to infinite recursion. > > pkg_version=version > > > > When the m4_include builtin is undefined (as was done in > > libtoolize and extract-trace scripts), the call to this 'version' > > macro gone to infinite recursion (until ENOMEM). So rather > > s/gone to infinite/enters an infinite/ > > > re-define all potentially dangerous macros by empty strings, > > suggested by Eric Blake. > > > > While we are on it, merge the macro-"blacklist" with similar list > > implemented in gettext, except the 'm4_esyscmd'. It's kept > > s/except the/except for/ > > > defined because we already trace AC_INIT macro for package > > version, while it is often specified by > > m4_esyscmd(git-version-gen). Similarly to m4_include, m4_esyscmd > > might be opt-in-blacklisted in future. > > > > References: > > http://lists.gnu.org/archive/html/libtool/2015-09/msg0.html > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=764580 > > > > With the commit message touched up, it looks okay to me. I'll fix the typos, thanks! Pavel
Re: [PATCH] Re: libtool-2.4.2 is fine but libtool-2.4.6 very slow.
Hello Pavel. Thank you for your advice. Workaround worked perfectly!!. https://github.com/groonga/groonga/commit/7d7a62a2fba74334b23d0021a9a9e78da046b21c Thanks. 2015年9月19日(土) 17:09 Pavel Raiskup: > Hi Hiroyuki Sato, > > On Wednesday 02 of September 2015 16:00:34 Hiroyuki Sato wrote: > > This configure.ac is extreme slow on libtool-2.4.6. > > But It run smoothly on libtool-2.4.2. > > https://github.com/groonga/groonga/blob/master/configure.ac > > thanks for reproducer! > > This _really_ looks like issue mentioned [1], though the thread is > believed to be resolved (with existing small slowdown between > libtool-2.4.5 and 2.4.2). Let me CC Robert whether this patch does not > actually fix the "1 sec" slowdown of libtoolize. > > > Could you tell me how to fix this problem?. > > You can workaround/fix this by: > -m4_define([nginx_version], m4_include(nginx_version)) > -NGINX_VERSION=nginx_version > +NGINX_VERSION=m4_include([nginx_version]) > > See the "recursion" in 'nginx_version' definition. Its not recursion > usually (because nginx_version is not defined in the time of m4_define) > -- but libtoolize script does 'm4_undefine(m4_include)' before > evaluating configure.ac. > > > Is this libtool's Bug? > > I would say so, see the attached patch I'll push very soon (if there are > no objections). > > > * OS OSX 10.10.5. and CentOS7 > > Should be IMO reproducible everywhere. > > [1] http://lists.gnu.org/archive/html/libtool/2015-02/msg0.html > > Thanks, Pavel > -- > > > Thanks. > > > > libtool-2.4.6 > > > > time /tmp/libtool246/bin/libtoolize --verbose --copy --force > > libtoolize: found 'configure.ac' > > libtoolize: export M4='/usr/bin/gm4' > > > > It takes around 10 minutes. > > > > libtool-2.4.2 > > > > time /tmp/libtool242/bin/libtoolize --verbose --copy --force > > libtoolize: rm -f './ltmain.sh' > > libtoolize: putting auxiliary files in `.'. > > libtoolize: copying file from > `/tmp/libtool242/share/libtool/config/ltmain.sh' > > libtoolize: rm -f './libtool.m4' > > libtoolize: rm -f './ltoptions.m4' > > libtoolize: rm -f './ltsugar.m4' > > libtoolize: rm -f './ltversion.m4' > > libtoolize: rm -f './lt~obsolete.m4' > > libtoolize: Not copying `./argz.m4', libltdl not used. > > libtoolize: putting macros in `.'. > > libtoolize: copying file from > `/tmp/libtool242/share/aclocal/libtool.m4' > > libtoolize: Not copying `./ltdl.m4', libltdl not used. > > libtoolize: copying file from > `/tmp/libtool242/share/aclocal/ltoptions.m4' > > libtoolize: copying file from > `/tmp/libtool242/share/aclocal/ltsugar.m4' > > libtoolize: copying file from > `/tmp/libtool242/share/aclocal/ltversion.m4' > > libtoolize: copying file from > `/tmp/libtool242/share/aclocal/lt~obsolete.m4' > > libtoolize: Consider adding `AC_CONFIG_MACRO_DIR([m4])' to > configure.ac and > > libtoolize: rerunning libtoolize, to keep the correct libtool macros > in-tree. > > > > real 0m1.003s > > user 0m0.364s > > sys 0m0.540s > > > > > > >
Re: [PATCH] Re: libtool-2.4.2 is fine but libtool-2.4.6 very slow.
Hi Eric, thanks for your review! On Saturday 19 of September 2015 15:05:07 Eric Blake wrote: > On 09/19/2015 02:09 AM, Pavel Raiskup wrote: > > Hi Hiroyuki Sato, > > > > On Wednesday 02 of September 2015 16:00:34 Hiroyuki Sato wrote: > >> This configure.ac is extreme slow on libtool-2.4.6. > >> But It run smoothly on libtool-2.4.2. > >> https://github.com/groonga/groonga/blob/master/configure.ac > > > > thanks for reproducer! > > > > This _really_ looks like issue mentioned [1], though the thread is > > believed to be resolved (with existing small slowdown between > > libtool-2.4.5 and 2.4.2). Let me CC Robert whether this patch does not > > actually fix the "1 sec" slowdown of libtoolize. > > This looks very much like the same bug that gettext had: > http://git.savannah.gnu.org/cgit/gettext.git/commit/?id=d75090f2 > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=764580 Indeed, I missed that report. > Your proposed solution doesn't seem quite right: > > > # Disable these macros. > > m4_undefine([m4_dnl]) > > -m4_undefine([m4_include]) > > m4_undefine([m4_m4exit]) > > m4_undefine([m4_m4wrap]) > > m4_undefine([m4_maketemp]) > > +# Rather do not use undefine here because people tend to define > > +# macros of the same name as file included in their bodies - which > > +# results in infinite recursion. > > +m4_define([m4_include], []) > > I'd recommend that you use the same fix as gettext, and define ALL of > these macros to an empty string, rather than special-casing m4_include > as the only one that does not get undefined. Agreed, m4_pushdef([trace_breaking_macro]) should not cause any harm. I fixed the patch (attached). Pavel >From 5e8a4c5173f1aa0786e8eba15fb07bfe04b83862 Mon Sep 17 00:00:00 2001 From: Pavel RaiskupDate: Fri, 18 Sep 2015 23:17:07 +0200 Subject: [PATCH] libtoolize: fix infinite recursion in m4 Some projects use this construct in configure.ac: m4_define([version], m4_include([version]) pkg_version=version When the m4_include builtin is undefined (as was done in libtoolize and extract-trace scripts), the call to this 'version' macro gone to infinite recursion (until ENOMEM). So rather re-define all potentially dangerous macros by empty strings, suggested by Eric Blake. While we are on it, merge the macro-"blacklist" with similar list implemented in gettext, except the 'm4_esyscmd'. It's kept defined because we already trace AC_INIT macro for package version, while it is often specified by m4_esyscmd(git-version-gen). Similarly to m4_include, m4_esyscmd might be opt-in-blacklisted in future. References: http://lists.gnu.org/archive/html/libtool/2015-09/msg0.html https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=764580 * gl/build-aux/extract-trace (_G_mini): Redefine trace-breaking macros to empty strings rather than undefining those. Use 'dnl' for comments. * bootstrap: Likewise, sync with extract-trace. * NEWS: Document. * NO-THANKS: Mention Hiroyuki Sato. --- NEWS | 4 NO-THANKS | 1 + bootstrap | 44 gl/build-aux/extract-trace | 44 4 files changed, 61 insertions(+), 32 deletions(-) diff --git a/NEWS b/NEWS index c5c9023..82dd037 100644 --- a/NEWS +++ b/NEWS @@ -2,6 +2,10 @@ NEWS - list of user-visible changes between releases of GNU Libtool * Noteworthy changes in release ?.? (-??-??) [?] +** Bug fixes: + + - Fix significant slowdown of libtoolize for certain projects (regression +introduced in 2.4.3 release) caused by infinite m4 macro recursion. * Noteworthy changes in release 2.4.6 (2015-02-15) [stable] diff --git a/NO-THANKS b/NO-THANKS index 10de16b..51ec8eb 100644 --- a/NO-THANKS +++ b/NO-THANKS @@ -82,6 +82,7 @@ Donn Washburn n5...@comcast.net Erik van Pienbroek erik-...@vanpienbroek.nl Ethan Mallove ethan.mall...@sun.com Fred Cox sailorf...@yahoo.com +Hiroyuki Sato hiroys...@gmail.com Jakub Bogusz qbo...@pld-linux.org James Su james...@gmail.com Jay Krell jay.kr...@cornell.edu diff --git a/bootstrap b/bootstrap index 4596413..17fb169 100755 --- a/bootstrap +++ b/bootstrap @@ -2472,29 +2472,41 @@ func_extract_trace () # arguments to Autocof functions, but without following # 'm4_s?include' files. _G_mini=' -# Initialisation. +dnl Initialisation. m4_changequote([,]) m4_define([m4_copy], [m4_define([$2], m4_defn([$1]))]) m4_define([m4_rename], [m4_copy([$1], [$2])m4_undefine([$1])]) -# Disable these macros. -m4_undefine([m4_dnl]) -m4_undefine([m4_include]) -m4_undefine([m4_m4exit]) -m4_undefine([m4_m4wrap]) -m4_undefine([m4_maketemp]) - -# Copy and rename macros not handled by "m4 --prefix". +dnl Replace macros which may abort m4
Re: [PATCH] Re: libtool-2.4.2 is fine but libtool-2.4.6 very slow.
On 09/23/2015 03:37 PM, Pavel Raiskup wrote: > >>From 5e8a4c5173f1aa0786e8eba15fb07bfe04b83862 Mon Sep 17 00:00:00 2001 > From: Pavel Raiskup> Date: Fri, 18 Sep 2015 23:17:07 +0200 > Subject: [PATCH] libtoolize: fix infinite recursion in m4 > > Some projects use this construct in configure.ac: > > m4_define([version], m4_include([version]) Missing a ) The faulty package used: m4_define([version], m4_include(version)) and the infinite recursion occurred _because_ the usage was underquoted. But as you wrote things here, you have sufficient quoting that you won't trigger infinite recursion. You need to drop the second [] for this to be an accurate representation of the failure. > pkg_version=version > > When the m4_include builtin is undefined (as was done in > libtoolize and extract-trace scripts), the call to this 'version' > macro gone to infinite recursion (until ENOMEM). So rather s/gone to infinite/enters an infinite/ > re-define all potentially dangerous macros by empty strings, > suggested by Eric Blake. > > While we are on it, merge the macro-"blacklist" with similar list > implemented in gettext, except the 'm4_esyscmd'. It's kept s/except the/except for/ > defined because we already trace AC_INIT macro for package > version, while it is often specified by > m4_esyscmd(git-version-gen). Similarly to m4_include, m4_esyscmd > might be opt-in-blacklisted in future. > > References: > http://lists.gnu.org/archive/html/libtool/2015-09/msg0.html > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=764580 > With the commit message touched up, it looks okay to me. -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature
Re: [PATCH] Re: libtool-2.4.2 is fine but libtool-2.4.6 very slow.
On 09/23/2015 03:37 PM, Pavel Raiskup wrote: > >>From 5e8a4c5173f1aa0786e8eba15fb07bfe04b83862 Mon Sep 17 00:00:00 2001 > From: Pavel Raiskup> Date: Fri, 18 Sep 2015 23:17:07 +0200 > Subject: [PATCH] libtoolize: fix infinite recursion in m4 > > Some projects use this construct in configure.ac: > > m4_define([version], m4_include([version]) Missing a ) The faulty package used: m4_define([version], m4_include(version)) and the infinite recursion occurred _because_ the usage was underquoted. But as you wrote things here, you have sufficient quoting that you won't trigger infinite recursion. You need to drop the second [] for this to be an accurate representation of the failure. > pkg_version=version > > When the m4_include builtin is undefined (as was done in > libtoolize and extract-trace scripts), the call to this 'version' > macro gone to infinite recursion (until ENOMEM). So rather s/gone to infinite/enters an infinite/ > re-define all potentially dangerous macros by empty strings, > suggested by Eric Blake. > > While we are on it, merge the macro-"blacklist" with similar list > implemented in gettext, except the 'm4_esyscmd'. It's kept s/except the/except for/ > defined because we already trace AC_INIT macro for package > version, while it is often specified by > m4_esyscmd(git-version-gen). Similarly to m4_include, m4_esyscmd > might be opt-in-blacklisted in future. > > References: > http://lists.gnu.org/archive/html/libtool/2015-09/msg0.html > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=764580 > With the commit message touched up, it looks okay to me. -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature ___ https://lists.gnu.org/mailman/listinfo/libtool
Re: [PATCH] Re: libtool-2.4.2 is fine but libtool-2.4.6 very slow.
Hello Pavel. Thank you for your advice. Workaround worked perfectly!!. https://github.com/groonga/groonga/commit/7d7a62a2fba74334b23d0021a9a9e78da046b21c Thanks. 2015年9月19日(土) 17:09 Pavel Raiskup: > Hi Hiroyuki Sato, > > On Wednesday 02 of September 2015 16:00:34 Hiroyuki Sato wrote: > > This configure.ac is extreme slow on libtool-2.4.6. > > But It run smoothly on libtool-2.4.2. > > https://github.com/groonga/groonga/blob/master/configure.ac > > thanks for reproducer! > > This _really_ looks like issue mentioned [1], though the thread is > believed to be resolved (with existing small slowdown between > libtool-2.4.5 and 2.4.2). Let me CC Robert whether this patch does not > actually fix the "1 sec" slowdown of libtoolize. > > > Could you tell me how to fix this problem?. > > You can workaround/fix this by: > -m4_define([nginx_version], m4_include(nginx_version)) > -NGINX_VERSION=nginx_version > +NGINX_VERSION=m4_include([nginx_version]) > > See the "recursion" in 'nginx_version' definition. Its not recursion > usually (because nginx_version is not defined in the time of m4_define) > -- but libtoolize script does 'm4_undefine(m4_include)' before > evaluating configure.ac. > > > Is this libtool's Bug? > > I would say so, see the attached patch I'll push very soon (if there are > no objections). > > > * OS OSX 10.10.5. and CentOS7 > > Should be IMO reproducible everywhere. > > [1] http://lists.gnu.org/archive/html/libtool/2015-02/msg0.html > > Thanks, Pavel > -- > > > Thanks. > > > > libtool-2.4.6 > > > > time /tmp/libtool246/bin/libtoolize --verbose --copy --force > > libtoolize: found 'configure.ac' > > libtoolize: export M4='/usr/bin/gm4' > > > > It takes around 10 minutes. > > > > libtool-2.4.2 > > > > time /tmp/libtool242/bin/libtoolize --verbose --copy --force > > libtoolize: rm -f './ltmain.sh' > > libtoolize: putting auxiliary files in `.'. > > libtoolize: copying file from > `/tmp/libtool242/share/libtool/config/ltmain.sh' > > libtoolize: rm -f './libtool.m4' > > libtoolize: rm -f './ltoptions.m4' > > libtoolize: rm -f './ltsugar.m4' > > libtoolize: rm -f './ltversion.m4' > > libtoolize: rm -f './lt~obsolete.m4' > > libtoolize: Not copying `./argz.m4', libltdl not used. > > libtoolize: putting macros in `.'. > > libtoolize: copying file from > `/tmp/libtool242/share/aclocal/libtool.m4' > > libtoolize: Not copying `./ltdl.m4', libltdl not used. > > libtoolize: copying file from > `/tmp/libtool242/share/aclocal/ltoptions.m4' > > libtoolize: copying file from > `/tmp/libtool242/share/aclocal/ltsugar.m4' > > libtoolize: copying file from > `/tmp/libtool242/share/aclocal/ltversion.m4' > > libtoolize: copying file from > `/tmp/libtool242/share/aclocal/lt~obsolete.m4' > > libtoolize: Consider adding `AC_CONFIG_MACRO_DIR([m4])' to > configure.ac and > > libtoolize: rerunning libtoolize, to keep the correct libtool macros > in-tree. > > > > real 0m1.003s > > user 0m0.364s > > sys 0m0.540s > > > > > > > ___ https://lists.gnu.org/mailman/listinfo/libtool
Re: [PATCH] Re: libtool-2.4.2 is fine but libtool-2.4.6 very slow.
On 09/19/2015 02:09 AM, Pavel Raiskup wrote: > Hi Hiroyuki Sato, > > On Wednesday 02 of September 2015 16:00:34 Hiroyuki Sato wrote: >> This configure.ac is extreme slow on libtool-2.4.6. >> But It run smoothly on libtool-2.4.2. >> https://github.com/groonga/groonga/blob/master/configure.ac > > thanks for reproducer! > > This _really_ looks like issue mentioned [1], though the thread is > believed to be resolved (with existing small slowdown between > libtool-2.4.5 and 2.4.2). Let me CC Robert whether this patch does not > actually fix the "1 sec" slowdown of libtoolize. This looks very much like the same bug that gettext had: http://git.savannah.gnu.org/cgit/gettext.git/commit/?id=d75090f2 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=764580 Your proposed solution doesn't seem quite right: > # Disable these macros. > m4_undefine([m4_dnl]) > -m4_undefine([m4_include]) > m4_undefine([m4_m4exit]) > m4_undefine([m4_m4wrap]) > m4_undefine([m4_maketemp]) > +# Rather do not use undefine here because people tend to define > +# macros of the same name as file included in their bodies - which > +# results in infinite recursion. > +m4_define([m4_include], []) I'd recommend that you use the same fix as gettext, and define ALL of these macros to an empty string, rather than special-casing m4_include as the only one that does not get undefined. -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature
Re: [PATCH] Re: libtool-2.4.2 is fine but libtool-2.4.6 very slow.
On 09/19/2015 02:09 AM, Pavel Raiskup wrote: > Hi Hiroyuki Sato, > > On Wednesday 02 of September 2015 16:00:34 Hiroyuki Sato wrote: >> This configure.ac is extreme slow on libtool-2.4.6. >> But It run smoothly on libtool-2.4.2. >> https://github.com/groonga/groonga/blob/master/configure.ac > > thanks for reproducer! > > This _really_ looks like issue mentioned [1], though the thread is > believed to be resolved (with existing small slowdown between > libtool-2.4.5 and 2.4.2). Let me CC Robert whether this patch does not > actually fix the "1 sec" slowdown of libtoolize. This looks very much like the same bug that gettext had: http://git.savannah.gnu.org/cgit/gettext.git/commit/?id=d75090f2 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=764580 Your proposed solution doesn't seem quite right: > # Disable these macros. > m4_undefine([m4_dnl]) > -m4_undefine([m4_include]) > m4_undefine([m4_m4exit]) > m4_undefine([m4_m4wrap]) > m4_undefine([m4_maketemp]) > +# Rather do not use undefine here because people tend to define > +# macros of the same name as file included in their bodies - which > +# results in infinite recursion. > +m4_define([m4_include], []) I'd recommend that you use the same fix as gettext, and define ALL of these macros to an empty string, rather than special-casing m4_include as the only one that does not get undefined. -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature ___ https://lists.gnu.org/mailman/listinfo/libtool