Re: [PATCH 3/4] maint: pick XSI funcs at runtime, not configure time.
Gary V. Vaughan skrev 2011-11-25 09:57: Determine, on a function by function basis, what XSI features are available in the shell that is actually running the script, rather than the one that was picked at configure time by the re-execution engine. Doesn't this mean that the libtool script will do a number of extra forks each time it is run? If that's the case, I would like to know how that impacts the execution time on Cygwin/MinGW... Cheers, Peter
Re: [PATCH 3/4] maint: pick XSI funcs at runtime, not configure time.
Hi Peter, On 28 Nov 2011, at 15:48, Peter Rosin wrote: Gary V. Vaughan skrev 2011-11-25 09:57: Determine, on a function by function basis, what XSI features are available in the shell that is actually running the script, rather than the one that was picked at configure time by the re-execution engine. Doesn't this mean that the libtool script will do a number of extra forks each time it is run? Well, yes and no. There will be an extra 4 or 8 evals (depending on whether the XSI function is chosen) as libtool starts up when compared to 2.4.2, but the fast_funcs will each save a fork or two on every invocation compared to if libtool configure found a non- XSI shell when libtool was installed. However, don't forget that almost always, libtool is generated by the configure script of the project that uses it, and the loss of seds, evals, and file renames at configure time to splice in the fast_funcs most likely means you'll be at a net win on total execution time for at least a couple of dozen libtool invocations. And let's not forget this actually fixes a bug... we've had reports of configure re-execing with a good shell (say /opt/bin/bash) and substituting in the XSI fast_funcs, but the libtool script doesn't have all the slow re-exec machinery and runs with /bin/sh which then chokes on the XSI extensions spliced in to libtool, unless the caller manually sets SHELL and CONFIG_SHELL in their environment before calling make (which is a bad idea for other reasons). If that's the case, I would like to know how that impacts the execution time on Cygwin/MinGW... I don't have access to Windows so I can't actually measure it, but I doubt that the difference will be noticeable for even a large project, and quite possibly a small project like libltdl that only runs libtool a hand full of times will actually be (imperceptibly) faster. Cheers, -- Gary V. Vaughan (gary AT gnu DOT org)
Re: [PATCH 3/4] maint: pick XSI funcs at runtime, not configure time.
Gary V. Vaughan skrev 2011-11-28 10:20: Hi Peter, On 28 Nov 2011, at 15:48, Peter Rosin wrote: Gary V. Vaughan skrev 2011-11-25 09:57: Determine, on a function by function basis, what XSI features are available in the shell that is actually running the script, rather than the one that was picked at configure time by the re-execution engine. Doesn't this mean that the libtool script will do a number of extra forks each time it is run? Well, yes and no. There will be an extra 4 or 8 evals (depending on whether the XSI function is chosen) as libtool starts up when That's the evals in sub-shells, right? Those with this pattern: if $_use_fast_funcs (eval ...) 2/dev/null then eval '... or this: if (eval ...) 2/dev/null then eval '... compared to 2.4.2, but the fast_funcs will each save a fork or two on every invocation compared to if libtool configure found a non- XSI shell when libtool was installed. However, don't forget that almost always, libtool is generated by the configure script of the project that uses it, and the loss of seds, evals, and file renames at configure time to splice in the fast_funcs most likely means you'll be at a net win on total execution time for at least a couple of dozen libtool invocations. And let's not forget this actually fixes a bug... we've had reports of configure re-execing with a good shell (say /opt/bin/bash) and substituting in the XSI fast_funcs, but the libtool script doesn't have all the slow re-exec machinery and runs with /bin/sh which then chokes on the XSI extensions spliced in to libtool, unless the caller manually sets SHELL and CONFIG_SHELL in their environment before calling make (which is a bad idea for other reasons). If that's the case, I would like to know how that impacts the execution time on Cygwin/MinGW... I don't have access to Windows so I can't actually measure it, but I doubt that the difference will be noticeable for even a large project, and quite possibly a small project like libltdl that only runs libtool a hand full of times will actually be (imperceptibly) faster. I think you are underestimating the Cygwin fork pain. BTW, libltdl runs libtool more like 20 times, but I agree it's very small and the number of forks will not matter much. My typical use case is mid-sized at a magnitude or so larger, and even there with a fork rate of approx 10-15 Hz as I'm seeing, it wouldn't be too harsh with a couple of extra forks - a minutes or so on the wall clock time. But it would really add to the pain on some (hypothetical?) large project with thousands of libtool invocations. That's all I'm saying, but *I* am not building any of those... Hmm, looking closer, shouldn't _use_fast_funcs have some kind of lt prefix? Cheers, Peter
Re: [PATCH 3/4] maint: pick XSI funcs at runtime, not configure time.
Hi Peter, On 28 Nov 2011, at 18:05, Peter Rosin wrote: Gary V. Vaughan skrev 2011-11-28 10:20: On 28 Nov 2011, at 15:48, Peter Rosin wrote: Gary V. Vaughan skrev 2011-11-25 09:57: Determine, on a function by function basis, what XSI features are available in the shell that is actually running the script, rather than the one that was picked at configure time by the re-execution engine. Doesn't this mean that the libtool script will do a number of extra forks each time it is run? Well, yes and no. There will be an extra 4 or 8 evals (depending on whether the XSI function is chosen) as libtool starts up when That's the evals in sub-shells, right? Those with this pattern: if $_use_fast_funcs (eval ...) 2/dev/null then eval '... or this: if (eval ...) 2/dev/null then eval '... Yes, that's them. [[snip]] If that's the case, I would like to know how that impacts the execution time on Cygwin/MinGW... I don't have access to Windows so I can't actually measure it, but I doubt that the difference will be noticeable for even a large project, and quite possibly a small project like libltdl that only runs libtool a hand full of times will actually be (imperceptibly) faster. I think you are underestimating the Cygwin fork pain. BTW, libltdl runs libtool more like 20 times, but I agree it's very small and the number of forks will not matter much. My typical use case is mid-sized at a magnitude or so larger, and even there with a fork rate of approx 10-15 Hz as I'm seeing, it wouldn't be too harsh with a couple of extra forks - a minutes or so on the wall clock time. But it would really add to the pain on some (hypothetical?) large project with thousands of libtool invocations. That's all I'm saying, but *I* am not building any of those... Maybe I should switch the sense of the _use_fast_funcs, so that the pattern becomes: if $libtool_force_fast_funcs || (eval ...) 2/dev/null then eval '... And then recommend that cygwin users always use an XSI and += capable shell, with libtool_force_fast_funcs=: in the environment to cut out the forks? Hmm, looking closer, shouldn't _use_fast_funcs have some kind of lt prefix? Good point. I'll add a prefix before pushing. Thanks. Cheers, -- Gary V. Vaughan (gary AT gnu DOT org)
Re: [PATCH 3/4] maint: pick XSI funcs at runtime, not configure time.
On Mon, 28 Nov 2011, Peter Rosin wrote: My typical use case is mid-sized at a magnitude or so larger, and even there with a fork rate of approx 10-15 Hz as I'm seeing, it wouldn't be too harsh with a couple of extra forks - a minutes or so on the wall clock time. But it would really add to the pain on some (hypothetical?) large project with thousands of libtool invocations. That's all I'm saying, but *I* am not building any of those... Is Windows fork performance observed to improve linearly as processor cores are added, or does it maintain pretty much a fixed rate? If it does not improve linearly as processor cores are added, then the extra forks will severely impact available performance of parallel builds. I have become used to seeing substantial speedup with 'make -j 4' on a Windows system with four cores. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: [PATCH 01/10] tests: migrate legacy tests/cdemo tests to Autotest.
On 11/25/2011 11:57 PM, Gary V. Vaughan wrote: On 26 Nov 2011, at 11:39, Charles Wilson wrote: a) This is a big holiday weekend in the US, so...a bit more than 72 hours is indicated. Most of us will still be catching up on post-holiday $realjob stuff by the time 72 hours expires. Ah, didn't think of that. Sure, I will be busy myself for a week or two, so I won't push for at least a week, probably more. Thanks. b) cygwin? mingw? msvc? I'm afraid I don't have (or want) access to any Windows machines, so I'm afraid that I am relying on you guys to tell me if I screwed up. Of course I'm not expecting you to debug or fix my mistakes for me, and I'm not anticipating any new problems, since everything is merely migrated from legacy testsuite to Autotest testsuite, with minimal changes required to keep everything working on my main machines. Hrm, well, not so much. See below. Although I do normally have access to more machines, the flooding in Bangkok has made any use of my Internet connection other than email intolerably slow... hence the recent flurry of work on libtool (which I can do offline, queueing emails for when my connection is next up) Ah, well, $realjob's loss, our gain. to fill my time while I wait for things to get back to normal. The reason I'll be too busy to do much more of that over the next week or two, is that last night I actually had a full-speed connection to the US again, so I'm anticipating playing catchup at $realjob myself. Sorry if I seem a bit short, but I'm rather annoyed to see my queue get filled with hours upon hours of work in the middle of a holiday. Please don't feel that it's your responsibility to painstakingly pick through every patch I post... I'd be more than happy just to get the test logs from anything I put on alpha.gnu.org for the architectures I don't use to help me restabilize the code closer to a release. Full test logs for failed cygwin tests sent privately (1MB). Enjoy the rest of your holiday, and sorry for working so much on libtool recently: Well, I really wasn't complaining about the *work* per se -- it's great that somebody is finally tackling some of those gee-wouldn't-it-be-nice issues, like *FINALLY* switching over to autotest throughout, with all the attendant benefits. It's just I didn't want to have to run the testsuite, on three platforms, over the holidays in order to meet the 72hour deadline. although my objective with the recent modernisations has been to try to decruft libtool a little, and to make the barrier to contribution much lower than it is currently if at all possible. decrufting is good. I rarely have the chance to put a lot of time into libtool, and things will slow down considerably again now if my Internet connection really is back to (something like) normal again. Yep, when the tool mostly Just Works the motivation to allocate scarce resources (like personal free time) to it is somewhat lacking. I think that's true for all of us. I have another 20 or so patches left incubating in my unpublished queue, and I will be done for now after those are polished and pushed over the next month or two. Too bad. If your inet stayed down longer, I was going to suggest implementing the long-desired if $CC=gcc $gnuld == yes; then use compiler driver to link rather than ld, for all languages optimization -- thus getting rid of the predeps/postdeps/prelibs/postlibs kludginess for GNU compilers (incl. cygming). Oh well. :-) Attached, see test log for $host=cygwin. I had to use 'make -k check gl_public_submodule_commit=' -- I'm not sure why, but perhaps your working tree is using private gnulib mods? I'll send testsuite.dir privately. -- Chuck make check-recursive make[1]: Entering directory `/usr/src/packages/libtool/git/build-libtool-master' GEN../libtool/tests/defs.in GENtests/defs Making check in . make[2]: Entering directory `/usr/src/packages/libtool/git/build-libtool-master' make check-TESTS check-local make[3]: Entering directory `/usr/src/packages/libtool/git/build-libtool-master' make[4]: Entering directory `/usr/src/packages/libtool/git/build-libtool-master' PASS: tests/link.test PASS: tests/link-2.test PASS: tests/nomode.test PASS: tests/objectlist.test PASS: tests/quote.test PASS: tests/suffix.test PASS: tests/tagtrace.test == All 7 tests passed == make[4]: Leaving directory `/usr/src/packages/libtool/git/build-libtool-master' GEN../libtool/tests/package.m4 GEN../libtool/tests/testsuite CC libltdl/loaders/libltdl_libltdlc_la-preopen.lo CC libltdl/libltdl_libltdlc_la-lt__alloc.lo CC libltdl/libltdl_libltdlc_la-lt_dlloader.lo CC libltdl/libltdl_libltdlc_la-lt_error.lo CC libltdl/libltdl_libltdlc_la-ltdl.lo CC libltdl/libltdl_libltdlc_la-slist.lo CCLD libltdl/libltdlc.la ## - ## ## GNU Libtool 2.4.2.133-fe91d-dirty test suite.
