Re: [MacOS X] consider useshrplib='false' by default
On Wed, 4 Jun 2003, Jarkko Hietaniemi wrote: Sounds reasonable to make the useshrplib to default to false (because of the significant startup slowness otherwise) and at the very least make it conditional (and I got a nod from Ed Moy of Apple, too). I had thought that the shared libperl provided by Apple was actually used by at least one other common application (and that not breaking that application was part of the background for Apple's original choice of installation location for their shared libperl). I've never used Mac OS X, so I don't have any firsthand knowledge either way. So I did (change #19681). Since useshrplib='false' is the default anyway, I suggest the following minor editorial change: diff -u perl-5.8.x/hints/darwin.sh perl-5.8.x-andy/hints/darwin.sh --- perl-5.8.x/hints/darwin.sh Wed Jun 4 02:40:33 2003 +++ perl-5.8.x-andy/hints/darwin.sh Wed Jun 4 11:18:23 2003 @@ -141,9 +141,7 @@ ldlibpthname='DYLD_LIBRARY_PATH'; # useshrplib=true results in much slower startup times. -case $useshrplib in -'') useshrplib='false' ;; -esac +# 'false' is the default value. Use Configure -Duseshrplib to override. cat UU/archname.cbu 'EOCBU' # This script UU/archname.cbu will get 'called-back' by Configure -- Andy Dougherty [EMAIL PROTECTED]
Re: [MacOS X] consider useshrplib='false' by default
In article [EMAIL PROTECTED], [EMAIL PROTECTED] (Dan Kogai) wrote: That's a great work but with all due respect I do not think making XS prebindable a good idea. Correct me if I am wrong but my understanding on prebinding is that it is a technique that makes dynamic libraries behave like static ones by predetermining ALL ADDRESSES IN NEED A PRIORI. Yes, I am not convinced prebinding is the best thing for XS. However, there may be other benefits to using -dynamiclib, and someone may have a good reason to use prebinding in the future, so I figure ... might as well include the option in DynaLoader, if it doesn't hurt anything, since DynaLoader is core and it is hard to add it later. :-) -- Chris Nandor [EMAIL PROTECTED]http://pudge.net/ Open Source Development Network[EMAIL PROTECTED] http://osdn.com/
Re: [MacOS X] consider useshrplib='false' by default
On Wednesday, June 4, 2003, at 03:33 PM, Jarkko Hietaniemi wrote:
Sounds reasonable to make the useshrplib to default to false (because
of the significant startup slowness otherwise) and at the very least
make it conditional (and I got a nod from Ed Moy of Apple, too).
So I did (change #19681).
Thank you.
How about the -prebind flags et alia? Should they be made default,
too?
I think we should but the biggest problem on slow startup was primarily
libperl.dylib (too may symbols for fix_prebinding daemon to tweak). I
am now working on benchmarking the difference but even without -prebind
flag the launch speed increase was significant.
One more caution if we want to make -prebind a default is here.
hints/darwin.sh
122:case $osvers in
123:1.[0-3].*)
124: lddlflags=${ldflags} -bundle -undefined suppress
125: ;;
126:1.*)
127: ldflags=${ldflags} -flat_namespace
128: lddlflags=${ldflags} -bundle -undefined suppress
129: ;;
130:[2-6].*)
131: ldflags=${ldflags} -flat_namespace
132: lddlflags=${ldflags} -bundle -undefined suppress
133: ;;
134:*) lddlflags=${ldflags} -bundle -undefined dynamic_lookup
135: case $ld in
136: *MACOSX_DEVELOPMENT_TARGET*) ;;
137: *) ld=MACOSX_DEPLOYMENT_TARGET=10.3 ${ld} ;;
138: esac
139: ;;
140:esac
As you see ${ldflags} is injected into $lddlflags but in case of
-prebind we need to avoid that. $ldflags is for perl linking while
$lddlflags is for XS. Since -prebind and -bundle are mutually
exclusive, we do not want -prebind in $lddlflags (though CC on darwin
is smart enough to ignore -prebind when -bundle, it still issues
warnings and I don't want to see that warning for each XS gets built).
Dan the Prebound Man
Re: [MacOS X] consider useshrplib='false' by default
Sounds reasonable to make the useshrplib to default to false (because of the significant startup slowness otherwise) and at the very least make it conditional (and I got a nod from Ed Moy of Apple, too). So I did (change #19681). How about the -prebind flags et alia? Should they be made default, too? -- Jarkko Hietaniemi [EMAIL PROTECTED] http://www.iki.fi/jhi/ There is this special biologist word we use for 'stable'. It is 'dead'. -- Jack Cohen
Re: [MacOS X] consider useshrplib='false' by default
On Wednesday, June 4, 2003, at 04:01 PM, Dan Kogai wrote:
I think we should but the biggest problem on slow startup was
primarily libperl.dylib (too may symbols for fix_prebinding daemon to
tweak). I am now working on benchmarking the difference but even
without -prebind flag the launch speed increase was significant.
And here is the benchmark.
with the new default as of 19681
% otool -hv ./perl
./perl:
Mach header
magic cputype cpusubtype filetype ncmds sizeofcmds flags
MH_MAGIC PPCALLEXECUTE 9 1604 NOUNDEFS
DYLDLINK
% redo_prebinding ./perl
redo_prebinding: file is not prebound: ./perl
% time /usr/bin/perl -le 'for (1..$ARGV[0]){system $ARGV[1], qw(-e
1)}' 1000 ./perl
2.860u 6.680s 0:12.75 74.8% 0+0k 0+4io 0pf+0w
with -Dldflags='-prebind' -Dlddflags='-bundle -undefined dynamic_lookup'
% otool -hv ./perl./perl:Mach header
magic cputype cpusubtype filetype ncmds sizeofcmds flags
MH_MAGIC PPCALLEXECUTE12 1884 NOUNDEFS
DYLDLINK PREBOUND TWOLEVEL
% redo_prebinding ./perl
% time /usr/bin/perl -le 'for (1..$ARGV[0]){system $ARGV[1], qw(-e
1)}' 1000 ./perl
2.290u 6.490s 0:11.59 75.7% 0+0k 0+3io 0pf+0w
As you see -prebind does help though not as significant as -Uuseshrplib.
man 8 fix_prebinding
hese hints are used by the dynamic loader to launch processes prebound,
eliminating the need to look up dynamically linked symbols. If
these
hints are out-of-date, the dynamic linker instead must lookup
referenced
symbols. As a result, the application may launch from 10-30%
slower.
So this still holds true at least for user time. Though the time
saving might not be significant for ordinary users, Busy web sites
with lots of CGIs should definitely benefit.
Dan the Benchmarker of Yours
P.S. Here is the comparison against our competitors and the script I
used to benchmark. see we compare by cusr and csys, not usr and sys.
Anything other than perl are pre-installed version.
#
use strict;
use Benchmark qw/:all/;
my $null = '/dev/null';
my $count = shift;
my %benches = map { $_ = eval qq/sub{ system '$_'='$null' }/ } @ARGV;
my $results = timethese($count = \%benches, 'all');
# Since we are benchmarkning system, we swap (usr,sys) and (cusr,csys)
# to get a correct values for cmpthese or the value gets bogus
# 0 1 23 4 5
# ($real, $user, $system, $children_user, $children_system, $iters)
for my $k (keys %$results){
my $v = $results-{$k};
@$v[1,2,3,4] = @$v[3,4,1,2]
}
cmpthese($results, 'all');
__END__
Ratepython ruby tclsh./perlsh
perl5.6.0
python16.7/s-- -29% -77% -82% -85%
-87%
ruby 23.6/s 41%-- -67% -74% -79%
-81%
tclsh 71.4/s 327% 202%-- -23% -38%
-43%
./perl92.6/s 454% 292% 30%-- -19%
-26%
sh 115/s 587% 386% 61% 24%--
-8%
perl5.6.0 125/s 647% 429% 75% 35%9%
--
We are doing great! Note ./perl is compiled with full of options like
-DDEBUGGING and -Dusethreads.
P^2.S. jhi, is there an option to make (time|cmp)these to compare by
cusr and csys so I don't have to tweak like the script above?
Re: [MacOS X] consider useshrplib='false' by default
In article [EMAIL PROTECTED],
[EMAIL PROTECTED] (Dan Kogai) wrote:
As you see ${ldflags} is injected into $lddlflags but in case of
-prebind we need to avoid that. $ldflags is for perl linking while
$lddlflags is for XS. Since -prebind and -bundle are mutually
exclusive, we do not want -prebind in $lddlflags (though CC on darwin
is smart enough to ignore -prebind when -bundle, it still issues
warnings and I don't want to see that warning for each XS gets built).
Another question is whether to use something other than bundle for XS, to
allow prebinding. I have a patch to dl_dyld.xs that seems to work; it
allows both bundle and dyld (thanks to some pointers from wsanchez).
--- dl_dyld.xs.orig Wed Jun 4 06:57:32 2003
+++ dl_dyld.xs.new Wed Jun 4 06:57:21 2003
@@ -109,14 +109,22 @@
NSModule handle = NULL;
dyld_result = NSCreateObjectFileImageFromFile(path, ofile);
-if (dyld_result != NSObjectFileImageSuccess)
- TranslateError(path, OFImage, dyld_result);
-else
+if (dyld_result == NSObjectFileImageSuccess)
{
// NSLinkModule will cause the run to abort on any link error's
// not very friendly but the error recovery functionality is limited.
handle = NSLinkModule(ofile, path, TRUE);
NSDestroyObjectFileImage(ofile);
+}
+else if ((dyld_result == NSObjectFileImageFormat ||
+ dyld_result == NSObjectFileImageInappropriateFile)
+ NSAddLibrary(path) == TRUE)
+{
+ handle = (NSModule)(void *)-1;
+}
+else
+{
+ TranslateError(path, OFImage, dyld_result);
}
return handle;
Then, instead of using -bundle, I used -dynamiclib. However, when I tried
to add -prebind I got a bunch of errors, and then I ran out of time to mess
with it. :-) But perhaps worth:
1. doublechecking my work :)
2. including it in perl's DynaLoader as an option for XS developers, even if
-bundle is still the default
--
Chris Nandor [EMAIL PROTECTED]http://pudge.net/
Open Source Development Network[EMAIL PROTECTED] http://osdn.com/
Re: [MacOS X] consider useshrplib='false' by default
On Wednesday, June 4, 2003, at 08:03 PM, Chris Nandor wrote:
In article [EMAIL PROTECTED],
[EMAIL PROTECTED] (Dan Kogai) wrote:
As you see ${ldflags} is injected into $lddlflags but in case of
-prebind we need to avoid that. $ldflags is for perl linking while
$lddlflags is for XS. Since -prebind and -bundle are mutually
exclusive, we do not want -prebind in $lddlflags (though CC on darwin
is smart enough to ignore -prebind when -bundle, it still issues
warnings and I don't want to see that warning for each XS gets built).
Another question is whether to use something other than bundle for XS,
to
allow prebinding. I have a patch to dl_dyld.xs that seems to work; it
allows both bundle and dyld (thanks to some pointers from wsanchez).
That's a great work but with all due respect I do not think making XS
prebindable a good idea. Correct me if I am wrong but my understanding
on prebinding is that it is a technique that makes dynamic libraries
behave like static ones by predetermining ALL ADDRESSES IN NEED A
PRIORI.
With -Uuseshrplib, the only dynamic library you need is libSystem
(which is what libc and libm really are). So it is safe to assume that
no addresses would collide.
% otool -L ./perl
./perl:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0,
current version 63.0.0)
But to make XS prebindable you have to make sure that addresses never
collide FOR ALL MODULES. When that fails XS may still work (so far as
I see dyld is smart enough to make sure to to collide symbols and
addresses and diverts that when it needs but when that happens a
warning is issued).
Since no one knows how many XSes a particular user needs, I think we
should leave -bundle for XS.
Theoretically, however, we can still resolve this by keeping a symbol
table handy somewhere and use that during XS builds but that I consider
a too much work.
Dan the Prebound Man
[MacOS X] consider useshrplib='false' by default
Porters,
I would like to propose that we make useshrplib='false' default for
darwin to resolve prebinding woes.
I said update_prebinding -root / should resolve the problem but only
temporarily. perl is in fact not prebind. This is what is actually
happning.
% env DYLD_PREBIND_DEBUG=1 /usr/local/bin/perl -e 1
dyld: perl: prebinding disabled because library:
/usr/local/lib/perl5/5.8.0/darwin-thread-multi-64int/CORE/
libperl.dylib got slid
dyld: in notify_prebinding_agent() determined the system shared
regions are NOT used
dyld: 2 two-level prebound libraries used out of 3
At this stage no /usr/libexec/fix_prebinding: /usr/local/bin/perl
couldn't be prebound in the past, and probably can't be prebound now.
is issued. But as time goes by and more system gets used, dyld: 2
two-level prebound libraries used out of 3 stops working and the
(in)?famous fix_prebinding warning starts appearing in your
/var/log/system.log.
When you attempt to compile perl with -prebind switch, however, this is
what happens
DYLD_LIBRARY_PATH=/Users/dankogai/work/perl-5.8.0 cc -prebind -o
miniperl \
miniperlmain.o opmini.o libperl.dylib -lm -lc
cc: unrecognized option `-prebind_allow_overlap'
ld: warning multiple definitions of symbol _Perl_listkids
opmini.o definition of _Perl_listkids in section (__TEXT,__text)
[lots of similar ld warnings to follow]
So it does not work in practice. But when you turn off useshrplib,
prebinding works beautifully.
To test that for yourself, just Configure with
-Dldflags='-prebind' -Dlddflags='-bundle -undefined dynamic_lookup'
-Uuseshrplib
And the change? Significant! See this
Reference: bundled version
% time /usr/bin/perl -le 'for (1..$ARGV[0]){system $ARGV[1], qw(-e
1)}' 1000 /usr/bin/perl5.6.0
1.560u 4.680s 0:07.46 83.6% 0+0k 0+0io 0pf+0w
-Duseshrplib
% time /usr/bin/perl -le 'for (1..$ARGV[0]){system $ARGV[1], qw(-e
1)}' 1000 /usr/local/bin/perl
21.490u 8.690s 0:33.22 90.8%0+0k 0+1io 0pf+0w
-Uuseshrplib and prebinding
%time /usr/bin/perl -le 'for (1..$ARGV[0]){system $ARGV[1], qw(-e 1)}'
1000 /usr/local/bin/perl
2.190u 5.810s 0:10.42 76.7% 0+0k 0+0io 0pf+0w
As we all know many systems now support -Duseshrplib but not many make
it a default.
% egrep useshrplib='?true'? hints/*.sh
hints/bsdos.sh:useshrplib='true'
hints/darwin.sh:useshrplib='true';
hints/openbsd.sh: useshrplib=true
hints/os2.sh:useshrplib='true'
hints/os390.sh:'') useshrplib='true' ;;
hints/rhapsody.sh:useshrplib='true';
hints/svr5.sh: useshrplib='true'
and those listed usually make useshrplib=true conditionally. It seems
only darwin makes it unconditionally true.
shared libperl save disk space and memory ONLY WHEN you have many apps
liked to libperl. In most cases, however, you have only perl itself
and mod_perl. So I suggest that we make useshrplib=false for darwin.
I know you all know but XS are still dynamically loaded. So there is
not much point for -D useshrplib.
Dan the Man too Tired of Unprebindable Camels
