Re: libc size

2002-11-07 Thread Andre Albsmeier
On Tue, 05-Nov-2002 at 14:22:41 -0800, Tim Kientzle wrote:
 Andre Albsmeier wrote:
 
  I would like to compile the whole base system (maybe even
  the ports) without the whole setlocale stuff. Do you have
  any ideas of how to do this easily?
 
 
 Replace setlocale() in lib/libc/locale with a stub. That
 should do it.
 
 Of course, this doesn't really save you that much.
 Most binaries are dynamically linked, so this
 saves nothing on disk space for those executables.
 It's a couple of K for those statically-linked
 executables that use it, but there aren't that

Yeah, you are right. I had a look at it and its size nearly
doesn't count.

 many of them.  If you're worried about disk space
 for the locale data itself, you can
 simply delete any locales you don't use from
 /usr/share/locale

These are 548 files consuming 606 kB. Regarding the space it
is not so much as well... BTW, I deleted the stuff in
/usr/X11R6/lib/X11/locale/* one day and my X didn't start
anymore. But for the stuff in the base system it might work...

Thanks,

-Andre

 
 It comes up in the context of 'cat' only
 because it more than doubles the size of
 an otherwise very small executable for a
 single option that is not standard and
 (probably) not ever used.  This is a pretty
 unusual situation.
 
 Tim Kientzle

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-11-06 Thread M. Warner Losh
In message: [EMAIL PROTECTED]
Tim Kientzle [EMAIL PROTECTED] writes:
: Several people have pointed out that FreeBSD has
: certain protections against LD_LIBRARY_PATH exploits,
: but there are still real questions here.  (Kernel
: races, possibly?)  Privilege elevation is an
: interesting idea, but tricky to audit.

There are no known issues in this area, and haven't been for a couple
of years now.  While this isn't proof, it is a compelling argument.
This isn't a real question, to be honest.  We've had dynamically
linked setuid/setgid programs for years.  The only issues have been in
the setuid/setgid code itself, not the dynamic linker.  Bugs of this
nature haven't really been a problem.

Warner

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-11-05 Thread Terry Lambert
Max Khon wrote:
 hi, there!
 Ok, I put the patch and test program to
 http://people.freebsd.org/~fjoe/libdl.tar.bz2.
 
 Patches are made against RELENG_4 (and all tests were done on RELENG_4)
 but it will not be that hard to port everything to -CURRENT.
 This is just a proof-of-concept work-in-progress.
 
 The plan is to add this stuff (rtld sources with -DLIBDL) to libc.a
 so statically linked programe will have dlopen/dlsym etc.

It's interesting, except for being bz2'ed.  It's not like the
Unisys patent hasn't expired yet... 8-).

I have some similar patches; the main difference is that mine
are actually a standalone library.  It utilizes the .init
section to cause the initialization to take place normally,
as per a standard constructor.

Right now, my patches don't work with C++ because the replace
the .init section.  I have a modified version that actually
uses the constructor list in the standard .init section,
instead.

The main issue is the initialization of crt.crt_bp (normally,
this is done via _callmain).

The problem is that in calling __do_dynamic_link(), the argv
argument is NULL in the constructor case.  Basically, this
means that the constructor code has to be modified to pass it
in to each initially constructed object... the constructor
has to change from a void to a void *, which most constructors
ignore, for it to work.

This is also a requirement, if you want the loaded shared obects
that resolve external symbol references to symbols defined by
the binary (e.g. as if they were dlopen'ed by a dymanically
linked binary).


 Problems with current patches are:
 - I do not know what to do on alpha with _GOT_END_ and
 _GLOBAL_OFFSET_TABLE_ symbols. I had to use ld.so.script
 to solve missing _GOT_END_ problem and ifdef out _GLOBAL_OFFSET_TABLE_
 usage from alpha/reloc.c for second problem. An advice from alpha rtld
 guru will be very useful

That's not me.  8-(.  I think this can be done without that,
though, if you make the changes in the crt0 code, instead.


 - gdb support for shared objecject dlopened from statically linked
 program is broken

This is because of the argv, which is really the startup frame base
address, which contains that information; it's not being initialized
in the case where you're calling it.  You really need to modify the
crt0 code, and duplicate the dlopen mappings in a libdlopen, instead
of modifying ld.so.


 - rtld_exit() is not called on exit so fini functions are not
 called on exit

I had the same problem; mine came from stealing the section entry;
the way to deal with this is to map the base frame so you can add
an internal .fini traversal function to the atexit().  This has to
be done in a reversible way, so you can't actually use atexit(3)
itself to do the job.


 - probably more stuff could be #ifdef'ed out from rtld when it is compiled
 with -DLIBDL

I don't think so, actually.  The question is how the symbols get
in, in the first place.


 - xmalloc and friends names in rtld sources probably should be prepended
 with an underscore to prevent name clashes (if this stuff will be included
 in libc.a)

You can actually deal with this by forcing the load of libc.so, the
first time the dlopen() itself is called, if it's not loaded already,
so the symbols will be available.  It's ugly, but it works, in lieu
of linking all .so's that make libc calls against libc.so (like you'd
reasonably expect).

-- Terry

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-11-05 Thread Max Khon
hi, there!

On Tue, Nov 05, 2002 at 12:12:45AM -0800, Terry Lambert wrote:

  Ok, I put the patch and test program to
  http://people.freebsd.org/~fjoe/libdl.tar.bz2.
  
  Patches are made against RELENG_4 (and all tests were done on RELENG_4)
  but it will not be that hard to port everything to -CURRENT.
  This is just a proof-of-concept work-in-progress.
  
  The plan is to add this stuff (rtld sources with -DLIBDL) to libc.a
  so statically linked programe will have dlopen/dlsym etc.
 
 I have some similar patches; the main difference is that mine
 are actually a standalone library.  It utilizes the .init
 section to cause the initialization to take place normally,
 as per a standard constructor.
 
standalone library will be harder to use. if you will name it
libdl then all configure scripts will find it and will link
with it (even in dynamically linked case).
the only way I see is to include dlopen and friend to static
version of libc OR move dlopen and friends to libdl from rtld-elf.

 Right now, my patches don't work with C++ because the replace
 the .init section.  I have a modified version that actually
 uses the constructor list in the standard .init section,
 instead.
 
can you show me an example of non-working program in C++?

 The main issue is the initialization of crt.crt_bp (normally,
 this is done via _callmain).
 
 The problem is that in calling __do_dynamic_link(), the argv
 argument is NULL in the constructor case.  Basically, this
 means that the constructor code has to be modified to pass it
 in to each initially constructed object... the constructor
 has to change from a void to a void *, which most constructors
 ignore, for it to work.
 
 This is also a requirement, if you want the loaded shared obects
 that resolve external symbol references to symbols defined by
 the binary (e.g. as if they were dlopen'ed by a dymanically
 linked binary).

that is hardly needed for nsswitch. btw libdl in Linux also is not able
to resolve external symbols refs to symbols defined in main
(statically linked) object.

  - gdb support for shared objecject dlopened from statically linked
  program is broken
 
 This is because of the argv, which is really the startup frame base
 address, which contains that information; it's not being initialized
 in the case where you're calling it.  You really need to modify the
 crt0 code, and duplicate the dlopen mappings in a libdlopen, instead
 of modifying ld.so.
 
 
  - rtld_exit() is not called on exit so fini functions are not
  called on exit
 
 I had the same problem; mine came from stealing the section entry;
 the way to deal with this is to map the base frame so you can add
 an internal .fini traversal function to the atexit().  This has to
 be done in a reversible way, so you can't actually use atexit(3)
 itself to do the job.
 
I think I can call atexit(rtld_exit) in rtld_init()

  - probably more stuff could be #ifdef'ed out from rtld when it is compiled
  with -DLIBDL
 
 I don't think so, actually.  The question is how the symbols get
 in, in the first place.

this is not high-priority task

  - xmalloc and friends names in rtld sources probably should be prepended
  with an underscore to prevent name clashes (if this stuff will be included
  in libc.a)
 
 You can actually deal with this by forcing the load of libc.so, the
 first time the dlopen() itself is called, if it's not loaded already,
 so the symbols will be available.  It's ugly, but it works, in lieu
 of linking all .so's that make libc calls against libc.so (like you'd
 reasonably expect).

this is not needed. malloc and friends will be linked from libc.a
I am talking about xmalloc and firends which are used in rtld-elf internally.

/fjkoe


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-11-05 Thread Andre Albsmeier
On Thu, 31-Oct-2002 at 10:54:55 -0800, Tim Kientzle wrote:
 I agree with David Schultz that dynamically linking
 /bin and /sbin is playing with fire.  I, too, have had
 ugly experiences on systems that did this:
 When /usr won't mount, it is not pleasant to be
 stuck with no tools.  (Consider a network environment
 where /usr is NFS-mounted as an extreme example.)
 
 As for the disk space concerns, I spent a couple of
 
 hours with some of the smaller utilities.  Identical
 functionality to the originals, still statically linked.
 Compare with 'ls -l /bin' on your system:
* echo: 3k
* sleep: 3k
* sync: 3k
* cat: 40k with setlocale, 12k without
  (cat uses setlocale for non-standard -v option)

This is something that comes up in my mind every now and then:
I would like to compile the whole base system (maybe even
the ports) without the whole setlocale stuff. Do you have
any ideas of how to do this easily? We have so many NO_*
knobs in make.conf but the NO_LOCALE one I am really
missing.

Well, for the ports it is not so difficult: I have added a

CONFIGURE_ARGS+=--disable-nls

line to Mk/bsd.port.mk for years now and it works in most
cases.

Thanks,

-Andre

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-11-05 Thread Jake Burkholder
Apparently, On Tue, Nov 05, 2002 at 01:21:42PM +0600,
Max Khon said words to the effect of;

 hi, there!
 
 On Tue, Nov 05, 2002 at 02:18:23AM -0500, Jake Burkholder wrote:
 
Before someone says you can dlopen() from static binaries in order to
implement nsswitch, please provide the patch proving it.  Our best
FreeBSD minds don't think it can be done properly and sanely.
   
   I have the patch. Currently it is made against RELENG_4 and I have a couple
   of questions about alpha (however it works on alpha too with a few hacks).
   Unfortunately, jdp does not have enough time to review it and I have
   lack of time to port it to -current (that would not be that hard but
   since sparc64 is now Tier-1 platform the patch should be ported to
   sparc64 too but I do not have sparc64 hardware and access to
   panther is very slow from my home).
   
   What is the right place to post the patch and test program
   demonstrating dlopen for statically linked programs?
  
  Put it up somehere on the web or email it to the list.  I'd
  be interested in looking at it.
 
 Ok, I put the patch and test program to
 http://people.freebsd.org/~fjoe/libdl.tar.bz2.
 
 Patches are made against RELENG_4 (and all tests were done on RELENG_4)
 but it will not be that hard to port everything to -CURRENT.
 This is just a proof-of-concept work-in-progress.
 
 The plan is to add this stuff (rtld sources with -DLIBDL) to libc.a
 so statically linked programe will have dlopen/dlsym etc.
 
 Problems with current patches are:
 - I do not know what to do on alpha with _GOT_END_ and
 _GLOBAL_OFFSET_TABLE_ symbols. I had to use ld.so.script
 to solve missing _GOT_END_ problem and ifdef out _GLOBAL_OFFSET_TABLE_
 usage from alpha/reloc.c for second problem. An advice from alpha rtld
 guru will be very useful
 - gdb support for shared objects dlopened from statically linked
 program is broken
 - rtld_exit() is not called on exit so fini functions are not
 called on exit
 - probably more stuff could be #ifdef'ed out from rtld when it is compiled
 with -DLIBDL
 - xmalloc and friends names in rtld sources probably should be prepended
 with an underscore to prevent name clashes (if this stuff will be included
 in libc.a)
 
 Any comments, suggestions, patches will be very appreciated.

I think there are more problems than you realize.  They are very hard
to fix.

You've basically hacked rtld to bits.  All the ifdefs make it hard
to read and maintain.

This statically links rtld into any static binary that wants to use
dlopen.  What was that about saving space on the root partition?

-rwxr-xr-x   1 jake  wheel  143177 Nov  5 11:57 hello

This is more than twice as big as a normal static binary which just
calls printf on my system.  ~90K bloat just for dlopen.

I don't see that you've dealt with getting the linker to generate
the tables that rtld needs; an _DYNAMIC section, a dynsym table,
a dynstr table etc.  These are needed in order to look up symbols in
the statically linked binary itself.  Getting the linker to do this is
not overly difficult, we do it for the kernel, but it bloats the static
binaries more.  It also creates a special case in the makefiles, unless
we do this for all static binaries, which would cause a lot of bloat.

As a result of the above, how do you deal with multiple implementations
of library services being present?  For example a statically linked
binary calls malloc, so it has a copy of malloc linked into it.  It tries
to dlopen a library which also calls malloc.  Which copy of malloc does
the library use?  How does it locate the malloc that's linked into the
static binary without the dynamic tables?  What happens when the application
tries to free a pointer allocated by the library, or vice versa?

When David said we didn't think it could be done properly or sanely, we
meant it.  It must work exactly like it would in a dynamic binary.

Jake

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-11-05 Thread Max Khon
hi, there!

On Tue, Nov 05, 2002 at 12:08:33PM -0500, Jake Burkholder wrote:

  The plan is to add this stuff (rtld sources with -DLIBDL) to libc.a
  so statically linked programe will have dlopen/dlsym etc.
  
  Problems with current patches are:
  - I do not know what to do on alpha with _GOT_END_ and
  _GLOBAL_OFFSET_TABLE_ symbols. I had to use ld.so.script
  to solve missing _GOT_END_ problem and ifdef out _GLOBAL_OFFSET_TABLE_
  usage from alpha/reloc.c for second problem. An advice from alpha rtld
  guru will be very useful
  - gdb support for shared objects dlopened from statically linked
  program is broken
  - rtld_exit() is not called on exit so fini functions are not
  called on exit
  - probably more stuff could be #ifdef'ed out from rtld when it is compiled
  with -DLIBDL
  - xmalloc and friends names in rtld sources probably should be prepended
  with an underscore to prevent name clashes (if this stuff will be included
  in libc.a)
  
  Any comments, suggestions, patches will be very appreciated.
 
 I think there are more problems than you realize.  They are very hard
 to fix.
 
 You've basically hacked rtld to bits.  All the ifdefs make it hard
 to read and maintain.

There number of #ifdef's is not large (for me) to make rtld unmaintainable.
If this is inappropriate for you there are two obvious ways to solve it:
- refactor rtld-elf and move common parts of libdl (or whatever) and
rtld-elf to separate files
- unifdef rtld-elf and put libdl sources separately

The reasons I implemented this as a patch for rtld-elf are:
- I did not want to create two almost identical pieces of code
- I wanted to make these patches as obvious as they can be.
This is just a proof-of-concept work.

 This statically links rtld into any static binary that wants to use
 dlopen.  What was that about saving space on the root partition?

I didn't tell you or anyone else that this work is done towards
saving space in /. The question was about nsswitch and (in particular)
dlopen in statically linked programs.
 
The only way to save space in / and to be able to use nsswitch is
make everyhting shared exactly like NetBSD did a few weeks ago.
I saw a number of complaints about loosing an ability to repair system
if something goes wrong. I guess that people just haven't looked at what
NetBSD folks have done. Yes, they have a number of statically linked
programs in /rescue. Yes, they have created /lib.

 -rwxr-xr-x   1 jake  wheel  143177 Nov  5 11:57 hello
 
 This is more than twice as big as a normal static binary which just
 calls printf on my system.  ~90K bloat just for dlopen.

This is the price you pay for dlopen. This is how things are
in other systems that are capable of using dlopen in statically
linked executables.

 I don't see that you've dealt with getting the linker to generate
 the tables that rtld needs; an _DYNAMIC section, a dynsym table,
 a dynstr table etc.  These are needed in order to look up symbols in
 the statically linked binary itself.  Getting the linker to do this is
 not overly difficult, we do it for the kernel, but it bloats the static
 binaries more.

I do not have Solaris at hand but Linux is not capable of resolving
symbols refs in shared objects to main binary.

This is not even possible at all. At a time of static
linking ld can not know which symbols will supposedly
loaded shared library need. Will it need fopen()? Will it need
fts_open()?

In Linux libdl implementation you can't also use dlopen(NULL,...)
in main program.

 It also creates a special case in the makefiles, unless
 we do this for all static binaries, which would cause a lot of bloat.

Are you talking about STATICOBJS and SHOBJS? This is how libpam is built
right now. You have different sets object files in shared and static
versions of libpam. Please take a look at src/lib/libpam/libpam/Makefile
and corresponding /usr/share/mk bits.

 As a result of the above, how do you deal with multiple implementations
 of library services being present?  For example a statically linked
 binary calls malloc, so it has a copy of malloc linked into it.  It tries
 to dlopen a library which also calls malloc.  Which copy of malloc does
 the library use?  How does it locate the malloc that's linked into the
 static binary without the dynamic tables?

A part of answer is above (about ld not knowing which symbols
shared object will want). The real solution is to link
shared libraries with all objects they will need (including libc).
This is how things are implemented in other systems.
As a result library will use malloc from libc that will be
loaded as a dependency (and this is demonstrated in the example I posted).

 What happens when the application
 tries to free a pointer allocated by the library, or vice versa?

This is not possible for obvious reasons.
On Linux you will get SEGFAULT (for the same obvious reasons).
Yes, this is a limitation.

 When David said we didn't think it could be done properly or sanely, we
 meant it.  It must 

Re: libc size

2002-11-05 Thread Max Khon
hi, there!

On Wed, Nov 06, 2002 at 12:20:50AM +0600, Max Khon wrote:

 The only way to save space in / and to be able to use nsswitch is
 make everyhting shared exactly like NetBSD did a few weeks ago.
 I saw a number of complaints about loosing an ability to repair system
 if something goes wrong. I guess that people just haven't looked at what
 NetBSD folks have done. Yes, they have a number of statically linked
 programs in /rescue. Yes, they have created /lib.

I was not 100% correct there. I know about a idea to use
a daemon (for NSS) that listens on unix domain socket but that would not save
space either because you had to have fallbacks in libc anyway
(for case when this daemon is not running).

  -rwxr-xr-x   1 jake  wheel  143177 Nov  5 11:57 hello
  
  This is more than twice as big as a normal static binary which just
  calls printf on my system.  ~90K bloat just for dlopen.
 
 This is the price you pay for dlopen. This is how things are
 in other systems that are capable of using dlopen in statically
 linked executables.
 
  I don't see that you've dealt with getting the linker to generate
  the tables that rtld needs; an _DYNAMIC section, a dynsym table,
  a dynstr table etc.  These are needed in order to look up symbols in
  the statically linked binary itself.  Getting the linker to do this is
  not overly difficult, we do it for the kernel, but it bloats the static
  binaries more.
 
 I do not have Solaris at hand but Linux is not capable of resolving
 symbols refs in shared objects to main binary.

I need to add (aside from what I told in previous letter) that requirement
for libdl to build _DYNAMIC section means changes to binutils
(because there is no information in statically linked executable
which libdl can use to build such section).
I think this is not the game anyone would like to play.

By the way, space bloat is not that large:

fjoe@husky:~$ls /bin /sbin | wc -l
 132
fjoe@husky:~$

Some of that programs are hard-linked and some of them do not use
getXXXbyYYY functions (or call other functions that use getXXXbyYYY, like
user_from_uid). The bloat will be about 10M in worst case
and this is quite appropriate before we switch to shared /

Of course, there should be NO_NSS knob for libc for embedded environments.

I do not consider my work as something that FreeBSD badly needs
and do not insist on getting it right now as is (yes, it needs some polishing
and maybe not everything works as expected, yes there are some obvious
limitations in using dlopen from statically linked programs),
but I think it removes some of limitations we currently have.
Yet another example of stuff that (I think) we need to have in libc and
which uses dlopen is Citrus Project (I18N Framework).

/fjoe


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-11-05 Thread Jake Burkholder
  You've basically hacked rtld to bits.  All the ifdefs make it hard
  to read and maintain.
 
 There number of #ifdef's is not large (for me) to make rtld unmaintainable.
 If this is inappropriate for you there are two obvious ways to solve it:
 - refactor rtld-elf and move common parts of libdl (or whatever) and
 rtld-elf to separate files
 - unifdef rtld-elf and put libdl sources separately
 
 The reasons I implemented this as a patch for rtld-elf are:
 - I did not want to create two almost identical pieces of code
 - I wanted to make these patches as obvious as they can be.
 This is just a proof-of-concept work.

Understood, but I think that either solution is not great.  We either
wedge libdl into rtld or we have 2 copies of large amounts of rtld.

 
  This statically links rtld into any static binary that wants to use
  dlopen.  What was that about saving space on the root partition?
 
 I didn't tell you or anyone else that this work is done towards
 saving space in /. The question was about nsswitch and (in particular)
 dlopen in statically linked programs.

Ok.

  It also creates a special case in the makefiles, unless
  we do this for all static binaries, which would cause a lot of bloat.
 
 Are you talking about STATICOBJS and SHOBJS? This is how libpam is built
 right now. You have different sets object files in shared and static
 versions of libpam. Please take a look at src/lib/libpam/libpam/Makefile
 and corresponding /usr/share/mk bits.

What I was referring to is a trick to force the linker to generate
a dynamic binary with all the usual elf dynamic tables, but which is
actually statically linked.

eg:

 cat test.c
int
main(void)
{
printf(hello world\n);
}
 cc -static -Wl,-r -o test test.c
 touch hack.c
 cc -shared -o hack.So hack.c
 ld -o test1 test /home/jake/hack.So 
 ./test1
hello world
 

Conceivably this would allow dlopen to work on the main program, and is
what we do to allow the kernel to link klds against itself.  But you
also need to do something about the .interp section that gets put in,
and the .dynamic, .dynsym and .dynstr sections aren't free.

 
  As a result of the above, how do you deal with multiple implementations
  of library services being present?  For example a statically linked
  binary calls malloc, so it has a copy of malloc linked into it.  It tries
  to dlopen a library which also calls malloc.  Which copy of malloc does
  the library use?  How does it locate the malloc that's linked into the
  static binary without the dynamic tables?
 
 A part of answer is above (about ld not knowing which symbols
 shared object will want). The real solution is to link
 shared libraries with all objects they will need (including libc).
 This is how things are implemented in other systems.
 As a result library will use malloc from libc that will be
 loaded as a dependency (and this is demonstrated in the example I posted).
 
  What happens when the application
  tries to free a pointer allocated by the library, or vice versa?
 
 This is not possible for obvious reasons.
 On Linux you will get SEGFAULT (for the same obvious reasons).
 Yes, this is a limitation.
 
  When David said we didn't think it could be done properly or sanely, we
  meant it.  It must work exactly like it would in a dynamic binary.
 
 You can't get exact behaviour. But this behaviour is sufficient
 for PAM and NSS. Maybe there are some other (less important) uses for
 this stuff.

What I'm suggesting is that for libdl to be a viable alternative to
making the binaries in / dynamic so that pam works, it has to work
better.  Preferably as good as in dynamic binaries, and without the
foot shooting potential that having 2 copies of malloc or other things
loaded entail.  Yes, a simple implementation may work for PAM, but
using full blown dynamic linking is architecturally much cleaner and
I think that this outweighs the disadvantages.

 
 Sorry for being too emotional, but none of your questions make sense.
 I highly appreciate your work on sparc64 but seems that
 you were in bad mood when typing an answer.

Very bad, sorry. :)

Jake

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-11-05 Thread Max Khon
hi, there!

On Tue, Nov 05, 2002 at 03:27:57PM -0500, Jake Burkholder wrote:

  Are you talking about STATICOBJS and SHOBJS? This is how libpam is built
  right now. You have different sets object files in shared and static
  versions of libpam. Please take a look at src/lib/libpam/libpam/Makefile
  and corresponding /usr/share/mk bits.
 
 What I was referring to is a trick to force the linker to generate
 a dynamic binary with all the usual elf dynamic tables, but which is
 actually statically linked.
 
 eg:
 
  cat test.c
 int
 main(void)
 {
 printf(hello world\n);
 }
  cc -static -Wl,-r -o test test.c
  touch hack.c
  cc -shared -o hack.So hack.c
  ld -o test1 test /home/jake/hack.So 
  ./test1
 hello world
  
 
 Conceivably this would allow dlopen to work on the main program, and is
 what we do to allow the kernel to link klds against itself.  But you
 also need to do something about the .interp section that gets put in,
 and the .dynamic, .dynsym and .dynstr sections aren't free.
 
Got it. But this will not solve the problem. All the symbols
in statically linked executable are resolved at linking stage
and there is no information in it that libdl can use in find_symdef()
(if we strip the executable. Do we want unstripped executables?).
Patching binutils is not easy and I doubt that someone would like to do this.

/fjoe


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-11-05 Thread Tim Kientzle
Andre Albsmeier wrote:


I would like to compile the whole base system (maybe even
the ports) without the whole setlocale stuff. Do you have
any ideas of how to do this easily?



Replace setlocale() in lib/libc/locale with a stub. That
should do it.

Of course, this doesn't really save you that much.
Most binaries are dynamically linked, so this
saves nothing on disk space for those executables.
It's a couple of K for those statically-linked
executables that use it, but there aren't that
many of them.  If you're worried about disk space
for the locale data itself, you can
simply delete any locales you don't use from
/usr/share/locale

It comes up in the context of 'cat' only
because it more than doubles the size of
an otherwise very small executable for a
single option that is not standard and
(probably) not ever used.  This is a pretty
unusual situation.

Tim Kientzle


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-11-04 Thread Tim Kientzle
Miguel Mendez wrote:


Tim Kientzle [EMAIL PROTECTED] wrote:



1) Fragility.  Could a naive sysadmin (or a dying
   disk) break /[s]bin?
   What if the ldconfig hints files were hosed?
   Is ld-elf.so truly bulletproof?


Agreed, and, fortunately, that was taken into account with the
introduction of the /rescue dir:

christine: {48} du -h /rescue
2.4M/rescue



Oh.  So the real size of NetBSD's /bin and /sbin includes
another 2.4M for /rescue.  That makes it less
impressive.  I don't find the duplication appealing, either.
(Why not just put the /rescue versions directly
into /bin and /sbin?  That would be smaller still,
wouldn't it?)



2) Security.  Can LD_LIBRARY_PATH (or other mechanisms)
   be used to deliberately subvert any of these programs?
   (especially the handful of suid/sgid programs here)



Several people have pointed out that FreeBSD has
certain protections against LD_LIBRARY_PATH exploits,
but there are still real questions here.  (Kernel
races, possibly?)  Privilege elevation is an
interesting idea, but tricky to audit.



the results from ls -l /bin on your NetBSD system



christine: {66} ls -l /bin
-r-xr-xr-x  1 root  wheel8480 Oct 29 22:59 cat



-r-xr-xr-x  1 root  wheel4892 Oct 29 23:00 echo


 -r-xr-xr-x  1 root  wheel5568 Oct 29 23:01 rmdir


-r-xr-xr-x  1 root  wheel5892 Oct 29 23:02 sleep
-r-xr-xr-x  1 root  wheel4652 Oct 29 23:02 sync


 [[ others omitted ]]



sigh  I've been looking at some of the FreeBSD standard utils,
and with a very little bit of work got this:

-rwxr-xr-x  1 tim  tim  9552 Nov  4 11:10 cat
-rwxr-xr-x  1 tim  tim  2776 Nov  4 11:10 echo
-rwxr-xr-x  1 tim  tim  3288 Nov  1 13:48 rmdir
-rwxr-xr-x  1 tim  tim  2904 Nov  4 11:10 sleep
-rwxr-xr-x  1 tim  tim  2424 Nov  4 11:10 sync

All statically linked, all portable C, with identical
functionality to the originals.  If statically-linked
versions can be 1/2 the size of the dynamic versions,
then I _really_ don't see the advantage of dynamic linking.
Perhaps some more careful programming is all that's needed?  ;-)
(Admittedly, a space-conscious overhaul of sh, csh, or ed
is not entirely trivial; but most of /bin and /sbin is pretty simple
to prune down.)



rcNG has been in work for a long time. Is it worth it? Absolutely,
try it once and you'll wonder how you could live with the old system, or
even with the sysV symlink crazyness.



As it happens, I've been looking closely at RCng
just recently.  Though I really like the core design, I do
have some quibbles with the implementation.  It
is usable today, and does address the worst problems
of SysV-style init.  Still needs some work, though.  ;-)

Tim Kientzle


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-11-04 Thread David O'Brien
On Mon, Nov 04, 2002 at 11:32:38AM -0800, Tim Kientzle wrote:
 Oh.  So the real size of NetBSD's /bin and /sbin includes
 another 2.4M for /rescue.  That makes it less
 impressive.  I don't find the duplication appealing, either.
 (Why not just put the /rescue versions directly
 into /bin and /sbin?  That would be smaller still,

Because that would nullify one of the big reasons for making /bin and
/sbin shared -- so one can dlopen(3).  We can't, for instance, get a
proper nsswitch implementation until we make /bin and /sbin dynamic.

Before someone says you can dlopen() from static binaries in order to
implement nsswitch, please provide the patch proving it.  Our best
FreeBSD minds don't think it can be done properly and sanely.

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-11-04 Thread Max Khon
hi, there!

On Mon, Nov 04, 2002 at 01:57:35PM -0800, David O'Brien wrote:

  another 2.4M for /rescue.  That makes it less
  impressive.  I don't find the duplication appealing, either.
  (Why not just put the /rescue versions directly
  into /bin and /sbin?  That would be smaller still,
 
 Because that would nullify one of the big reasons for making /bin and
 /sbin shared -- so one can dlopen(3).  We can't, for instance, get a
 proper nsswitch implementation until we make /bin and /sbin dynamic.
 
 Before someone says you can dlopen() from static binaries in order to
 implement nsswitch, please provide the patch proving it.  Our best
 FreeBSD minds don't think it can be done properly and sanely.

I have the patch. Currently it is made against RELENG_4 and I have a couple
of questions about alpha (however it works on alpha too with a few hacks).
Unfortunately, jdp does not have enough time to review it and I have
lack of time to port it to -current (that would not be that hard but
since sparc64 is now Tier-1 platform the patch should be ported to
sparc64 too but I do not have sparc64 hardware and access to
panther is very slow from my home).

What is the right place to post the patch and test program
demonstrating dlopen for statically linked programs?

/fjoe


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-11-04 Thread Jake Burkholder
Apparently, On Tue, Nov 05, 2002 at 12:54:54PM +0600,
Max Khon said words to the effect of;

 hi, there!
 
 On Mon, Nov 04, 2002 at 01:57:35PM -0800, David O'Brien wrote:
 
   another 2.4M for /rescue.  That makes it less
   impressive.  I don't find the duplication appealing, either.
   (Why not just put the /rescue versions directly
   into /bin and /sbin?  That would be smaller still,
  
  Because that would nullify one of the big reasons for making /bin and
  /sbin shared -- so one can dlopen(3).  We can't, for instance, get a
  proper nsswitch implementation until we make /bin and /sbin dynamic.
  
  Before someone says you can dlopen() from static binaries in order to
  implement nsswitch, please provide the patch proving it.  Our best
  FreeBSD minds don't think it can be done properly and sanely.
 
 I have the patch. Currently it is made against RELENG_4 and I have a couple
 of questions about alpha (however it works on alpha too with a few hacks).
 Unfortunately, jdp does not have enough time to review it and I have
 lack of time to port it to -current (that would not be that hard but
 since sparc64 is now Tier-1 platform the patch should be ported to
 sparc64 too but I do not have sparc64 hardware and access to
 panther is very slow from my home).
 
 What is the right place to post the patch and test program
 demonstrating dlopen for statically linked programs?

Put it up somehere on the web or email it to the list.  I'd
be interested in looking at it.

Jake

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-11-04 Thread Max Khon
hi, there!

On Tue, Nov 05, 2002 at 02:18:23AM -0500, Jake Burkholder wrote:

   Before someone says you can dlopen() from static binaries in order to
   implement nsswitch, please provide the patch proving it.  Our best
   FreeBSD minds don't think it can be done properly and sanely.
  
  I have the patch. Currently it is made against RELENG_4 and I have a couple
  of questions about alpha (however it works on alpha too with a few hacks).
  Unfortunately, jdp does not have enough time to review it and I have
  lack of time to port it to -current (that would not be that hard but
  since sparc64 is now Tier-1 platform the patch should be ported to
  sparc64 too but I do not have sparc64 hardware and access to
  panther is very slow from my home).
  
  What is the right place to post the patch and test program
  demonstrating dlopen for statically linked programs?
 
 Put it up somehere on the web or email it to the list.  I'd
 be interested in looking at it.

Ok, I put the patch and test program to
http://people.freebsd.org/~fjoe/libdl.tar.bz2.

Patches are made against RELENG_4 (and all tests were done on RELENG_4)
but it will not be that hard to port everything to -CURRENT.
This is just a proof-of-concept work-in-progress.

The plan is to add this stuff (rtld sources with -DLIBDL) to libc.a
so statically linked programe will have dlopen/dlsym etc.

Problems with current patches are:
- I do not know what to do on alpha with _GOT_END_ and
_GLOBAL_OFFSET_TABLE_ symbols. I had to use ld.so.script
to solve missing _GOT_END_ problem and ifdef out _GLOBAL_OFFSET_TABLE_
usage from alpha/reloc.c for second problem. An advice from alpha rtld
guru will be very useful
- gdb support for shared objects dlopened from statically linked
program is broken
- rtld_exit() is not called on exit so fini functions are not
called on exit
- probably more stuff could be #ifdef'ed out from rtld when it is compiled
with -DLIBDL
- xmalloc and friends names in rtld sources probably should be prepended
with an underscore to prevent name clashes (if this stuff will be included
in libc.a)

Any comments, suggestions, patches will be very appreciated.

/fjoe


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-11-03 Thread Miguel Mendez
On Thu, 31 Oct 2002 14:13:58 -0800
Tim Kientzle [EMAIL PROTECTED] wrote:

Hi,
 
 I can think of three concerns:
 
 1) Fragility.  Could a naive sysadmin (or a dying
 disk) break /[s]bin?
 What if the ldconfig hints files were hosed?
 Is ld-elf.so truly bulletproof?

Agreed, and, fortunately, that was taken into account with the
introduction of the /rescue dir:

christine: {48} du -h /rescue
2.4M/rescue
christine: {49} ls -l /rescue
-r-xr-xr-x  127 root  wheel  2535308 Oct 29 23:02 [
-r-xr-xr-x  127 root  wheel  2535308 Oct 29 23:02 atactl
-r-xr-xr-x  127 root  wheel  2535308 Oct 29 23:02 badsect
-r-xr-xr-x  127 root  wheel  2535308 Oct 29 23:02 brconfig
-r-xr-xr-x  127 root  wheel  2535308 Oct 29 23:02 bunzip2
-r-xr-xr-x  127 root  wheel  2535308 Oct 29 23:02 bzcat
-r-xr-xr-x  127 root  wheel  2535308 Oct 29 23:02 bzip2
-r-xr-xr-x  127 root  wheel  2535308 Oct 29 23:02 cat
-r-xr-xr-x  127 root  wheel  2535308 Oct 29 23:02 ccdconfig
-r-xr-xr-x  127 root  wheel  2535308 Oct 29 23:02 chgrp
-r-xr-xr-x  127 root  wheel  2535308 Oct 29 23:02 chio
-r-xr-xr-x  127 root  wheel  2535308 Oct 29 23:02 chmod
-r-xr-xr-x  127 root  wheel  2535308 Oct 29 23:02 chown
-r-xr-xr-x  127 root  wheel  2535308 Oct 29 23:02 clri
-r-xr-xr-x  127 root  wheel  2535308 Oct 29 23:02 cp
-r-xr-xr-x  127 root  wheel  2535308 Oct 29 23:02 csh
-r-xr-xr-x  127 root  wheel  2535308 Oct 29 23:02 date
[...]

As you all system critical tools are there, statically linked, of
course, so it's no big deal.

 2) Security.  Can LD_LIBRARY_PATH (or other mechanisms)
 be used to deliberately subvert any of these programs?
 (especially the handful of suid/sgid programs here)

Agreed, a quick find shows these set[ug]id programs:

christine: {63} find /bin /sbin -perm -u+s
/bin/rcmd
/sbin/ping
/sbin/ping6
/sbin/shutdown
christine: {64} find /bin /sbin -perm -g+s
/sbin/ccdconfig
/sbin/dump
/sbin/dump_lfs
/sbin/rdump
/sbin/rdump_lfs

I can't come up right now with an idea of how exploiting LD_LIBRARY_PATH
could be useful with any of these, but the possibility exists. OTOH, the
recently added priviledge elevation feature should make it possible to
have *no* setuid programs on a system, and have the kernel elevate
priviledges for certain syscalls, based on the policy created by
systrace.
(I'm talking NetBSD here, this feature is not (yet) in FreeBSD)

 3) Upgrade breakage.  Will this make upgrades more fragile?
 A broken or incomplete upgrade could damage ld-elf.so
 or introduce version skew between /bin and libc.so.
 (Yes, people do rebuild libc without rebuilding world.)

Not a problem because you have /rescue
 
 That's impressive; FreeBSD's /bin is over 7M by
 itself right now.  I would be curious to see
 the results from ls -l /bin on your NetBSD system
 as well.

christine: {66} ls -l /bin
total 2494
-r-xr-xr-x  2 root  wheel8512 Oct 29 23:02 [
-r-xr-xr-x  1 root  wheel8480 Oct 29 22:59 cat
-r-xr-xr-x  1 root  wheel   11296 Oct 29 22:59 chio
-r-xr-xr-x  1 root  wheel7008 Oct 29 22:59 chmod
-r-xr-xr-x  1 root  wheel   13960 Oct 29 22:59 cp
-r-xr-xr-x  3 root  wheel  107976 Oct 29 23:01 cpio
-r-xr-xr-x  1 root  wheel  116812 Oct 29 23:00 csh
-r-xr-xr-x  1 root  wheel9936 Oct 29 23:00 date
-r-xr-xr-x  1 root  wheel   21068 Oct 29 23:00 dd
-r-xr-xr-x  1 root  wheel9268 Oct 29 23:00 df
-r-xr-xr-x  1 root  wheel5320 Oct 29 23:00 domainname
-r-xr-xr-x  1 root  wheel4892 Oct 29 23:00 echo
-r-xr-xr-x  1 root  wheel   43312 Oct 29 23:00 ed
-r-xr-xr-x  1 root  wheel   12352 Oct 29 23:00 expr
-r-xr-xr-x  1 root  wheel5572 Oct 29 23:00 hostname
-r-xr-xr-x  1 root  wheel6720 Oct 29 23:00 kill
-r-xr-xr-x  1 root  wheel  171864 Oct 29 23:00 ksh
-r-xr-xr-x  1 root  wheel6268 Oct 29 23:00 ln
-r-xr-xr-x  1 root  wheel   19108 Oct 29 23:00 ls
-r-xr-xr-x  1 root  wheel6688 Oct 29 23:01 mkdir
-r-xr-xr-x  1 root  wheel   13040 Oct 29 23:01 mt
-r-xr-xr-x  1 root  wheel9692 Oct 29 23:01 mv
-r-xr-xr-x  3 root  wheel  107976 Oct 29 23:01 pax
-r-xr-xr-x  1 root  wheel   27924 Oct 29 23:01 ps
-r-xr-xr-x  1 root  wheel5884 Oct 29 23:01 pwd
-r-sr-xr-x  1 root  wheel9276 Oct 29 23:01 rcmd
-r-xr-xr-x  1 root  wheel   16904 Oct 29 23:01 rcp
-r-xr-xr-x  1 root  wheel9536 Oct 29 23:01 rm
lrwxr-xr-x  1 root  wheel  18 Aug 18  2001 rmail -
/usr/libexec/rmail
-r-xr-xr-x  1 root  wheel5568 Oct 29 23:01 rmdir
-r-xr-xr-x  1 root  wheel   97548 Oct 29 23:01 sh
-r-xr-xr-x  1 root  wheel5892 Oct 29 23:02 sleep
-r-xr-xr-x  1 root  wheel   17860 Oct 29 23:02 stty
-r-xr-xr-x  1 root  wheel4652 Oct 29 23:02 sync
-r-xr-xr-x  1 root  wheel  133656 Oct 29 23:02 systrace
-r-xr-xr-x  3 root  wheel  107976 Oct 29 23:01 tar
-r-xr-xr-x  2 root  wheel8512 Oct 29 23:02 test

   ... a knob in /etc/mk.conf to get the old behaviour,
 
  how about something like that?
 
 Knobs are dangerous because you have to test
 all of the settings.

Knobs are hard, let's go shopping :) Seriously, of course it 

Re: libc size

2002-11-03 Thread Robert Watson

On Sun, 3 Nov 2002, Miguel Mendez wrote:

  2) Security.  Can LD_LIBRARY_PATH (or other mechanisms)
  be used to deliberately subvert any of these programs?
  (especially the handful of suid/sgid programs here)
 ..
 
 I can't come up right now with an idea of how exploiting LD_LIBRARY_PATH
 could be useful with any of these, but the possibility exists. OTOH, the
 recently added priviledge elevation feature should make it possible to
 have *no* setuid programs on a system, and have the kernel elevate
 priviledges for certain syscalls, based on the policy created by
 systrace. 

LD_LIBRARY_PATH is disabled for setuid binaries -- the kernel sets the
P_ISSETUGID flag, which is exported to userspace by issetugid(), which is
in turn checked by the rtld, which will refuse to observe that
environmental variable (and a number of others) as a result.  We have
plenty of dynamically linked setuid binaires in the system already, and
it's not a problem. 

Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
[EMAIL PROTECTED]  Network Associates Laboratories


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-11-03 Thread Robert Watson

On Sun, 3 Nov 2002, Robert Watson wrote:

 On Sun, 3 Nov 2002, Miguel Mendez wrote:
 
   2) Security.  Can LD_LIBRARY_PATH (or other mechanisms)
   be used to deliberately subvert any of these programs?
   (especially the handful of suid/sgid programs here)
  ..
  
  I can't come up right now with an idea of how exploiting LD_LIBRARY_PATH
  could be useful with any of these, but the possibility exists. OTOH, the
  recently added priviledge elevation feature should make it possible to
  have *no* setuid programs on a system, and have the kernel elevate
  priviledges for certain syscalls, based on the policy created by
  systrace. 
 
 LD_LIBRARY_PATH is disabled for setuid binaries -- the kernel sets the
 P_ISSETUGID flag, which is exported to userspace by issetugid(), which is
 in turn checked by the rtld, which will refuse to observe that
 environmental variable (and a number of others) as a result.  We have
 plenty of dynamically linked setuid binaires in the system already, and
 it's not a problem. 

Due to sucky latency, I didn't realize y fingers had typed the constant
there incorrectly, that should read P_SUGID.  That same protection also
prevents debugging of processes following privilege downgrade, amongst
other things.

On the systrace issue -- I have lasting concerns about the race conditions
present in fine-grained SMP and threaded systems, such as FreeBSD 5, or
present in systems providing Linux clone() emulation.  Neils has addressed
some but not all of these concerns; until they are fully addressed, the
race conditions there will remain a serious problem.  When the scheduler
activation work hits the main NetBSD tree, I would expect similar
problems.

Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
[EMAIL PROTECTED]  Network Associates Laboratories





To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-11-03 Thread David Schultz
Thus spake Miguel Mendez [EMAIL PROTECTED]:
 Why? I'd love to hear some real reasons for this. NetBSD-current has
 just gone fully dynamic, let's see how much space that needs...
 
 christine: {16} uname -srnm
 NetBSD christine.energyhq.tk 1.6J i386
 christine: {17} du -h /bin /sbin /lib
 999K/bin
 1.7M/sbin
 2.0M/lib
 
 /lib keeps the required shared libs for those programs residing in
 /[s]bin.
 
 IMHO it would be beneficial to, at least, have the option to build a
 fully dynamic system on FreeBSD.

Keep in mind that NetBSD's /bin and /sbin were significantly
smaller beforehand.  FreeBSD's equivalent is somewhat bloated, but
it's not so unreasonably large that people are having trouble
making it fit on hard drives purchased in the last four years.  If
you're installing FreeBSD on an embedded system, that's one thing,
but you have PicoBSD for that.  Otherwise, you're sacrificing
performance, reliability, and security for the sake of a few dozen
megs of disk space.  I don't think this is an acceptable tradeoff.

I'm not opposed to having a knob that allows people to dynamically
link their /bin and /sbin if they so desire, as long as it's done
right.  I worry a little bit that once the feature is available,
someone will decide that it's a bright idea to turn it on by
default, and support for the static linking case will decline.
(This is similar to what happened with GEOM, although I happen to
be all in favor of the latter.  C'est la vie.)

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-11-03 Thread David Schultz
Thus spake Robert Watson [EMAIL PROTECTED]:
  I can't come up right now with an idea of how exploiting LD_LIBRARY_PATH
  could be useful with any of these, but the possibility exists. OTOH, the
  recently added priviledge elevation feature should make it possible to
  have *no* setuid programs on a system, and have the kernel elevate
  priviledges for certain syscalls, based on the policy created by
  systrace. 
 
 LD_LIBRARY_PATH is disabled for setuid binaries -- the kernel sets the
 P_ISSETUGID flag, which is exported to userspace by issetugid(), which is
 in turn checked by the rtld, which will refuse to observe that
 environmental variable (and a number of others) as a result.  We have
 plenty of dynamically linked setuid binaires in the system already, and
 it's not a problem. 

Setuid binaries are in pretty good shape, since LD_LIBRARY_PATH is
ignored for them.  But most of the other standard binaries on the
system are problematic because they are trusted, but they in turn
trust the value of LD_LIBRARY_PATH, which is not always a safe
thing to do.  For example, if an administrator disables a user's
account by setting the shell to a dynamically linked /sbin/nologin,
the user can override the restriction by setting an LD_LIBRARY_PATH
in his ~.ssh/environment file pointing to a trojaned libc.  (This
attack requires that he be able to access his home directory,
either before being locked out or over NFS or FTP.)  You can
correctly argue that setting a user's shell to /sbin/nologin is
the wrong way to disable the account.  Fine.  Remove /sbin/nologin.
I still claim that this is just one example of what can go wrong
when you divide trust between libc and your standard binaries.

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-11-01 Thread M. Warner Losh
In message: [EMAIL PROTECTED]
Wesley Morgan [EMAIL PROTECTED] writes:
: And of course the answer to that is to create a /lib. Something that I
: would *never ever* want to see. Sure, a few people might throw around the
: idea of an extremely light-weight set of libraries to go into /lib blah
: blah. But I just don't like the idea. Why not create a minimalist C
: library, build with -nostdlib and staticly link against exactly what you
: need.
: 
: I usually create a 128 or 64mb root, and the only time this gets tight
: is when I keep too many kernels around in /boot. I seem to recall other
: arguments being settled by the disk space is extremely cheap issue.
: 
: Call me crazy, but FreeBSD just has this zen feeling to it, and making
: this kind of change doesnt feel very zennish. I'm sure there are greater
: minds than mine working over this issue, but thats my $0.02.

Actually, NetBSD has done exactly that (created a /lib).  They found
that putting only the libraries necessary for boot in there that they
were able to save aboue 10M on the size of /, even after one creates a
few static binaries in /something-like-recover.

Warner

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-10-31 Thread Ollivier Robert
According to Terry Lambert:
 The PIC overhead is likely unavoidable.  I'd actually like to see
 the benchmark run on statically linked PIC vs. non-PIC code, so

I remember that when I was working on Perl and the FreeBSD port (back in the
early 5.000 days), having libperl shared was adding a fairly large
overhead. make test ran in between 15% and 25% more time in the shared
libperl case...
-- 
Ollivier ROBERT -=- FreeBSD: The Power to Serve! -=- [EMAIL PROTECTED]
FreeBSD keltia.freenix.fr 5.0-CURRENT #80: Sun Jun  4 22:44:19 CEST 2000

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-10-31 Thread Ollivier Robert
According to David Schultz:
 Memory is even less of an issue; if a thousand copies of a shell
 are running, their text gets shared regardless of how they are
 linked.

IIRC not exactly. In the dynamic case, some fixups are done by the dynamic
linker to link with the shared libs and that force the pages to be COW'd
thus taking more VM. That's why static binaries are more efficient too.

(someone who understand these issues please correct me if necessary)
-- 
Ollivier ROBERT -=- FreeBSD: The Power to Serve! -=- [EMAIL PROTECTED]
FreeBSD keltia.freenix.fr 5.0-CURRENT #80: Sun Jun  4 22:44:19 CEST 2000

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-10-31 Thread Tim Robbins
On Wed, Oct 30, 2002 at 12:46:15PM -0800, Nate Lawson wrote:

 After a discussion on cvs-all regarding size of our libc, I wrote a quick
 script to see where the problems are.  A cursory glance at its output
 shows there are numerous things we can improve, including:
 
   * setproctitle(3) uses 4k of static scratch buffers when it could
 allocate these on the stack (let alone reducing the length of the
 proc title to something more reasonable than 2k).
 
   * vfwprintf and vfprintf are near duplicates of each other (in fact,
 the former is derived from the latter).  Each uses 14k of text so
 this could be split in half by combining them and selecting different
 behavior with a flag.

They are similar enough at the C source level to be merged into a single
source file (in fact, Microsoft's C library does this), but they are
probably not similar enough at the machine code level to merge and save
text space. I have no idea how you could merge them in C to save a
significant amount of text space given that sizeof(char) != sizeof(wchar_t).

Basically, if you don't want wide character printf/scanf support in libc
for some kind of embedded system, remove them from stdio/Makefile.inc.
Nothing in the base system uses them, and few programs in the ports
collection do either. You could also replace the locale stuff with the
dummy versions from 4.4BSD if you don't want it (see OpenBSD's
src/lib/libc/stdlib/multibyte.c, etc.).



Tim

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-10-31 Thread Terry Lambert
Ollivier Robert wrote:
 According to Terry Lambert:
  The PIC overhead is likely unavoidable.  I'd actually like to see
  the benchmark run on statically linked PIC vs. non-PIC code, so
 
 I remember that when I was working on Perl and the FreeBSD port (back in the
 early 5.000 days), having libperl shared was adding a fairly large
 overhead. make test ran in between 15% and 25% more time in the shared
 libperl case...

I'm talking about statically linking the PIC code, to differentiate
the PIC overhead from the shared library mechanism overhead.

-- Terry

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-10-31 Thread Terry Lambert
Ollivier Robert wrote:
 According to David Schultz:
  Memory is even less of an issue; if a thousand copies of a shell
  are running, their text gets shared regardless of how they are
  linked.
 
 IIRC not exactly. In the dynamic case, some fixups are done by the dynamic
 linker to link with the shared libs and that force the pages to be COW'd
 thus taking more VM. That's why static binaries are more efficient too.
 
 (someone who understand these issues please correct me if necessary)

There are one or more pages of indirection pointers that will
initially point to fixup code, so that the first time you
indirect through them, they get fixed up and then indirect
through to the real code, and subsequent indirects indirect to
the real code, rather than the fixup code.

The number of pages that end up COW'ed is pretty minimal.

-- Terry

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-10-31 Thread Tim Kientzle
I agree with David Schultz that dynamically linking
/bin and /sbin is playing with fire.  I, too, have had
ugly experiences on systems that did this:
When /usr won't mount, it is not pleasant to be
stuck with no tools.  (Consider a network environment
where /usr is NFS-mounted as an extreme example.)

As for the disk space concerns, I spent a couple of

hours with some of the smaller utilities.  Identical
functionality to the originals, still statically linked.
Compare with 'ls -l /bin' on your system:
  * echo: 3k
  * sleep: 3k
  * sync: 3k
  * cat: 40k with setlocale, 12k without
(cat uses setlocale for non-standard -v option)
Total savings: over 150k just from these four.
This is under 4.x, though; last I checked, -CURRENT still
had some ugly bloat in crt.  (crt requires atexit, which
requires malloc, ugh.  Maybe this has been fixed?)

Easily-commitable details on request.

As for the resolver issue, maybe there's
another way to separate the resolver?  For
example, a simple command-line resolver utility
(something like a stripped-down 'dig') could
be invoked.  That would leave one utility
with the resolver libs statically-linked,
and remove it from many other places.  There's
an obvious performance hit, but resolving is a
fairly high-overhead operation in any case.

Alternatively, some sort of simple resolver
daemon could be implemented.  The BIND
folks are already moving in this direction
because of IP6 resolver bloat, I understand.
Either approach would shed resolver bloat from
a lot of places without the headaches of
dynamic linking.

Tim Kientzle




To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-10-31 Thread Wesley Morgan
On Thu, 31 Oct 2002, Tim Kientzle wrote:

 I agree with David Schultz that dynamically linking
 /bin and /sbin is playing with fire.  I, too, have had
 ugly experiences on systems that did this:
 When /usr won't mount, it is not pleasant to be
 stuck with no tools.  (Consider a network environment
 where /usr is NFS-mounted as an extreme example.)

And of course the answer to that is to create a /lib. Something that I
would *never ever* want to see. Sure, a few people might throw around the
idea of an extremely light-weight set of libraries to go into /lib blah
blah. But I just don't like the idea. Why not create a minimalist C
library, build with -nostdlib and staticly link against exactly what you
need.

I usually create a 128 or 64mb root, and the only time this gets tight
is when I keep too many kernels around in /boot. I seem to recall other
arguments being settled by the disk space is extremely cheap issue.

Call me crazy, but FreeBSD just has this zen feeling to it, and making
this kind of change doesnt feel very zennish. I'm sure there are greater
minds than mine working over this issue, but thats my $0.02.


-- 
Hi! I'm a .signature virus! Copy me into your ~/.signature to help me spread!


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-10-31 Thread Miguel Mendez
On Thu, 31 Oct 2002 14:19:39 -0500 (EST)
Wesley Morgan [EMAIL PROTECTED] wrote:

Hi,

 And of course the answer to that is to create a /lib. Something that
 I would *never ever* want to see. Sure, a few people might throw
 around the
^^

Why? I'd love to hear some real reasons for this. NetBSD-current has
just gone fully dynamic, let's see how much space that needs...

christine: {16} uname -srnm
NetBSD christine.energyhq.tk 1.6J i386
christine: {17} du -h /bin /sbin /lib
999K/bin
1.7M/sbin
2.0M/lib

/lib keeps the required shared libs for those programs residing in
/[s]bin.

IMHO it would be beneficial to, at least, have the option to build a
fully dynamic system on FreeBSD.

NetBSD now defaults to fully dynamic, but you can set a knob in
/etc/mk.conf to get the old behaviour, how about something like that?

Cheers,
-- 
Miguel Mendez - [EMAIL PROTECTED]
GPG Public Key :: http://energyhq.homeip.net/files/pubkey.txt
EnergyHQ :: http://www.energyhq.tk
NetBSD :: Unix without hype

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-10-31 Thread Tim Kientzle
Wesley Morgan [EMAIL PROTECTED] wrote:

... create a /lib ... that I would *never ever* want to see.



Miguel Mendez wrote:

Why? I'd love to hear some real reasons for this.



I can think of three concerns:

1) Fragility.  Could a naive sysadmin (or a dying
   disk) break /[s]bin?
   What if the ldconfig hints files were hosed?
   Is ld-elf.so truly bulletproof?

2) Security.  Can LD_LIBRARY_PATH (or other mechanisms)
   be used to deliberately subvert any of these programs?
   (especially the handful of suid/sgid programs here)

3) Upgrade breakage.  Will this make upgrades more fragile?
   A broken or incomplete upgrade could damage ld-elf.so
   or introduce version skew between /bin and libc.so.
   (Yes, people do rebuild libc without rebuilding world.)

I am certain these concerns could be addressed,
and a dynamic /bin could be made workable, but
it would require a lot of care.



christine: {16} uname -srnm
NetBSD christine.energyhq.tk 1.6J i386
christine: {17} du -h /bin /sbin /lib
999K/bin
1.7M/sbin
2.0M/lib



That's impressive; FreeBSD's /bin is over 7M by
itself right now.  I would be curious to see
the results from ls -l /bin on your NetBSD system
as well.



... a knob in /etc/mk.conf to get the old behaviour,



how about something like that?


Knobs are dangerous because you have to test
all of the settings.

Tim Kientzle


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-10-30 Thread Terry Lambert
Nate Lawson wrote:
 Here is a link to the size of various components of libc, sorted by text
 size.  If you can find some way to reduce or even remove some of this,
 please submit a patch.
 
   http://www.root.org/~nate/freebsd/lib_size.out

Move the resolver code out to ibresolv.so, and link libc.so
against libresolv.so so that legacy applications are happy, as
long as they are compiled shared.  Non-network apps can ignore
most of it.  Internal use of some of the biggest chunks is
limited, so this should avoid dragging in a lot of it.

-- Terry

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-10-30 Thread Peter Wemm
Terry Lambert wrote:
 Nate Lawson wrote:
  Here is a link to the size of various components of libc, sorted by text
  size.  If you can find some way to reduce or even remove some of this,
  please submit a patch.
  
http://www.root.org/~nate/freebsd/lib_size.out
 
 Move the resolver code out to ibresolv.so, and link libc.so
 against libresolv.so so that legacy applications are happy, as
 long as they are compiled shared.  Non-network apps can ignore
 most of it.  Internal use of some of the biggest chunks is
 limited, so this should avoid dragging in a lot of it.

We've been over this before.  To make this work right, we need to make
/bin and /sbin dynamically linked.  NetBSD's /rescue/* approach would
solve the oops! and other foot shooting problems.

Cheers,
-Peter
--
Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]
All of this is for nothing if we don't go to the stars - JMS/B5


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-10-30 Thread Doug Rabson
On Wed, 30 Oct 2002, Peter Wemm wrote:

 Terry Lambert wrote:
  Nate Lawson wrote:
   Here is a link to the size of various components of libc, sorted by text
   size.  If you can find some way to reduce or even remove some of this,
   please submit a patch.
  
 http://www.root.org/~nate/freebsd/lib_size.out
 
  Move the resolver code out to ibresolv.so, and link libc.so
  against libresolv.so so that legacy applications are happy, as
  long as they are compiled shared.  Non-network apps can ignore
  most of it.  Internal use of some of the biggest chunks is
  limited, so this should avoid dragging in a lot of it.

 We've been over this before.  To make this work right, we need to make
 /bin and /sbin dynamically linked.  NetBSD's /rescue/* approach would
 solve the oops! and other foot shooting problems.

Yes please. Our root filesystem space requirements are too high, IMHO.

-- 
Doug Rabson Mail:  [EMAIL PROTECTED]
Phone: +44 20 8348 6160



To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-10-30 Thread Terry Lambert
Peter Wemm wrote:
 Terry Lambert wrote:
  Nate Lawson wrote:
   Here is a link to the size of various components of libc, sorted by text
   size.  If you can find some way to reduce or even remove some of this,
   please submit a patch.
  
 http://www.root.org/~nate/freebsd/lib_size.out
 
  Move the resolver code out to ibresolv.so, and link libc.so
  against libresolv.so so that legacy applications are happy, as
  long as they are compiled shared.  Non-network apps can ignore
  most of it.  Internal use of some of the biggest chunks is
  limited, so this should avoid dragging in a lot of it.
 
 We've been over this before.  To make this work right, we need to make
 /bin and /sbin dynamically linked.  NetBSD's /rescue/* approach would
 solve the oops! and other foot shooting problems.

Or add:

LDFLAGS+=   -lresolv

To the Makefiles of the things that need to be statically linked,
and access the network code.

I'm going to go out on a limb here, though, and guess that without
a resolv.conf, most of the resolver library is going to be really
useless.  8-) 8-).

-- Terry

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-10-30 Thread Peter Wemm
Terry Lambert wrote:
 Peter Wemm wrote:
  Terry Lambert wrote:
   Nate Lawson wrote:
Here is a link to the size of various components of libc, sorted by tex
t
size.  If you can find some way to reduce or even remove some of this,
please submit a patch.
   
  http://www.root.org/~nate/freebsd/lib_size.out
  
   Move the resolver code out to ibresolv.so, and link libc.so
   against libresolv.so so that legacy applications are happy, as
   long as they are compiled shared.  Non-network apps can ignore
   most of it.  Internal use of some of the biggest chunks is
   limited, so this should avoid dragging in a lot of it.
  
  We've been over this before.  To make this work right, we need to make
  /bin and /sbin dynamically linked.  NetBSD's /rescue/* approach would
  solve the oops! and other foot shooting problems.
 
 Or add:
 
   LDFLAGS+=   -lresolv
 
 To the Makefiles of the things that need to be statically linked,
 and access the network code.

Except that getpwent() etc have got hard coded references to yp, and yp has
got hard coded references to getXXXbyYYY(), which has references to
libresolv.  The number of things that are infected by this is quite big.
This means a good number of things in /bin and /sbin would have to have
-lresolv added.  And that defeats the point.  All that we do is move
the bloat from one library to another.

It's the same problem with the db embedded in there - getpwent() uses
it via the pwd.db stuff.  And then there's the cgetent() stuff that
that *curses uses to access termcap.db. And login.conf.db. And so on.

 I'm going to go out on a limb here, though, and guess that without
 a resolv.conf, most of the resolver library is going to be really
 useless.  8-) 8-).

Except to satisfy internal dependencies.

 -- Terry
 

Cheers,
-Peter
--
Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]
All of this is for nothing if we don't go to the stars - JMS/B5


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-10-30 Thread Dan Nelson
In the last episode (Oct 30), Doug Rabson said:
 On Wed, 30 Oct 2002, Peter Wemm wrote:
  We've been over this before.  To make this work right, we need to
  make /bin and /sbin dynamically linked.  NetBSD's /rescue/*
  approach would solve the oops! and other foot shooting problems.
 
 Yes please. Our root filesystem space requirements are too high,
 IMHO.

Note that dynamically-linked executables take significantly longer to
exec than statically-linked ones.  Running the following simple script
with getfacl and grep linked dynamically runs a little over twice as
slow as with them both static:

#! /bin/sh
for i in /bin/* /sbin/* /usr/bin/* /usr/sbin/* ; do
 if ! getfacl $i | grep -q owner:0 ; then
   echo owner of $i is not root
 fi
done

Static: 3.20r 0.16u 3.59s   Dynamic: 6.88r 1.60u 7.03s
3.62r 0.18u 3.50s6.82r 2.08u 6.56s
3.26r 0.22u 3.52s6.92r 1.35u 7.26s
3.23r 0.16u 3.62s7.59r 1.25u 7.48s
3.26r 0.19u 3.65s7.74r 1.66u 7.42s
3.63r 0.16u 3.76s7.17r 1.67u 7.15s

6 runs, alternatic static and dynamic scripts for each run.

I actually link quite a few binaries in /usr/bin and /usr/sbin static
(awk, basename, dirname, find, head, install, sed, tr, uniq, wc).  Try
this: enable process accounting, run make fetch-recursive-list in
ports/www/mozilla, then run lastcomm and see how many times tr, sed,
basename, sh, and grep got called.  Linking these static makes a big
difference.

As a compromise, how about converting the major space wasters in /bin
and /sbin to dynamic but leaving the rest?  Except for /bin/sh, none of
them are run often enough to matter.

-r-xr-xr-x  1 root  wheel  475108 Sep  7 17:50 /sbin/routed*
-r-xr-xr-x  1 root  wheel  495943 Sep  7 17:50 /sbin/fore_dnld*
-r-xr-xr-x  1 root  wheel  521654 Oct 22 12:14 /sbin/tunefs*
-r-xr-xr-x  1 root  wheel  528572 Sep  7 17:46 /bin/pax*
-r-xr-xr-x  1 root  wheel  532428 Sep  7 17:46 /bin/ls*
-r-xr-xr-x  1 root  wheel  605588 Sep  7 17:50 /sbin/ipfstat*
-r-xr-xr-x  1 root  wheel  616277 Sep  7 17:50 /sbin/ilmid*
-r-xr-xr-x  1 root  wheel  670188 Sep  7 17:50 /sbin/fsdb*
-r-xr-xr-x  1 root  wheel  695512 Sep  7 17:50 /sbin/vinum*
-r-xr-xr-x  1 root  wheel  713372 Oct  8 00:44 /bin/sh*
-r-xr-xr-x  1 root  wheel  752168 Sep  7 17:58 /sbin/dhclient*
-r-xr-xr-x  2 root  wheel  844512 Sep  7 17:46 /bin/tcsh*
-r-xr-xr-x  1 root  wheel 3214381 Oct 25 10:29 /sbin/ipfw*
-r-xr-xr-x  2 root  wheel 3420946 Oct 25 16:02 /sbin/rdump*
-r-xr-xr-x  3 root  wheel 3464372 Oct 22 12:07 /sbin/fsck_ufs*

-- 
Dan Nelson
[EMAIL PROTECTED]

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-10-30 Thread Terry Lambert
Dan Nelson wrote:
 In the last episode (Oct 30), Doug Rabson said:
  On Wed, 30 Oct 2002, Peter Wemm wrote:
   We've been over this before.  To make this work right, we need to
   make /bin and /sbin dynamically linked.  NetBSD's /rescue/*
   approach would solve the oops! and other foot shooting problems.
 
  Yes please. Our root filesystem space requirements are too high,
  IMHO.
 
 Note that dynamically-linked executables take significantly longer to
 exec than statically-linked ones.  Running the following simple script
 with getfacl and grep linked dynamically runs a little over twice as
 slow as with them both static:

I can't tell if the time difference is coming from where you
say it's coming from, or if it's coming from somewhere else.

If it's coming from where I think it's coming from, the answer
is very easy: in the original ELF specification, the program
offset in the UVM was very large, to permit enough room for
the kernel to pre-mmap both the ld.so and the libc.so into the
address space of each program before it started, making the
decision after it faulted in the first page.

This was the intentional design of the ELF initial offset being
so large.

Basically, this means that the template process that's used to
start new processes following an exec can have a preinitialized
shared address space for these areas, which will save the overhead
that's probably contributing most to the problem.

In other words, this is an exec/crt0 problem, not a static vs.
dynamic problem.

-- Terry

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-10-30 Thread Peter Wemm
Dan Nelson wrote:
 In the last episode (Oct 30), Doug Rabson said:
  On Wed, 30 Oct 2002, Peter Wemm wrote:
   We've been over this before.  To make this work right, we need to
   make /bin and /sbin dynamically linked.  NetBSD's /rescue/*
   approach would solve the oops! and other foot shooting problems.
  
  Yes please. Our root filesystem space requirements are too high,
  IMHO.
 
 Note that dynamically-linked executables take significantly longer to
 exec than statically-linked ones.

Indeed yes.  Running ld-elf.so.1 isn't free.   Also, calling PIC libraries
isn't free either.  Not only that, but even fork(2) is slower when you come
*from* a dynamic executable.

But for /bin/sh, then ~user etc doesn't work too well when your account
exists only somewhere like ldap.  Just like /bin/ls -l doesn't show the
user and groups.

Fortunately the costs associated with this are mostly one-off per exec and
are mostly lost in the noise when the commands being run are doing real
work.

In the days of monsters like mozilla, kde, gnome, gcc-3.x etc, dynamic
binary exec times seem to be the least of our worries. :-(

Cheers,
-Peter
--
Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]
All of this is for nothing if we don't go to the stars - JMS/B5


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-10-30 Thread Terry Lambert
Peter Wemm wrote:
  Note that dynamically-linked executables take significantly longer to
  exec than statically-linked ones.
 
 Indeed yes.  Running ld-elf.so.1 isn't free.   Also, calling PIC libraries
 isn't free either.  Not only that, but even fork(2) is slower when you come
 *from* a dynamic executable.

I can't tell if he's complaining about runtime costs, or startup
costs.

Much of the runtime cost can be gotten rid of by thunking the code,
at the expense of dirting the pages that make the references by
replacing their reference targets with the addresses of the real
functions.  That's way overboard, for my tastes, but it's possible
to remove the indirection overhead that way.

The PIC overhead is likely unavoidable.  I'd actually like to see
the benchmark run on statically linked PIC vs. non-PIC code, so
we can assign the proper portion of the blame for PIC-ness.  IMO,
the cost is actually very small, given that you have to go over a
64K boundary before it becomes really expensive; relative branches
are very common in non-PIC code.

I think the expense in the the LD_PRELOAD treatment, and in the
finding and mapping of the ld.so and library images... and that was
designed to be avoidable by the ELF designers, even if most OSs
never chose to properly implement it, and wedged ld.so loading into
crt0, instead.


 But for /bin/sh, then ~user etc doesn't work too well when your account
 exists only somewhere like ldap.  Just like /bin/ls -l doesn't show the
 user and groups.
 
 Fortunately the costs associated with this are mostly one-off per exec and
 are mostly lost in the noise when the commands being run are doing real
 work.

I agree.  If you are concerned about the overhead, then at least
assign the blame to the proper place; but for the most part, the
concern is unwarranted.


 In the days of monsters like mozilla, kde, gnome, gcc-3.x etc, dynamic
 binary exec times seem to be the least of our worries. :-(

AMEN.

-- Terry

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-10-30 Thread M. Warner Losh
In message: [EMAIL PROTECTED]
Peter Wemm [EMAIL PROTECTED] writes:
: Terry Lambert wrote:
:  Nate Lawson wrote:
:   Here is a link to the size of various components of libc, sorted by text
:   size.  If you can find some way to reduce or even remove some of this,
:   please submit a patch.
:   
: http://www.root.org/~nate/freebsd/lib_size.out
:  
:  Move the resolver code out to ibresolv.so, and link libc.so
:  against libresolv.so so that legacy applications are happy, as
:  long as they are compiled shared.  Non-network apps can ignore
:  most of it.  Internal use of some of the biggest chunks is
:  limited, so this should avoid dragging in a lot of it.
: 
: We've been over this before.  To make this work right, we need to make
: /bin and /sbin dynamically linked.  NetBSD's /rescue/* approach would
: solve the oops! and other foot shooting problems.

Let's make / and /usr on the same partition by default, which makes
this easy... :-)

Warner

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-10-30 Thread David Schultz
Thus spake Doug Rabson [EMAIL PROTECTED]:
   Move the resolver code out to ibresolv.so, and link libc.so
   against libresolv.so so that legacy applications are happy, as
   long as they are compiled shared.  Non-network apps can ignore
   most of it.  Internal use of some of the biggest chunks is
   limited, so this should avoid dragging in a lot of it.
 
  We've been over this before.  To make this work right, we need to make
  /bin and /sbin dynamically linked.  NetBSD's /rescue/* approach would
  solve the oops! and other foot shooting problems.
 
 Yes please. Our root filesystem space requirements are too high, IMHO.

Why is it absolutely necessary to dynamically link everything just
to move the resolver out of libc?  I'm all for doing the latter,
but I think dynamically linking /bin and /sbin is a really bad
idea, for several reasons:

- If the single user mode runtime includes several shared
  libraries, you have several additional points of failure.

- The use of shared libraries incurs a performance hit for
  programs like sh and echo, which should be fast.
  o Exec time is greatly increased because you have to map in the
libraries.
  o The -fPIC code in the shared libraries is slower, particularly
on the register-starved i386.
  o The VM system has to do more work when you have a sparse virtual
memory map, which is what you get when you stick shared libraries
in the middle of your address space.

- Using shared libraries for trusted binaries like sh has negative
  consequences for security.  For example, a `feature' of OpenSSH
  allows users to circumvent restrictions imposed using /sbin/nologin,
  provided that /sbin/nologin is dynamically linked.

I don't think the disk space argument outweighs these problems.
There are a few binaries in /sbin that are bigger than they ought
to be, but it isn't as though /bin and /sbin occupy 500 MB.
Memory is even less of an issue; if a thousand copies of a shell
are running, their text gets shared regardless of how they are
linked.

I have run into problems with dynamic linking in the past.  In one
case, I couldn't log into a Linux machine because the
administrator (who used bash, of course) had tcsh linked against a
version of glibc that didn't exist on his system.  I would really
hate to see FreeBSD open itself up to the same kinds of
foot-shooting opportunities just because of the resolver, or for
the sake of a few dozen megs of disk space.  Of all things, don't
dynamically link my shell.

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-10-30 Thread Tim Kientzle
Nate Lawson wrote:
 Here is a link to the size of various components of libc, ...




Terry Lambert wrote:

Move the resolver code out to ibresolv.so, ...



Peter Wemm:


We've been over this before.  To make this work right, we



need to make /bin and /sbin dynamically linked.



Peter,

Could you elaborate?  I'm not sure I follow you.
How would dynamically linking /bin and /sbin
make this work right?

Tim Kientzle


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-10-30 Thread Terry Lambert
David Schultz wrote:
   We've been over this before.  To make this work right, we need to make
   /bin and /sbin dynamically linked.  NetBSD's /rescue/* approach would
   solve the oops! and other foot shooting problems.
 
  Yes please. Our root filesystem space requirements are too high, IMHO.
 
 Why is it absolutely necessary to dynamically link everything just
 to move the resolver out of libc?

Because ELF supports linking a shared library to another shared
library, which will automatically get you the appearance of the
historical libresolv is integrated into libc.  But it does not
support the linking of a static library to a static library, or
a static library to a shared library, the same way.

The ELF specification never expected things to be linked statically.

-- Terry

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-10-30 Thread David Schultz
Thus spake Terry Lambert [EMAIL PROTECTED]:
 David Schultz wrote:
We've been over this before.  To make this work right, we need to make
/bin and /sbin dynamically linked.  NetBSD's /rescue/* approach would
solve the oops! and other foot shooting problems.
  
   Yes please. Our root filesystem space requirements are too high, IMHO.
  
  Why is it absolutely necessary to dynamically link everything just
  to move the resolver out of libc?
 
 Because ELF supports linking a shared library to another shared
 library, which will automatically get you the appearance of the
 historical libresolv is integrated into libc.  But it does not
 support the linking of a static library to a static library, or
 a static library to a shared library, the same way.

At least in the case of the base system, it should be easy to link
all programs that actually use the resolver with -lresolv.  Is
there some standard that says that the resolver is an integral
part of the C library, such that separating the two would break
compatibility beyond comprehension?

If you wanted to be really evil, I suppose you could have a libc.a
that included the resolver and a libc.so that didn't.  ;-)

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message



Re: libc size

2002-10-30 Thread Terry Lambert
David Schultz wrote:
 At least in the case of the base system, it should be easy to link
 all programs that actually use the resolver with -lresolv.  Is
 there some standard that says that the resolver is an integral
 part of the C library, such that separating the two would break
 compatibility beyond comprehension?

It is a historical matter of pride in BSD-land that the resolver
is both available merely by linking libc (BSD is a true networking
OS), and that the resolver code is perpetually out of datem
relative to the ISC BIND distribution version of the resolver
code.


 If you wanted to be really evil, I suppose you could have a libc.a
 that included the resolver and a libc.so that didn't.  ;-)

It's a lot easier to just have universal rules, and ignore the
fact that ELF lets you do these things, if you want to ignore the
fact that ELF was never written for static libraries in the first
place, and let people link things static.  8-) 8-).

-- Terry

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message