Re: Unfortunate dynamic linking for everything

2003-11-25 Thread Tony Finch
Matthew D. Fuller [EMAIL PROTECTED] wrote:

Not just the startup scripts, but ANY script.  I dare say there's a long,
long list of scripts that use ~-expansion, to say nothing of the
homegrown ones we all have working quietly and forgotten for years.

It's required for POSIX compliance.

Tony.
-- 
f.a.n.finch  [EMAIL PROTECTED]  http://dotat.at/
IRISH SEA: SOUTHWESTERLY 5 OR 6. SHOWERS. GOOD.
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-25 Thread Tony Finch
Robert Watson [EMAIL PROTECTED] wrote:

Someone must be using /bin/sh as a shell, because apparently someone
spent a lot of time adding things like character input editing, filename
completion, etc.  We even use sh as the default in adduser(8).

Command-line editing is required for POSIX compliance.

Tony.
-- 
f.a.n.finch  [EMAIL PROTECTED]  http://dotat.at/
FAIR ISLE: SOUTHWESTERLY BACKING EASTERLY 5 TO 7, PERHAPS GALE 8 LATER.
SHOWERS THEN RAIN. MODERATE OR GOOD.
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-25 Thread Tim Kientzle
Tony Finch wrote:
Matthew D. Fuller [EMAIL PROTECTED] wrote:

Not just the startup scripts, but ANY script.  I dare say there's a long,
long list of scripts that use ~-expansion, to say nothing of the
homegrown ones we all have working quietly and forgotten for years.


It's required for POSIX compliance.

Tony.
Ouch!  Very good point, Tony.

Does POSIX require that such expansion work for
usernames that may not be in the current passwd
file?
That could dictate a lot.

Tim Kientzle

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-24 Thread Matthew D. Fuller
[ Lots of CC trimming ]

On Sun, Nov 23, 2003 at 06:27:01PM -0500 I heard the voice of
Richard Coleman, and lo! it spake thus:
 
 You would need to make sure that startup scripts never use tilde 
 expansion.  I'm not sure how common that is with RCNG.

Not just the startup scripts, but ANY script.  I dare say there's a long,
long list of scripts that use ~-expansion, to say nothing of the
homegrown ones we all have working quietly and forgotten for years.


-- 
Matthew Fuller (MF4839)   |  [EMAIL PROTECTED]
Systems/Network Administrator |  http://www.over-yonder.net/~fullermd/

The only reason I'm burning my candle at both ends, is because I
  haven't figured out how to light the middle yet
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-24 Thread Duncan Barclay
 
 From: David O'Brien [EMAIL PROTECTED]
 
  I'll seriously argue against the 2nd point above.  I don't know of a
  SINGLE person that uses /bin/sh as their interactive shell when
  multi-user.  Not ONE.  Every Bourne shell'ish user I've ever met uses
  Bash, ATT ksh, pdksh, zsh.
 
 I don't know anyone that farms lama's, so there cannot be any lama
farmers.
 
 computer$ grep dmlb /etc/passwd
 dmlb:*:1166:1166:Duncan Barclay:/home/dmlb:/bin/sh
 
 Duncan
 So, imagine, i'm accidentally deleted /bin with your most wanted
 static sh... And, of course, due to static nature of /bin/sh it was
 removed from /rescue? Nothing will protect you from shooting in the leg,
 neither static linking, nor assumption that /lib is OK.


 MOST people uses /bin/sh only for rc scripts (to be correct, their system
 uses it). David O'Brien just tried to told, that NOBODY he knows will be
 REALLY impacted by performance loss, caused due dynamic /bin/sh linking.
 You will... So, because Duncan Barclay is impacted by performance
 loss due dynamic /bin/sh linking, ENTIRE FreeBSD community will have
 troubles (at least with NSS) due to static linking...

Maxim, I was merely rising to David's bait for some proof that people use
/bin/sh as an interactive shell. You're correct in saying that if /bin/sh on
my machines gets hosed it won't
matter too much - I'll use another shell to rebuild the machine, or boot
from an install CD
to get a shell. But as part of me fixing the machine, I'll put /bin/sh back
on.

I didn't say anything about NSS and I don't intend to as I don't need it.
The debate should be held between people that need the functionality but
want it implemented in different ways.
As to performance loss, I really don't think I'm going to notice it - the
cheapest machine I can find in the UK is an Athlon 1800XP, that has a lot
more grunt than I need. This may not be true for others.

Duncan

Sincerely, Maxim M. Kazachek
mailto:[EMAIL PROTECTED]
mailto:[EMAIL PROTECTED]




___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-24 Thread Duncan Barclay
 On Mon, Nov 24, 2003 at 12:14:39AM -, Duncan Barclay wrote:
 
  From: David O'Brien [EMAIL PROTECTED]
 
   I'll seriously argue against the 2nd point above.  I don't know of a
   SINGLE person that uses /bin/sh as their interactive shell when
   multi-user.  Not ONE.  Every Bourne shell'ish user I've ever met uses
   Bash, ATT ksh, pdksh, zsh.
 
  I don't know anyone that farms lama's, so there cannot be any lama
farmers.

 One has to make a strong statement to get the people to come out of the
 woodwork.

Ack.

  computer$ grep dmlb /etc/passwd
  dmlb:*:1166:1166:Duncan Barclay:/home/dmlb:/bin/sh

 Good.  Now do you need NSS support?  Do the benefits of supporting NSS in
 /bin/sh for you out-weigh the performance issue of building it
 dynamically?  Couldn't you just as easily use the pdksh port?

The machine use I generate doesn't really require a lot of /bin/sh
invocations. Either I have file servers, shell boxes, or compute engines
running CPU bound jobs for half an hour upwards. Whether it takes a gnats
longer to start /bin/sh doesn't really matter to me. However, NSS is likely
to feature as needed sometime soon.

Duncan

 --
 -- David  ([EMAIL PROTECTED])



___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-24 Thread Robert Watson

On Mon, 24 Nov 2003, Maxim M. Kazachek wrote:

 MOST people uses /bin/sh only for rc scripts (to be correct, their
 system uses it). David O'Brien just tried to told, that NOBODY he knows
 will be REALLY impacted by performance loss, caused due dynamic /bin/sh
 linking.  You will... So, because Duncan Barclay is impacted by
 performance loss due dynamic /bin/sh linking, ENTIRE FreeBSD community
 will have troubles (at least with NSS) due to static linking... 

Actually, you appear to be agreeing with him, not disagreeing with him. 
Duncan was pointing out that he *does* use /bin/sh as his shell, in
response to David's suggestion that on one uses it and therefore that
making it statically linked wouldn't hurt. 

It strikes me that this whole conversation has gotten a little
confrontational...  The middle ground of adding a static /sbin/sh for
scripts soudds like a reasonable choice, and has precedent in other
systems (Solaris).  We can set the boot and periodic scripts to use that,
and interactive users can keep using /bin/sh.  Someone must be using
/bin/sh as a shell, because apparently someone spent a lot of time adding
things like character input editing, filename completion, etc.  We even
use sh as the default in adduser(8).

Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
[EMAIL PROTECTED]  Network Associates Laboratories


___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-24 Thread Andrew Gallatin

Robert Watson writes:
  
  It strikes me that this whole conversation has gotten a little
  confrontational...  The middle ground of adding a static /sbin/sh for
  scripts soudds like a reasonable choice, and has precedent in other
  systems (Solaris).  We can set the boot and periodic scripts to use that,
  and interactive users can keep using /bin/sh.  Someone must be using
  /bin/sh as a shell, because apparently someone spent a lot of time adding
  things like character input editing, filename completion, etc.  We even
  use sh as the default in adduser(8).

Tru64 also does the /sbin/sh (static) and /bin/sh (dynamic) thing.
I've never liked it because standard, portable shell scipts expect to
use /bin/sh.  Standard, portable shell scripts don't need tilde
expansion for ldap/nss usernames.

So I think the best solution (*) would be to keep /bin/sh statically
linked, and build a dynamic version in /usr/bin that people can use as
an interactive shell.  Root's shell remains /bin/sh

1) All three (;-) interactive bourne shell users that use nss/ldap get 
   tilde expansion.

2) 3rd party scripts, which are written to be portable and don't use
   tilde, don't have to pay for dynamic linking.

3) System startup scripts are faster because they use the static
   /bin/sh


(*) The real best solution would be to backout the dynamic linking
changes and put the onus on those who want/need nss/ldap to prove that
the dynamic linking changes do not produce a measurable slowdown (as
they were asked to do 6 months ago, and never did).  If it really
slows things down, then they need to fix the performance problems by
either speeding up dynamic linking,  or getting the features they need
in another, less invasive way (as discussed previously in this
thread).

Drew

[reply-to set to freebsd-current, as I don't want to be CC'ed on this
thread forever.. ]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-24 Thread David O'Brien
On Mon, Nov 24, 2003 at 10:47:24AM -0500, Robert Watson wrote:
 It strikes me that this whole conversation has gotten a little
 confrontational...  The middle ground of adding a static /sbin/sh for
 scripts soudds like a reasonable choice, and has precedent in other
 systems (Solaris).

Time for a pdksh import!  We can install it as the dynamic /bin/sh, and
there is precedent in other systems (Solaris) for /bin/ksh.

The issue with static /sbin/sh is that we have to fix all the base
scripts (doable), and then try to educate people that they need to do the
same to their scripts (very hard to do).


 Someone must be using /bin/sh as a shell, because apparently someone
 spent a lot of time adding things like character input editing,
 filename completion, etc.

Filename complete isn't in stock /bin/sh as its a private hack someone
did, it is a hack and isn't commitable.
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-24 Thread Richard Coleman
Andrew Gallatin wrote:
So I think the best solution (*) would be to keep /bin/sh statically
linked, and build a dynamic version in /usr/bin that people can use as
an interactive shell.  Root's shell remains /bin/sh
1) All three (;-) interactive bourne shell users that use nss/ldap get 
   tilde expansion.

2) 3rd party scripts, which are written to be portable and don't use
   tilde, don't have to pay for dynamic linking.
3) System startup scripts are faster because they use the static
   /bin/sh
Are you suggesting that (t)csh also move to /usr/bin to match 
/usr/bin/sh?  The screams caused by such a change would be deafening.

Richard Coleman
[EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-24 Thread Andrew Gallatin

Richard Coleman writes:
  Are you suggesting that (t)csh also move to /usr/bin to match 
  /usr/bin/sh?  The screams caused by such a change would be deafening.

Of course not.  Nobody in their right mind uses csh for scripting.

Drew
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-24 Thread Matthias Andree
Richard Coleman [EMAIL PROTECTED] writes:

 Are you suggesting that (t)csh also move to /usr/bin to match
 /usr/bin/sh?  The screams caused by such a change would be deafening.

Would there be any screams at all?

chsh -s /bin/sh root# prevent lock-out
rm -f /bin/csh /bin/tcsh
echo NO_TCSH=true /etc/make.conf  # avoid resurrection of evil
# next make world

Looks that a system without [t]csh is supported. And it works for me.

-- 
Matthias Andree

Encrypt your mail: my GnuPG key ID is 0x052E7D95
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-24 Thread Brad Knowles
At 8:59 PM -0500 2003/11/24, Andrew Gallatin wrote:

 Of course not.  Nobody in their right mind uses csh for scripting.
	To my great horror, csh is used in most of the DNS debugging and 
many of the log-processing scripts that I have inherited.  One of 
these days, I will finally live up to my threat of importing all this 
functionality into other programs that use other languages (toss 
doc and incorporate that functionality into dnswalk, etc...), but 
that has not happened yet.

	Meanwhile, I don't know that a dynamic vs. static csh does me any 
measurable harm -- the delays waiting for responses from nameservers 
will overwhelm any local delays caused by dynamic vs. static linking.

--
Brad Knowles, [EMAIL PROTECTED]
They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety.
-Benjamin Franklin, Historical Review of Pennsylvania.
GCS/IT d+(-) s:+(++): a C++(+++)$ UMBSHI$ P+++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+() DI+() D+(++) G+() e++ h--- r---(+++)* z(+++)
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-23 Thread David O'Brien
 As I pointed out earlier, some of the heat here comes
 from the fact that /bin/sh is currently overloaded:

  * It is the default system script interpreter, used
by the rc scripts and many other things.  As such,
it must start quickly.

  * It is the default user shell for many users.  As such,
it must support NSS.

 So far, I haven't seen anyone in this thread seriously
 argue against either of these points.

I'll seriously argue against the 2nd point above.  I don't know of a
SINGLE person that uses /bin/sh as their interactive shell when
multi-user.  Not ONE.  Every Bourne shell'ish user I've ever met uses
Bash, ATT ksh, pdksh, zsh.


 Richard Coleman wrote:
 It seems /bin/sh is the real sticking point. 
 
 There is a problem here: Unix systems have historically used
 /bin/sh for two somewhat contradictory purposes:
   * the system script interpreter
   * as a user shell
 
 The user shell must be dynamically linked in order
 to support centralized administration.  I personally
 see no way around that.  Given that many users do
 rely on /bin/sh, it seems that /bin/sh must be
 dynamically linked.

 There are good reasons to want the system script
 interpreter statically linked.
 
 Maybe it's time to separate these two functions?

I argue the two functions are already separated as /bin/sh as
interactive shell doesn't really exist outside of single user.

We should build /bin/sh static and be done with the argument.
Or rather, lets find a /bin/sh interactive user and have him argue that
/bin/sh needs NSS support.  I dare say that will be a thread two orders
of magnitude shorter than this one.

-- 
-- David  ([EMAIL PROTECTED])
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-23 Thread Richard Coleman
David O'Brien wrote:
We should build /bin/sh static and be done with the argument.
Or rather, lets find a /bin/sh interactive user and have him argue that
/bin/sh needs NSS support.  I dare say that will be a thread two orders
of magnitude shorter than this one.
Statically linking /bin/sh wouldn't be that big a deal.  It would be 
trivial to add a port to compile a dynamic version of sh for customers 
that *really* want sh.  Personally, I can't remember a customer 
specifically requesting that.

You would need to make sure that startup scripts never use tilde 
expansion.  I'm not sure how common that is with RCNG.

But it would be sorta odd to have statically linked versions of sh in 
both /bin and /rescue.

Richard Coleman
[EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-23 Thread Duncan Barclay

From: David O'Brien [EMAIL PROTECTED]
 
 I'll seriously argue against the 2nd point above.  I don't know of a
 SINGLE person that uses /bin/sh as their interactive shell when
 multi-user.  Not ONE.  Every Bourne shell'ish user I've ever met uses
 Bash, ATT ksh, pdksh, zsh.

I don't know anyone that farms lama's, so there cannot be any lama farmers.

computer$ grep dmlb /etc/passwd
dmlb:*:1166:1166:Duncan Barclay:/home/dmlb:/bin/sh

Duncan
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-23 Thread Thomas David Rivers
  So far, I haven't seen anyone in this thread seriously
  argue against either of these points.
 
 I'll seriously argue against the 2nd point above.  I don't know of a
 SINGLE person that uses /bin/sh as their interactive shell when
 multi-user.  Not ONE.  Every Bourne shell'ish user I've ever met uses
 Bash, ATT ksh, pdksh, zsh.
 

 I'm one...

 I have to operate on many disparate systems with the
 same home directory some of the bourne-shell variants
 (e.g. HP/UX's ksh) will lock-up on my simple bourne-shell .profile.

 So - I stick with the plain/old/boring sh.

- Dave R. -

--
[EMAIL PROTECTED]Work: (919) 676-0847
Get your mainframe programming tools at http://www.dignus.com
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-23 Thread David O'Brien
On Mon, Nov 24, 2003 at 12:14:39AM -, Duncan Barclay wrote:
 
 From: David O'Brien [EMAIL PROTECTED]
  
  I'll seriously argue against the 2nd point above.  I don't know of a
  SINGLE person that uses /bin/sh as their interactive shell when
  multi-user.  Not ONE.  Every Bourne shell'ish user I've ever met uses
  Bash, ATT ksh, pdksh, zsh.
 
 I don't know anyone that farms lama's, so there cannot be any lama farmers.

One has to make a strong statement to get the people to come out of the
woodwork.

 computer$ grep dmlb /etc/passwd
 dmlb:*:1166:1166:Duncan Barclay:/home/dmlb:/bin/sh

Good.  Now do you need NSS support?  Do the benefits of supporting NSS in
/bin/sh for you out-weigh the performance issue of building it
dynamically?  Couldn't you just as easily use the pdksh port?

-- 
-- David  ([EMAIL PROTECTED])
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-23 Thread David O'Brien
On Sun, Nov 23, 2003 at 06:27:01PM -0500, Richard Coleman wrote:
 But it would be sorta odd to have statically linked versions of sh in 
 both /bin and /rescue.

We'd remove it from /rescue if the /bin/sh one was static. :-)
 
-- 
-- David  ([EMAIL PROTECTED])
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-23 Thread Maxim M. Kazachek
On Mon, 24 Nov 2003, Duncan Barclay wrote:


From: David O'Brien [EMAIL PROTECTED]

 I'll seriously argue against the 2nd point above.  I don't know of a
 SINGLE person that uses /bin/sh as their interactive shell when
 multi-user.  Not ONE.  Every Bourne shell'ish user I've ever met uses
 Bash, ATT ksh, pdksh, zsh.

I don't know anyone that farms lama's, so there cannot be any lama farmers.

computer$ grep dmlb /etc/passwd
dmlb:*:1166:1166:Duncan Barclay:/home/dmlb:/bin/sh

Duncan
So, imagine, i'm accidentally deleted /bin with your most wanted
static sh... And, of course, due to static nature of /bin/sh it was
removed from /rescue? Nothing will protect you from shooting in the leg,
neither static linking, nor assumption that /lib is OK.

MOST people uses /bin/sh only for rc scripts (to be correct, their system
uses it). David O'Brien just tried to told, that NOBODY he knows will be
REALLY impacted by performance loss, caused due dynamic /bin/sh linking.
You will... So, because Duncan Barclay is impacted by performance
loss due dynamic /bin/sh linking, ENTIRE FreeBSD community will have
troubles (at least with NSS) due to static linking...

   Sincerely, Maxim M. Kazachek
   mailto:[EMAIL PROTECTED]
   mailto:[EMAIL PROTECTED]

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-23 Thread David Wolfskill
Date: Mon, 24 Nov 2003 09:34:08 +0600 (NOVT)
From: Maxim M. Kazachek [EMAIL PROTECTED]

   So, imagine, i'm accidentally deleted /bin with your most wanted
static sh... And, of course, due to static nature of /bin/sh it was
removed from /rescue? Nothing will protect you from shooting in the leg,
neither static linking, nor assumption that /lib is OK.

So go ahead and make /bin/sh also have a (hard) link to /rescue/sh.
Then the referenced action merely decrements the link count, and the
executable itself doesn't go away.

Sure, you could be more imaginative with foot-shooting, but the stated
problem is really easy to avoid.

Peace,
david   (reluctant to contribute to this thread)
-- 
David H. Wolfskill  [EMAIL PROTECTED]
If you want true virus-protection for your PC, install a non-Microsoft OS
on it.  Plausible candidates include FreeBSD, Linux, NetBSD, OpenBSD, and
Solaris (in alphabetical order).
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-23 Thread Maxim M. Kazachek
On Sun, 23 Nov 2003, David Wolfskill wrote:

Date: Mon, 24 Nov 2003 09:34:08 +0600 (NOVT)
From: Maxim M. Kazachek [EMAIL PROTECTED]

  So, imagine, i'm accidentally deleted /bin with your most wanted
static sh... And, of course, due to static nature of /bin/sh it was
removed from /rescue? Nothing will protect you from shooting in the leg,
neither static linking, nor assumption that /lib is OK.

So go ahead and make /bin/sh also have a (hard) link to /rescue/sh.
Then the referenced action merely decrements the link count, and the
executable itself doesn't go away.

Sure, you could be more imaginative with foot-shooting, but the stated
problem is really easy to avoid.

Peace,
david   (reluctant to contribute to this thread)
--
David H. Wolfskill [EMAIL PROTECTED]
If you want true virus-protection for your PC, install a non-Microsoft OS
on it.  Plausible candidates include FreeBSD, Linux, NetBSD, OpenBSD, and
Solaris (in alphabetical order).

I'm a real masochist, and removed /rescue/sh /sbin/sh whatsoever.
And after that will begin complain, that FreeBSD is ugly, non bullet-proof
operating system... :-)
But, in fact, as I can remember, all /rescue stuff is one
hadlinked executable. I don't think that hardlinking /bin/sh into /rescue
would be nice idea... IMHO it's not clear.

   Sincerely, Maxim M. Kazachek
   mailto:[EMAIL PROTECTED]
   mailto:[EMAIL PROTECTED]

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-22 Thread Dimitry Andric
On 2003-11-22 at 00:39:45 Tim Kientzle wrote:

 Right now, /sbin/init is statically linked.

Not here... I've built everything with WITH_DYNAMICROOT since the time
the option was introduced, and as such:

# file /sbin/init
/sbin/init: ELF 32-bit LSB executable, Intel 80386, version 1 (FreeBSD), for FreeBSD 
5.0.1, dynamically linked (uses shared libs), stripped
# ldd /sbin/init
/sbin/init:
libutil.so.3 = /lib/libutil.so.3 (0x28074000)
libcrypt.so.2 = /lib/libcrypt.so.2 (0x2807f000)
libc.so.5 = /lib/libc.so.5 (0x28097000)

In fact, the only statically linked executable I can currently find in
my base system (= -CURRENT as of 2003-11-11) is /sbin/devd...


pgp0.pgp
Description: PGP signature


Re: Unfortunate dynamic linking for everything

2003-11-22 Thread Richard Coleman
Garrett Wollman wrote:

You forgot:

* Allow statically-linked programs to use dynamic NSS modules
  by forking a (dynamically-linked) resolver process when
  needed.
This leads to a related, but widely disparaged option:

* Have a persistent NSS caching daemon with an RPC interface
  that all programs can access for NSS lookups.  You might
  call such a program `nscd'.  (Might as well be honest about
  it.)
Both of these options may incidentally help to resolve threading
issues in the C library (although that would not be the preferred way
of doing so).
Regardless of how NSS is implemented, it will be useful to have some 
type of caching (even if optional).  On a large system, you can beat the 
hell out of your LDAP server otherwise.  Of course, you can use a local 
replica of your LDAP server.  But that doesn't help if are using a 
different database like Postgres as the backend for your nss/pam setup.

But if a nscd (or equivalent) is added to FreeBSD, we need to do a 
better job than the Linux nscd.  We had real problems with the Linux 
nscd in a previous project.  If I remember, it assumes that the mapping 
between username - uid was 1-1.  We added an attribute to our LDAP 
schema so we could specify the reverse mapping in situations where more 
than one username mapped to the same uid.  But we could never get it to 
work correctly, since Linux nscd made some assumption that were 
difficult to change.

Richard Coleman
[EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-22 Thread Michael Edenfield
* Tim Kientzle [EMAIL PROTECTED] [031121 18:40]:
 Leo Bicknell wrote:
 To boot a machine into single user mode you need a kernel, init,
 and /bin/sh (minimally).  It would seem to me that alone is a good
 argument for those three things to be static.

  * Rewrite dlopen() to not require dynamic linking.
 
There were some patches for this submitted at one point.
As I recall, the people who looked at them were not entirely
comfortable with them.  (I'd be concerned about version
conflict problems with this approach:  what happens when
a dynamically-loaded NSS module refers to a libc function?
Does that get resolved to the already statically-linked
version?  Or does another copy of libc get dynamically linked
with potential version conflicts?  Does anyone know?)
 
I personally think this is worth researching, though I
have my doubts.

I took a look at the glibc implementation of dlopen() breifly, since
that does function from within libc.a.  It appears that you *do* get
more than one loaded copy of libc.  The copy of dlopen() that is built
when #ifndef SHARED includes a flag: __libc_multiple_libcs that is set
to 1.

Additionally, I was reading comments from some of the glibc developers
who basically claim that dlopen() in a static binary *only* works if you
dlopen() a NSS module.  It isn't guaranteed to work in the general case
because the static binary has no DYNAMIC elf section to resolve external
references etc.  I suspect this means NSS modules are limited in what
they are allowed to reference and still work?  I haven't looked in much
detail on their implementation but it certainly looks like a hack just
to make NSS work, which I don't think is a good long-term solution.

--Mike



pgp0.pgp
Description: PGP signature


Re: Unfortunate dynamic linking for everything

2003-11-21 Thread Peter Jeremy
On Thu, Nov 20, 2003 at 09:29:30PM -0500, Richard Coleman wrote:
But I've often wondered how frequently a production system has such 
problems.  I've been a sysadmin for many years and can't remember this 
ever happening.  It's much more common to blow a hard drive, or have 
flaky memory, etc.

We've had a customer whose system administrator installed a root
cronjob including rm -r / somedirectory/tmp (note the space) - on
all four production machines as well as their model system.  (The
person in question is a classic example of a little knowledge is a
dangerous thing).  The console logs (which were on another unaffected
system) made interesting reading - he had spent a significant amount
of time attempting to reboot the systems before calling for help.

Peter
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-21 Thread Peter Jeremy
On Thu, Nov 20, 2003 at 09:51:48PM +0100, boyd, rounin wrote:
From: Peter Jeremy [EMAIL PROTECTED]
 Shouldn't that be 'chmod +t /bin/sh' ???
 
 Definitely.  Why waste a new bit when there's already a perfectly good
 one that is (or was) defined for the purpose.

the 't' bit was known as the 'sticky' bit to keep frequently used,
sharable (judged by a human) text segments in core.  since then
that bit has been overloaded to death.

The purpose for setting the 't' bit was to speed up exec'ing of
frequently used executables by avoiding the need to load the text
segment from the filesystem.  This is exactly what the suggested
pre-linking achieves.

As for overloading the 't' bit, I don't believe it's ever been used
for anything else on executable files.  I agree it grew a distinct
meaning for directories but the 'x' bits have always meant different
things on files and directories and the setgid bit means different
things on executable and non-executable files.

shared libraries have always been a mistake.

That is a matter of opinion.  Shared libraries have both advantages
and disadvantages.  Statically linking /usr/X11R6/bin demonstrates
one advantage of shared libraries.

Peter
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-21 Thread boyd, rounin
From: Peter Jeremy [EMAIL PROTECTED]
 As for overloading the 't' bit, I don't believe it's ever been used
 for anything else on executable files.

directories

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-21 Thread othermark
I got bit by this just two days ago.  I have one machine that tracks
-current.  It upgraded to DYNAMICROOT just fine.  I nfs mounted /usr/src
and /usr/obj on another 5.0-release machine made the necessary adjustments,
installed the kernel, rebooted, remounted and began make installworld.  It
failed.  The only thing that saved me was being able to nfs mount /rescue
from the -current machine.  I then selectively copied stuff from /usr/obj
into the respective places to get various 'make install' stuff to work, in
order to get to a point where I could re-run make installworld.

THANKFULLY mount was statically linked (and make installworld had not yet
overwritten it).  Yes I still could have recovered this using a cd/floppy
(neither of which this headless machine has), but it would have been a
serious PITA.

-- 
othermark
atkin901 at nospam dot yahoo dot com
(!wired)?(coffee++):(wired);

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-21 Thread Bruce M Simpson
On Thu, Nov 20, 2003 at 09:29:30PM -0500, Richard Coleman wrote:
 But I've often wondered how frequently a production system has such 
 problems.  I've been a sysadmin for many years and can't remember this 
 ever happening.  It's much more common to blow a hard drive, or have 
 flaky memory, etc.

During my time in an investment bank, installations were usually hosed
in this way by human error (systems administrators removing a file by
accident, etc) or by third party package installation. The OS in
question was Solaris.

BMS
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-21 Thread Garance A Drosihn
At 8:52 PM +1100 11/20/03, Peter Jeremy wrote:
On Wed, Nov 19, 2003, Lyndon Nerenberg wrote:
 --On Wed, Nov 19, 2003, Garance A Drosihn [EMAIL PROTECTED] wrote:
  
  have a:  chflags ldcache /bin/sh
 
Shouldn't that be 'chmod +t /bin/sh' ???
Definitely.  Why waste a new bit when there's already a
perfectly good one that is (or was) defined for the purpose.
Two reasons I didn't suggest that:

a) Well, actually it never occurred to me...

b) I thought that you might want to have this an admin-only
   command, so nefarious users couldn't abuse it on a shared
   system.
I'm certainly fine with it using +t.  Now it's just a matter
of figuring out how to implement it...  :-)
--
Garance Alistair Drosehn=   [EMAIL PROTECTED]
Senior Systems Programmer   or  [EMAIL PROTECTED]
Rensselaer Polytechnic Instituteor  [EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-21 Thread Tim Kientzle
Leo Bicknell wrote:
To boot a machine into single user mode you need a kernel, init,
and /bin/sh (minimally).  It would seem to me that alone is a good
argument for those three things to be static.
You need a static shell, yes.  That does not have to be /bin/sh.
init does prompt, and /rescue/sh is (and always will be) static.
As I pointed out earlier, some of the heat here comes
from the fact that /bin/sh is currently overloaded:
 * It is the default system script interpreter, used
   by the rc scripts and many other things.  As such,
   it must start quickly.
 * It is the default user shell for many users.  As such,
   it must support NSS.
So far, I haven't seen anyone in this thread seriously
argue against either of these points.
Today, these two issues are somewhat contradictory.  Right now,
NSS requires dlopen() which requires dynamic linking which
negatively impacts startup.
There have been a lot of proposed solutions:

 * Rewrite NSS to not require dlopen().

   A lot of people have suggested this, but Jacques
   made an important point about long-term support.
   In short, this would probably just gaurantee that our NSS
   is perpetually out-of-date, since it would become a major
   headache to port existing NSS modules.
 * Rewrite dlopen() to not require dynamic linking.

   There were some patches for this submitted at one point.
   As I recall, the people who looked at them were not entirely
   comfortable with them.  (I'd be concerned about version
   conflict problems with this approach:  what happens when
   a dynamically-loaded NSS module refers to a libc function?
   Does that get resolved to the already statically-linked
   version?  Or does another copy of libc get dynamically linked
   with potential version conflicts?  Does anyone know?)
   I personally think this is worth researching, though I
   have my doubts.
 * Don't support NSS in /bin/sh.

   This essentially amounts to: Don't use /bin/sh as
   an end-user shell.  Given the number of commits to
   improve command-line usage of /bin/sh, I think it's clear
   that a lot of people do rely on /bin/sh as a user shell.
 * Change the default script interpreter for rc and such.

   (e.g., use a statically-linked, optimized /sbin/sh for
   that purpose.)  Again, not particularly attractive, and
   it certainly breaks a lot of administrator's assumptions.
 * Make dynamic linking faster.

   As many people have pointed out, FreeBSD's dynamic linking is
   slower than it should be.  Advances here would benefit Mozilla,
   OpenOffice, KDE, Gnome, Apache, Perl, bash, TCL, Python, PHP,
   etc.  (I believe these are all slower to start on FreeBSD
   than on Linux, for example.)  This is a major end-user
   experience issue that I find much more compelling than
   anything mentioned so far.  I personally find OpenOffice only
   barely usable on FreeBSD, in large part because it takes so
   long to open.
   I suspect that most of our users would gladly give up a few
   seconds booting (or a few minutes from portupgrade or buildworld)
   if they could have Mozilla and OpenOffice open a few seconds faster.
   Benchmarks do not necessarily reflect user experience.
As many people have pointed out, there are other advantages and
disadvantages to dynamic linking:  library upgrades are simplified, disk
requirements are reduced, memory usage is less efficient, system upgrade
and repair is more complicated, etc., but I think the above summarizes
the issues that people seem to really care about.
Tim

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-21 Thread Tim Kientzle
Leo Bicknell wrote:
The more I think about init the more I don't like dynamic linking for
it.  init needs to have as few failure modes as possible.  I do still
think it's fine for all the other /bin and /sbin things.
Right now, /sbin/init is statically linked.

Tim Kientzle



___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-21 Thread Garrett Wollman
On Fri, 21 Nov 2003 15:38:49 -0800, Tim Kientzle [EMAIL PROTECTED] said:


 There have been a lot of proposed solutions:

   * Rewrite NSS to not require dlopen().
   * Rewrite dlopen() to not require dynamic linking.
   * Don't support NSS in /bin/sh.
   * Change the default script interpreter for rc and such.
   * Make dynamic linking faster.

You forgot:

* Allow statically-linked programs to use dynamic NSS modules
  by forking a (dynamically-linked) resolver process when
  needed.

This leads to a related, but widely disparaged option:

* Have a persistent NSS caching daemon with an RPC interface
  that all programs can access for NSS lookups.  You might
  call such a program `nscd'.  (Might as well be honest about
  it.)

Both of these options may incidentally help to resolve threading
issues in the C library (although that would not be the preferred way
of doing so).

-GAWollman

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-21 Thread Tim Kientzle
Bruce Evans wrote:
It obviously uses NSS.  How else could it be so bloated? :

$ ls -l /sbin/init
-r-x--  1 root  wheel  453348 Nov 18 10:30 /sbin/init
I believe it's actually DNS, not NSS.

Pre-5.0, the resolver ballooned significantly.
A lot of the bloat in /bin and /sbin came
from the NIS functions which in turn pull in
the resolver.
Example: /bin/date on 5.1 is also over 450k
because of a single call to getservbyname().
Removing that one call shrinks a static /bin/date
to a quite reasonable size. (I seem to recall 80k when
I did this experiment.)
I note that /sbin/init calls getpwnam();
I expect that's where the bloat gets pulled in.
Tim Kientzle

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-21 Thread Tim Kientzle
Garrett Wollman wrote:
On Fri, 21 Nov 2003 15:38:49 -0800, Tim Kientzle [EMAIL PROTECTED] said:
There have been a lot of proposed solutions:
 * Rewrite NSS to not require dlopen().
 * Rewrite dlopen() to not require dynamic linking.
 * Don't support NSS in /bin/sh.
 * Change the default script interpreter for rc and such.
 * Make dynamic linking faster.
You forgot:

* Allow statically-linked programs to use dynamic NSS modules
  by forking a (dynamically-linked) resolver process when
  needed.
This leads to a related, but widely disparaged option:

* Have a persistent NSS caching daemon with an RPC interface
  that all programs can access for NSS lookups.  You might
  call such a program `nscd'.  (Might as well be honest about
  it.)
Jacques seems to be of the opinion that the current
'nscd' is not up to the task.  He would know better than I.
Tim

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-21 Thread Bruce Evans
On Fri, 21 Nov 2003, Tim Kientzle wrote:

 Bruce Evans wrote:
  It obviously uses NSS.  How else could it be so bloated? :
 
  $ ls -l /sbin/init
  -r-x--  1 root  wheel  453348 Nov 18 10:30 /sbin/init

 I believe it's actually DNS, not NSS.

 Pre-5.0, the resolver ballooned significantly.
 A lot of the bloat in /bin and /sbin came
 from the NIS functions which in turn pull in
 the resolver.

Perhaps both.

 Example: /bin/date on 5.1 is also over 450k
 because of a single call to getservbyname().
 Removing that one call shrinks a static /bin/date
 to a quite reasonable size. (I seem to recall 80k when
 I did this experiment.)

The 2 calls to logwtmp() must also be removed, at least now.
I get the following text sizes: for /bin/date:

RELENG_4: 137491
-current*: 93214 (* = getservbyname() and logwtmp() calls removed)
-current: 371226 (only 412492 total, not 450K yet)

 I note that /sbin/init calls getpwnam();
 I expect that's where the bloat gets pulled in.

Yes, except it's only the latest 200+K of bloat (from 413558 bytes text
to 633390).  Before that there was 100+K of miscellaneous bloat
relative to RELENG_4 (text size 305289 there).  Before that there was
another 200+K of bloat from implementing history.  Compiling with
-DNO_HISTORY removes history support and reduces the text size to
162538 (this is without getpwnam()).  Then there is another 30K of
mostly non-bloat for actual changes within /bin/sh, since compiling
the FreeBSD-1 /bin/sh with current libraries gives a text size of
132966.  Finally, IIRC the text size of the FReeeBSD-1 /bin/sh is
70K (total size 90K), so there is another 60K of miscellaneous bloat
in current libraries to increase the text size from 70K to 130K.

Total text sizes for /bin/sh's internals:
FreeBSD-1 sh compiled with -current's compiler: 55350
current sh compiled with -current's compiler: 87779
87:55 is about right for the increased functionality.

Bruce
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-21 Thread boyd, rounin
From: William Josephson [EMAIL PROTECTED]
 People at Berkeley (and elsewhere) have done user studies to try to
 quantify this sort of thing.  It is pretty clear that with modern
 hardware, most failures are due to human error.  That's not to say
 that hardware and software faults aren't real problems, too, but it
 is more common that someone, say, pulls the wrong drive from the
 RAID-5 array, resulting in an unnecessary double disk fault.

that means your raid 5 is bust.  i've seen raid 5 fail and it just picks
another disk in the 'free' pool like nothing has happened.

a study?  it's bleeding obvious.

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-21 Thread Peter Jeremy
On Fri, Nov 21, 2003 at 04:52:00PM -0500, Garance A Drosihn wrote:
Shouldn't that be 'chmod +t /bin/sh' ???

b) I thought that you might want to have this an admin-only
   command, so nefarious users couldn't abuse it on a shared
   system.

I would make one change to your proposal:  Instead of having the
pre-linked image wired in RAM, I'd allow it to be paged as for an
ordinary process image.  This means the overhead of starting a
pre-linked process would be similar to starting a statically-linked
executable.

Once you remove the store image in RAM only requirement, I don't see
that allowing an ordinary user to set the sticky bit would allow any
new attacks.  The user can only set the bit on their own files.  The
address space overhead of holding a pre-linked image is roughly the
same as executing that image so allowing a user to create sticky
executables is roughly the same as allowing the user to create an
additional process running that executable.

I'm certainly fine with it using +t.  Now it's just a matter
of figuring out how to implement it...  :-)

I thought it was obvious so I left it as an exercise for the reader :-).

My guess is that if a pre-linked image didn't exist, imgact_elf()
would run as normal, transferring control to ld-elf.so.  ld-elf.so
would do symbol binding as per normal (probably as if with LD_BIND_NOW
was set) and would then invoke a new save_image() syscall just before
it transferred control to main().  This would trigger the kernel to
save the whole process image as COW.  This image would be associated
back to the original executable using a mechanism similar to that used
for sharing the text segment.  Next time that executable is execve()'d,
imgact_elf() would note that there was a pre-linked image available and
attach to it, rather than re-mapping the executable.  It would pass a
flag to ld-elf.so to inform the latter that it should just start main().

A few rough edges still need to be resolved:
- ld-elf.so runs as the invoking user but the above model has it telling
  the kernel what the process image is - which shouldn't be trusted by a
  different uid.  The easiest solution is to make save_image() a root-
  only syscall so pre-binding only works if the executable has been run
  by root.  An alternative would be to have the process images tied into
  the user's credentials - this would mean more significant changes
  because there's nothing that currently maintains information about
  an executable on a per-uid basis outside the process.
- ld-elf.so environment (eg LD_LIBRARY_PATH, LD_PRELOAD) impact.  This
  gets messy - it is undesirable to totally inhibit pre-linking if
  either is set so maybe ld-elf.so needs to verify that the pre-linked
  image matches the result if the environment is taken into account.
- ld-elf.so needs to make sure that the libraries haven't changed since
  the original image was mapped - to handle (eg) libc.so being replaced.
- There needs to be some way for the VM system to throw away the
  pre-linked images if the system runs low on RAM+swap.

An alternative approach might be to have a dummy 'placeholder' process
which solely exists to hold the pre-linked image.  This process would
be started by invoking the executable with a special LD_PRELINK_ONLY
flag.  When the executable was later exec'd, the kernel would fork() the
dummy process giving it the credentials, pid and ppid of the process
being exec'd.

I'm sure deeper thought will reveal problems...

Peter
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-20 Thread Stijn Hoop
On Wed, Nov 19, 2003 at 09:27:55PM -0800, Tim Kientzle wrote:
 Richard Coleman wrote:
 It seems /bin/sh is the real sticking point. 
 
 There is a problem here: Unix systems have historically used
 /bin/sh for two somewhat contradictory purposes:
   * the system script interpreter
   * as a user shell
 
 The user shell must be dynamically linked in order
 to support centralized administration.  I personally
 see no way around that.  Given that many users do
 rely on /bin/sh, it seems that /bin/sh must be
 dynamically linked.
 
 There are good reasons to want the system script
 interpreter statically linked.
 
 Maybe it's time to separate these two functions?
 I would be content to have a static /sbin/sh
 that is used as the system script interpreter for
 rc scripts, etc.

And /usr/bin/sh as a user shell?

--Stijn

-- 
I'm not under the alkafluence of inkahol that some thinkle peep I am.  It's
just the drunker I sit here the longer I get.


pgp0.pgp
Description: PGP signature


Re: Unfortunate dynamic linking for everything

2003-11-20 Thread Peter Jeremy
On Wed, Nov 19, 2003 at 11:18:47AM -0700, Lyndon Nerenberg wrote:
--On Wednesday, November 19, 2003 12:30 AM -0500 Garance A Drosihn 
[EMAIL PROTECTED] wrote:

have a:  chflags ldcache /bin/sh

Shouldn't that be 'chmod +t /bin/sh' ???

Definitely.  Why waste a new bit when there's already a perfectly good
one that is (or was) defined for the purpose.

As for the implementation, I presume the desired behaviour would be to
snapshot the process image (make it copy-on-write) at the point where
ld-elf.so invokes main().  You'd probably want LD_BIND_NOW behaviour
to minimise the runtime overheads.  I don't see any need for this to
need massive amounts of RAM - there's no reason why the snapshot
couldn't be paged normally.

I think this would be a big win compared to what we have now - the
full benefits of dynamic linking remain and most of the run-time
binding overheads are removed.

Of course it's not perfect.  The snapshot image permanently occupies
virtual space (RAM/swap).  And there's still the PIC overhead -
especially on register-starved architectures like the i386.

I wonder how difficult this would be to implement in userland only?

Peter
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-20 Thread Tony Finch
Jacques A. Vidrine [EMAIL PROTECTED] wrote:

Finally, if we could call `dlopen' from statically-linked binaries,
this wouldn't be an issue.

One of the performance problems that John Dyson mentioned (the sparse
dirtying of libc's data section) would still remain, because the whole
of libc has to be linked into programs that use NSS.

Tony.
-- 
f.a.n.finch  [EMAIL PROTECTED]  http://dotat.at/
THE WASH TO NORTH FORELAND: SOUTHWEST 4 OR 5, PERHAPS 6 FOR A TIME IN EVENING
IN THAMES, VEERS NORTH TO NORTHEAST 2 OR 3 FROM NORTH, BACKS NORTHWEST LATER.
OCCASIONAL RAIN, CLEARS ERRATICALLY FROM NORTH, EXCEPT FAR SOUTH. MODERATE OR
GOOD, LOCALLY POOR IN SOUTH. SLIGHT , LOCALLY MODERATE IN SOUTH.
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-20 Thread Tim Kientzle
Stijn Hoop wrote:
On Wed, Nov 19, 2003 at 09:27:55PM -0800, Tim Kientzle wrote:
Maybe it's time to separate these two functions?
I would be content to have a static /sbin/sh
that is used as the system script interpreter for
rc scripts, etc.


And /usr/bin/sh as a user shell?
I was thinking /bin/sh for the user shell
and /sbin/sh for the system script
interpreter.
There must be a /bin/sh and given that
there are many user scripts that rely on
it, it seems that it must be a user
shell.
Tim Kientzle

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-20 Thread Dimitry Andric
On 2003-11-20 at 21:51:48 boyd, rounin wrote:

 think about a dynamically linked init(8) ...

% sudo ldd /sbin/init
/sbin/init:
libutil.so.3 = /lib/libutil.so.3 (0x28074000)
libcrypt.so.2 = /lib/libcrypt.so.2 (0x2807f000)
libc.so.5 = /lib/libc.so.5 (0x28097000)

Yes, working fine here. What should the problem be?


pgp0.pgp
Description: PGP signature


Re: Unfortunate dynamic linking for everything

2003-11-20 Thread boyd, rounin
From: Dimitry Andric [EMAIL PROTECTED]

% sudo ldd /sbin/init
/sbin/init:
libutil.so.3 = /lib/libutil.so.3 (0x28074000)
libcrypt.so.2 = /lib/libcrypt.so.2 (0x2807f000)
libc.so.5 = /lib/libc.so.5 (0x28097000)

Yes, working fine here. What should the problem be?

the day /lib gets smashed.

you're building a house of cards.  once, if /etc/init and
/bin/sh and some other pieces where in place a smashed
file-system could be easily fixed.  now you have to have
3 shared libs and a viable /lib.

do you want systems that work?  or houses of cards?

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-20 Thread Christopher Vance
On Fri, Nov 21, 2003 at 12:23:16AM +0100, boyd, rounin wrote:
you're building a house of cards.  once, if /etc/init and
/bin/sh and some other pieces where in place a smashed
file-system could be easily fixed.  now you have to have
3 shared libs and a viable /lib.
do you want systems that work?  or houses of cards?
Personally, I think init should be static, and can't think of any way
it would benefit from shared libraries.  I'm not qualified to comment
on the various things people have said about /bin/sh.
(Possibly irrelevant data point: Solaris 10, if it ever flies, will
supposedly have only shared libraries.)
Given that you've got a knob if you really care enough to change the
default, static init and /rescue should be adequate to get past all
the other bickering here, so please stop it already...
--
Christopher Vance
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-20 Thread boyd, rounin
From: Christopher Vance [EMAIL PROTECTED]
 Personally, I think init should be static, and can't think of any way
 it would benefit from shared libraries.

plan 9 has everything static.  the kernel compiles in about 20 seconds
or less -- no compression -- and you can boot it off a floppy.

if i can sit in /sys/src and type:

mk install

and have everything re-built (and i could do it for all the supported
architectures) in minutes i have eliminated unnecessary complexity.

if it's not there, it can't break.


btw: say hi to maltby for me.

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-20 Thread Steve Kargl
On Fri, Nov 21, 2003 at 12:23:16AM +0100, boyd, rounin wrote:
 From: Dimitry Andric [EMAIL PROTECTED]
 
 % sudo ldd /sbin/init
 /sbin/init:
 libutil.so.3 = /lib/libutil.so.3 (0x28074000)
 libcrypt.so.2 = /lib/libcrypt.so.2 (0x2807f000)
 libc.so.5 = /lib/libc.so.5 (0x28097000)
 
 Yes, working fine here. What should the problem be?
 
 the day /lib gets smashed.
 

This is only marginally different than /sbin/init
getting smashed.  If the root partition develops
problems, you need to restore for tape.

-- 
Steve
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-20 Thread boyd, rounin
From: Steve Kargl [EMAIL PROTECTED]
 This is only marginally different than /sbin/init
 getting smashed.  If the root partition develops
 problems, you need to restore for tape.

tape?  who uses tape?  optical, my son.

brother, can you spare a TU-16?

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-20 Thread M. Warner Losh
In message: [EMAIL PROTECTED]
boyd, rounin [EMAIL PROTECTED] writes:
: Yes, working fine here. What should the problem be?
: 
: the day /lib gets smashed.

/rescue/sh

Warner
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-20 Thread Richard Coleman
boyd, rounin wrote:

From: Dimitry Andric [EMAIL PROTECTED]

% sudo ldd /sbin/init
/sbin/init:
libutil.so.3 = /lib/libutil.so.3 (0x28074000)
libcrypt.so.2 = /lib/libcrypt.so.2 (0x2807f000)
libc.so.5 = /lib/libc.so.5 (0x28097000)
Yes, working fine here. What should the problem be?

the day /lib gets smashed.

you're building a house of cards.  once, if /etc/init and
/bin/sh and some other pieces where in place a smashed
file-system could be easily fixed.  now you have to have
3 shared libs and a viable /lib.
do you want systems that work?  or houses of cards?
I would prefer to solve this problem using a fixit floppy or cdrom 
anyways.  I don't think that creates a house of cards.  My systems work 
just fine.

But I've often wondered how frequently a production system has such 
problems.  I've been a sysadmin for many years and can't remember this 
ever happening.  It's much more common to blow a hard drive, or have 
flaky memory, etc.

Richard Coleman
[EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-20 Thread Richard Coleman
boyd, rounin wrote:

From: Christopher Vance [EMAIL PROTECTED]

Personally, I think init should be static, and can't think of any way
it would benefit from shared libraries.


plan 9 has everything static.  the kernel compiles in about 20 seconds
or less -- no compression -- and you can boot it off a floppy.
if i can sit in /sys/src and type:

mk install

and have everything re-built (and i could do it for all the supported
architectures) in minutes i have eliminated unnecessary complexity.
if it's not there, it can't break.

btw: say hi to maltby for me.
plan9 doesn't count.  It's so minimalistic, it's useless.  It has many 
beautiful and brilliant ideas.  But it's not useful to many people as a 
production system.  It's a shame, really.

Richard Coleman
[EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-20 Thread boyd, rounin
 plan9 doesn't count.  It's so minimalistic, it's useless.

well, try to think about non-minimalism when you're trying to track
down a kernel bug in a zillion SLOC ...

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-20 Thread Garrett Wollman
On Fri, 21 Nov 2003 03:37:20 +0100, boyd, rounin [EMAIL PROTECTED] said:

 well, try to think about non-minimalism when you're trying to track
 down a kernel bug in a zillion SLOC ...

How about trying to think about FreeBSD when posting on the FreeBSD
mailing-lists.

-GAWollman

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-20 Thread boyd, rounin
From: Bruce M Simpson [EMAIL PROTECTED]
 During my time in an investment bank, installations were usually hosed
 in this way by human error (systems administrators removing a file by
 accident, etc) ...

yup, it's rare i've seen flakey h/w.  but i do remember one sysadmin
(when i was a contract sysadmin) who on day 2 chown'd the whole
source tree to himself on a development m/c.  ugly.  there were
backups but 'that would be too costly [in time]' to do a clean restore.

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-20 Thread Cy Schubert
In message [EMAIL PROTECTED], boyd, rounin 
write
s:
 From: Bruce M Simpson [EMAIL PROTECTED]
  During my time in an investment bank, installations were usually hosed
  in this way by human error (systems administrators removing a file by
  accident, etc) ...
 
 yup, it's rare i've seen flakey h/w.  but i do remember one sysadmin
 (when i was a contract sysadmin) who on day 2 chown'd the whole
 source tree to himself on a development m/c.  ugly.  there were
 backups but 'that would be too costly [in time]' to do a clean restore.

I've seen that too. An end user got permission from management to the root 
account, e.g. management ordered us to give her the root pw on a Tru64 box. 
She chowned every file on the system to herself. Very ugly indeed.


Cheers,
--
Cy Schubert [EMAIL PROTECTED]http://www.komquats.com/
BC Government .   FreeBSD UNIX
[EMAIL PROTECTED] . [EMAIL PROTECTED]
http://www.gov.bc.ca/ .http://www.FreeBSD.org/


___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-19 Thread Gunther Nikl
On Tue, Nov 18, 2003 at 06:26:21PM -0800, Matthew Dillon wrote:
 
 :Our rationale for encouraging Gordon is as follows:
 :
 :1.  4.x upgrade path:  As we approach 5-STABLE, a lot of users might want
 :to upgrade from 4-STABLE.  Historically in 4.x, the / partition has
 :been very modest in size.  One just simply cannot cram the bloat that
 :has grown in 5.x into a 4.x partition scheme.  Of course there is the
 :venerable 'dump - clean install - restore' scheme, but we were looking
 :for something a little more user-friendly.
 
 This argument would apply to very old 4.x users but not to anyone who
 installed it as of March 2001.  I bumped the nominal size of the
 root partition to 128MB in 1.98.2.7 of sysinstall/label.c.

  Don't you think that people are able to change defaults if they think
  thats appropriate?

 Prior to that Jordan had bumped the root partition size to 100MB
 in 1.98.2.3 in March 2001.  It was 50MB before then, which is too
 small even for 4.x.

  Hm, then why do I have still room on my 50MB root partition?

$ df
Filesystem 1K-blocksUsed   Avail Capacity  Mounted on
/dev/vinum/root49583   29040   1657764%/

  Thats enough for installworld (if softupdates are turned off)

  Gunther
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-19 Thread Jan Grant
On Tue, 18 Nov 2003 [EMAIL PROTECTED] wrote:

 Garrett Wollman said:
  On Tue, 18 Nov 2003 18:07:32 -0500 (EST), [EMAIL PROTECTED] said:
 
   If the object is to maximally 'share',
 
  The object, AIUI, is for ~username expansion to work in the shells
  when the user stored somewhere defined by an external NSS module.  I
  don't believe that there is anything else in a (sane) shell that
  cares.
 
 It is a fallacy that EVERYTHING (e.g. libc) needs to be dynamically
 linked to support that feature.  A single, self contained library
 that might even be sparse WRT memory isn't going to hurt much.

It's a fallacy that ANYTHING (except nscd/lookupd) needs to be
dynamically linked to support that feature :-)

-- 
jan grant, ILRT, University of Bristol. http://www.ilrt.bris.ac.uk/
Tel +44(0)117 9287088 Fax +44 (0)117 9287112 http://ioctl.org/jan/
User interface? I hardly know 'er!
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-19 Thread Jacques A. Vidrine
[cc: dropped]

I suppose I should comment on this thread, since I'm closely related
to at least two of the rationales mentioned for moving towards an
all-dynamically-linked system.  (I would prefer to stay out of this
thread.  In my mind we've had all these arguments in various
forums months ago and the issue was put to rest.)

On Tue, Nov 18, 2003 at 05:06:06PM -0700, Scott Long wrote:
 2.  NSS support out-of-the-box:  Again, this is a user-experience issue
 in that forcing NSS users to recompile world was seen as a potential
 roadblock to them.

Some followups mentioned that a different implementation of NSS would
not require dynamically linked binaries.  This is technically true.
Certainly, I explored that avenue.  But practical considerations made
that alternative a poor choice.  Supporting NSS really also means
supporting (in as much as possible) the existing base of NSS modules.
These modules were all designed to be loaded by libc and invoked
in-process. (nscd doesn't do what some seem to think it does, and in
practice it is not well-loved.)

Existing NSS modules also shaped some other decisions that I made
(such as API entry points and thread awareness or lack thereof).
Breaking with the fundamental designs of NSS as found on Solaris and
Linux means practically starting from scratch when `porting' NSS
modules.

I'm sure someone more clever and with more time could make it work
out-of-process.  Based on my experience, I think the result would be
overengineered. *shrug*

Finally, if we could call `dlopen' from statically-linked binaries,
this wouldn't be an issue.  I'd really like to see dlopen support for
statically-linked binaries, for NSS and many other applications.


 3.  Binary security updates: there is a lot of interest in providing a
 binary update mechanism for doing security updates.  Having a dynamic
 root means that vulnerable libraries can be updated without having to
 update all of the static binaries that might use them.

Actually, not only binary security updates but any security updates,
or bug fixes for that matter.  If there is a bug in a library, fixing
that bug on your system is much more straightforward if everything is
dynamically linked.  It is much easier to rebuild libc or libcrypto
and restart applications then to have to go through an entire `make
world'.  It can be hard for many administrators to avoid `make world',
because it is not always easy to ascertain what applications are using
what APIs and libraries.


There's been a lot of talk about performance, but for my environment
all the workhorse applications are already dynamically linked.  I'd
guess that is the case for most FreeBSD users.  And of course, most
FreeBSD binaries--- in the base system, in ports, and commercially
available--- are already dynamically linked.  For the minority of
users that it actually has a performance impact (show of hands?), of
course they are sophisticated enough to build the affected binaries
statically.  Unless we are talking about /bin/sh, they probably already
have to go through special measures to get a statically linked binary.

Cheers,
-- 
Jacques Vidrine   NTT/Verio SME  FreeBSD UNIX   Heimdal
[EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-19 Thread Leo Bicknell
In a message written on Wed, Nov 19, 2003 at 08:10:59AM -0600, Jacques A. Vidrine 
wrote:
 statically.  Unless we are talking about /bin/sh, they probably already
 have to go through special measures to get a statically linked binary.

Something has been bothering me about the whole /bin/sh function,
and today when fixing a box I realized what it was.

To boot a machine into single user mode you need a kernel, init,
and /bin/sh (minimally).  It would seem to me that alone is a good
argument for those three things to be static.  Yes, I'm one of those
people who rebuilds kernels without modules.  The kernel problem
doesn't bother me as much, as you can always boot without loading
modules, so it's not really a part of the required kernel is
dynamic problem.

Outside of these issues I have no problem with a dynamic root and a
/rescue.

-- 
   Leo Bicknell - [EMAIL PROTECTED] - CCIE 3440
PGP keys at http://www.ufp.org/~bicknell/
Read TMBG List - [EMAIL PROTECTED], www.tmbg.org


pgp0.pgp
Description: PGP signature


Re: Unfortunate dynamic linking for everything

2003-11-19 Thread Ken Smith
On Wed, Nov 19, 2003 at 09:19:50AM -0500, Leo Bicknell wrote:

 To boot a machine into single user mode you need a kernel, init,
 and /bin/sh (minimally).

Roughly the same thing was bothering me last night.  You get a chance
to specify the shell when init is in the last phase of getting you to
single-user mode so you can say /rescue/sh at that point.  init is
another story and I asked someone about that, they said it either is
or will shortly be a loader option so you can override that to be
/rescue/init that way.

-- 
Ken Smith
- From there to here, from here to  |   [EMAIL PROTECTED]
  there, funny things are everywhere.   |
  - Theodore Geisel |
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-19 Thread Leo Bicknell
In a message written on Wed, Nov 19, 2003 at 09:25:35AM -0500, Ken Smith wrote:
 Roughly the same thing was bothering me last night.  You get a chance
 to specify the shell when init is in the last phase of getting you to
 single-user mode so you can say /rescue/sh at that point.  init is
 another story and I asked someone about that, they said it either is
 or will shortly be a loader option so you can override that to be
 /rescue/init that way.

Perhaps /rescue/sh should be the default when booting into single user.

The more I think about init the more I don't like dynamic linking for
it.  init needs to have as few failure modes as possible.  I do still
think it's fine for all the other /bin and /sbin things.

-- 
   Leo Bicknell - [EMAIL PROTECTED] - CCIE 3440
PGP keys at http://www.ufp.org/~bicknell/
Read TMBG List - [EMAIL PROTECTED], www.tmbg.org


pgp0.pgp
Description: PGP signature


Re: Unfortunate dynamic linking for everything

2003-11-19 Thread Matthew Dillon

:  Don't you think that people are able to change defaults if they think
:  thats appropriate?
:
: Prior to that Jordan had bumped the root partition size to 100MB
: in 1.98.2.3 in March 2001.  It was 50MB before then, which is too
: small even for 4.x.
:
:  Hm, then why do I have still room on my 50MB root partition?
:
:$ df
:Filesystem 1K-blocksUsed   Avail Capacity  Mounted on
:/dev/vinum/root49583   29040   1657764%/
:
:  Thats enough for installworld (if softupdates are turned off)
:
:  Gunther

Try running installkernel and see what happens when it tries to
rename the old kernel and modules and install a new one.  Or try
installing a kernel.debug (which is 14MB+) instead of a kernel.

The point here is that just because you can barely squeeze the system
into 50MB doesn't mean it's a good idea to.  It might work for a very
narrow use pattern, but it will create terrible problems if you ever
tried to expand your use and the system defaults have to cover more
then just generic users reasonably.

This is why the default is no longer 50MB.

-Matt
Matthew Dillon 
[EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-19 Thread Marcel Moolenaar
On Wed, Nov 19, 2003 at 09:25:35AM -0500, Ken Smith wrote:
 On Wed, Nov 19, 2003 at 09:19:50AM -0500, Leo Bicknell wrote:
 
  To boot a machine into single user mode you need a kernel, init,
  and /bin/sh (minimally).
 
 Roughly the same thing was bothering me last night.  You get a chance
 to specify the shell when init is in the last phase of getting you to
 single-user mode so you can say /rescue/sh at that point.  init is
 another story and I asked someone about that, they said it either is
 or will shortly be a loader option so you can override that to be
 /rescue/init that way.

set init_path=/rescue/init

It's rather non-intuitive. It works, but having a static /sbin/init
avoids having to muck around in the loader in order to get to the
rescue bits. If you need the rescue bits, you pretty much always
need to use /rescue/init anyway. A dynamicly linked /sbin/init just
makes it harder to get to the rescue bits, so it makes sense to
link init(8) staticly. Especially since there's no advantage to
dynamic linking init(8) that compensates for the inconvience.

-- 
 Marcel Moolenaar USPA: A-39004  [EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-19 Thread Lyndon Nerenberg
--On Wednesday, November 19, 2003 12:30 AM -0500 Garance A Drosihn 
[EMAIL PROTECTED] wrote:

have a:  chflags ldcache /bin/sh
Shouldn't that be 'chmod +t /bin/sh' ???

--lyndon
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-19 Thread Bruce Evans
On Wed, 19 Nov 2003, Marcel Moolenaar wrote:

 set init_path=/rescue/init

If dynamic root were ready to be turned on, then /rescue/init would be
in the default init_path.

 A dynamicly linked /sbin/init just
 makes it harder to get to the rescue bits, so it makes sense to
 link init(8) staticly. Especially since there's no advantage to
 dynamic linking init(8) that compensates for the inconvience.

It obviously uses NSS.  How else could it be so bloated? :

$ ls -l /sbin/init
-r-x--  1 root  wheel  453348 Nov 18 10:30 /sbin/init

(My version is linked statically of course.)

The NSS parts of init might not be needed in normal operation, but its
hard to tell.

Bruce
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-19 Thread Ken Smith
On Thu, Nov 20, 2003 at 06:27:31AM +1100, Bruce Evans wrote:

  set init_path=/rescue/init
 
 If dynamic root were ready to be turned on, then /rescue/init would be
 in the default init_path.

I had that explained to me too. :-)

There is a loop in sys/kern/init_main.c that probes for an init
to run.  But it only does what you want for cases of the files
not existing or otherwise just totally not executable.  It won't
handle the started but then dumped core case the way it would
need to if /sbin/init were to fail because of shared library
problems.  So if just relying on this mechanism it would either
not work right (/sbin/init in the path before /rescue/init) or
it would always start /rescue/init (/rescue/init before /sbin/init
in the path).

-- 
Ken Smith
- From there to here, from here to  |   [EMAIL PROTECTED]
  there, funny things are everywhere.   |
  - Theodore Geisel |
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-19 Thread Robert Watson

On Thu, 20 Nov 2003, Bruce Evans wrote:

 On Wed, 19 Nov 2003, Marcel Moolenaar wrote:
 
  set init_path=/rescue/init
 
 If dynamic root were ready to be turned on, then /rescue/init would be
 in the default init_path. 

The fallback path only works if the exec() fails cleanly without actually
starting the userspace code.  Peter and I have been talking about
improving failure mode handling by having rtld return a special error code
if (getpid() == 1  oops_shlib_problem) and having special kernel
handling, but that's complicated by the fact that presumably you'd then
need to pick up the whole init thing in exit() rather than continuing it
in start_init().  In the meantime, Gordon has committed a change to always
link init statically.

  A dynamicly linked /sbin/init just
  makes it harder to get to the rescue bits, so it makes sense to
  link init(8) staticly. Especially since there's no advantage to
  dynamic linking init(8) that compensates for the inconvience.
 
 It obviously uses NSS.  How else could it be so bloated? :
 
 $ ls -l /sbin/init
 -r-x--  1 root  wheel  453348 Nov 18 10:30 /sbin/init
 
 (My version is linked statically of course.)
 
 The NSS parts of init might not be needed in normal operation, but its
 hard to tell. 

There appear to be at least two dependencies there, from my reading:

(1) getpwnam(root) so that we can check the root password when the
console isn't marked as secure.

(2) Calls to setusercontext() to set up resources when running rc, also
for uid 0, which results in libutil looking up the user to find the
class.

I think that while home directory completion is useful for sh, it's safe
to have a working assumption of a local root user in the password file,
and that when prompting for a single user mode password, access to
directory services is probably going to fail anyway.

Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
[EMAIL PROTECTED]  Network Associates Laboratories

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-19 Thread E.B. Dreger
SL Date: Tue, 18 Nov 2003 17:06:06 -0700 (MST)
SL From: Scott Long

SL 3.  Binary security updates: there is a lot of interest in providing a
SL binary update mechanism for doing security updates.  Having a dynamic
SL root means that vulnerable libraries can be updated without having to
SL update all of the static binaries that might use them.

Although this doesn't help the upgrade process, what if one
symbol (such as function name + CVS tag) were exported per
function?  One could check for a vulnerability by strings | grep
funcname | inspect CVS tag.  A more elegant approach would be to
store such versioning in another segment and have a tool that
understands the data, a la debugger symbols.

On a different note:

+ Some of us have had a few bad experiences with glibc (granted,
  it's glibc) upgrades when the shell, cp, ls, et cetera are
  dynamically linked.

+ I put the shell of choice and all of SSH's guts on the root
  partition... if /usr gets clobbered, I still want to be able
  to boot and log in remotely.  If / gets clobbered, I have
  bigger problems. :-)


Eddy
--
Brotsman  Dreger, Inc. - EverQuick Internet Division
Bandwidth, consulting, e-commerce, hosting, and network building
Phone: +1 785 865 5885 Lawrence and [inter]national
Phone: +1 316 794 8922 Wichita
_
  DO NOT send mail to the following addresses :
  [EMAIL PROTECTED] -or- [EMAIL PROTECTED] -or- [EMAIL PROTECTED]
Sending mail to spambait addresses is a great way to get blocked.

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-19 Thread E.B. Dreger
GAD Date: Tue, 18 Nov 2003 21:54:53 -0500
GAD From: Garance A Drosihn


GAD Many freebsd users (me for one) are still living on a modem,
GAD where even one bump of 1.5 meg is a significant issue...
GAD
GAD Remember that the issue we're talking about is security
GAD updates, not full system upgrades.  Everyone would want
GAD the security updates, even if they're on a slow link.

When security updates change but a few bytes, it seems that some
xdiff- or rsync-like algorithm would be an apropriate way to
distribute patches.


Eddy
--
Brotsman  Dreger, Inc. - EverQuick Internet Division
Bandwidth, consulting, e-commerce, hosting, and network building
Phone: +1 785 865 5885 Lawrence and [inter]national
Phone: +1 316 794 8922 Wichita
_
  DO NOT send mail to the following addresses :
  [EMAIL PROTECTED] -or- [EMAIL PROTECTED] -or- [EMAIL PROTECTED]
Sending mail to spambait addresses is a great way to get blocked.

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-19 Thread Matthew Dillon
:GAD Many freebsd users (me for one) are still living on a modem,
:GAD where even one bump of 1.5 meg is a significant issue...
:GAD
:GAD Remember that the issue we're talking about is security
:GAD updates, not full system upgrades.  Everyone would want
:GAD the security updates, even if they're on a slow link.
:
:When security updates change but a few bytes, it seems that some
:xdiff- or rsync-like algorithm would be an apropriate way to
:distribute patches.
:
:
:Eddy
:--
:Brotsman  Dreger, Inc. - EverQuick Internet Division

Security updates are a red hearing regardless because they are few
and far between compared to even a modem-user's bandwidth (especially
those modem users who are likely to read the security lists aren't going
to care if it takes an hour to download a non optimal binary patch if
it only happens a few times a year).

-Matt

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-19 Thread Bruce Evans
On Wed, 19 Nov 2003, Ken Smith wrote:

 On Thu, Nov 20, 2003 at 06:27:31AM +1100, Bruce Evans wrote:

   set init_path=/rescue/init
 
  If dynamic root were ready to be turned on, then /rescue/init would be
  in the default init_path.

 I had that explained to me too. :-)

 There is a loop in sys/kern/init_main.c that probes for an init
 to run.  But it only does what you want for cases of the files
 not existing or otherwise just totally not executable.  It won't
 handle the started but then dumped core case the way it would
 need to if /sbin/init were to fail because of shared library
 problems.  So if just relying on this mechanism it would either
 not work right (/sbin/init in the path before /rescue/init) or
 it would always start /rescue/init (/rescue/init before /sbin/init
 in the path).

Oops, better add ... and error handling for init_path would be fixed -).

I should have remembered this since I got bitten by it recently.  I was
trying to boot RELENG_3 and had a backup init that worked but that didn't
help because there was an execable init earlier in the path.

Bruce
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-19 Thread Richard Coleman
Gordon Tetlow wrote:
On Tue, Nov 18, 2003 at 08:03:23PM -0500, [EMAIL PROTECTED] wrote:

However, PAM and NSS 'tricks' really seem to be exactly that,
and certainly worthy of special builds.  However, that isn't
necessary, yet still not building everything with a shared
libc.


Things like nss_ldap (which is used *heavily* at my place of employment)
are some reasons that FreeBSD doesn't make it into more places. It was
the reason why FreeBSD isn't being used here. Calling them 'tricks'
(and succumbing to the name calling you wanted to avoid) doesn't change
the fact that every major contender (IRIX, Solaris, Linux to name a few)
all support this feature set.
The fact that you can't easily do centralized authentication (nss_ldap 
and pam_ldap) with FreeBSD is a major show stopper.  At my former 
employer, we built several very large systems that required centralized 
authentication using ldap.  We had to use Linux, since none of the BSD's 
supported this correctly at the time.

I don't really care whether everything is statically or dynamically 
linked.  With the fast machines and huge disks these days, bloat is not 
much of an issue.  But nss and pam need to work correctly.  If the folks 
that are against dynamic linking have an alternate method to make this 
work, I'm all for it.  But it needs to be more than theory.  We need code.

To be honest, I've never understood the (seemingly irrational) 
resistance against this change.  Solaris made this change 10 years ago.

Richard Coleman
[EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-19 Thread Dan Nelson
In the last episode (Nov 19), Richard Coleman said:
 I don't really care whether everything is statically or dynamically 
 linked.  With the fast machines and huge disks these days, bloat is not 
 much of an issue.  But nss and pam need to work correctly.  If the folks 
 that are against dynamic linking have an alternate method to make this 
 work, I'm all for it.  But it needs to be more than theory.  We need code.
 
 To be honest, I've never understood the (seemingly irrational) 
 resistance against this change.  Solaris made this change 10 years ago.

Not completely:

$ uname -a
SunOS pd1 5.9 Generic_112233-08 sun4u sparc SUNW,Ultra-Enterprise
$ file /bin/sh
/sbin/sh:   ELF 32-bit MSB executable SPARC Version 1, statically linked, stripped
$ file /sbin/* | grep statically | cut -d: -f1 | fmt
/sbin/autopush /sbin/fdisk /sbin/jsh /sbin/mount /sbin/sh
/sbin/soconfig /sbin/sync /sbin/umount /sbin/uname

-- 
Dan Nelson
[EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-19 Thread Richard Coleman
Dan Nelson wrote:

In the last episode (Nov 19), Richard Coleman said:

I don't really care whether everything is statically or dynamically 
linked.  With the fast machines and huge disks these days, bloat is not 
much of an issue.  But nss and pam need to work correctly.  If the folks 
that are against dynamic linking have an alternate method to make this 
work, I'm all for it.  But it needs to be more than theory.  We need code.

To be honest, I've never understood the (seemingly irrational) 
resistance against this change.  Solaris made this change 10 years ago.


Not completely:

$ uname -a
SunOS pd1 5.9 Generic_112233-08 sun4u sparc SUNW,Ultra-Enterprise
$ file /bin/sh
/sbin/sh:   ELF 32-bit MSB executable SPARC Version 1, statically linked, stripped
$ file /sbin/* | grep statically | cut -d: -f1 | fmt
/sbin/autopush /sbin/fdisk /sbin/jsh /sbin/mount /sbin/sh
/sbin/soconfig /sbin/sync /sbin/umount /sbin/uname
I have no problem with FreeBSD doing something similar and leaving a few 
binaries static.  I think most of the resistance to that was due to the 
increased complexity of the build system.

It seems /bin/sh is the real sticking point.  But if the compromise is 
to statically link /bin/sh, that would be cool with me.  Other than 
tilde expansion not working when using nss_ldap, I can't think of any 
other problems.  I consider that a minor blemish I could easily live 
with.  Normal users will not generally have /bin/sh has their shell 
anyways.  And I could always compile a dynamically linked version into 
/usr/bin if necessary.

To be honest, 98% of the time that someone notices brokeness due to 
nss_ldap, it comes when using /bin/ls.

Richard Coleman
[EMAIL PROTECTED]


___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-19 Thread Robert Watson

On Wed, 19 Nov 2003, Dan Nelson wrote:

 In the last episode (Nov 19), Richard Coleman said:
  I don't really care whether everything is statically or dynamically 
  linked.  With the fast machines and huge disks these days, bloat is not 
  much of an issue.  But nss and pam need to work correctly.  If the folks 
  that are against dynamic linking have an alternate method to make this 
  work, I'm all for it.  But it needs to be more than theory.  We need code.
  
  To be honest, I've never understood the (seemingly irrational) 
  resistance against this change.  Solaris made this change 10 years ago.
 
 Not completely:
 
 $ uname -a
 SunOS pd1 5.9 Generic_112233-08 sun4u sparc SUNW,Ultra-Enterprise
 $ file /bin/sh
 /sbin/sh:   ELF 32-bit MSB executable SPARC Version 1, statically linked, 
 stripped
 $ file /sbin/* | grep statically | cut -d: -f1 | fmt
 /sbin/autopush /sbin/fdisk /sbin/jsh /sbin/mount /sbin/sh
 /sbin/soconfig /sbin/sync /sbin/umount /sbin/uname

Seems to depend a bit on the install, maybe...

raven:~ uname -a
SunOS raven 5.8 Generic_108528-06 sun4u sparc SUNW,Ultra-250
raven:~ file /bin/sh /sbin/sh
/bin/sh:ELF 32-bit MSB executable SPARC Version 1, dynamically linked, stripped
/sbin/sh:   ELF 32-bit MSB executable SPARC Version 1, statically linked, stripped

Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
[EMAIL PROTECTED]  Network Associates Laboratories


___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-19 Thread Tim Kientzle
Richard Coleman wrote:
It seems /bin/sh is the real sticking point. 
There is a problem here: Unix systems have historically used
/bin/sh for two somewhat contradictory purposes:
  * the system script interpreter
  * as a user shell
The user shell must be dynamically linked in order
to support centralized administration.  I personally
see no way around that.  Given that many users do
rely on /bin/sh, it seems that /bin/sh must be
dynamically linked.
There are good reasons to want the system script
interpreter statically linked.
Maybe it's time to separate these two functions?
I would be content to have a static /sbin/sh
that is used as the system script interpreter for
rc scripts, etc.
Tim Kientzle

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-19 Thread dyson
Tim Kientzle said:
 Richard Coleman wrote:
  It seems /bin/sh is the real sticking point. 
 
 There is a problem here: Unix systems have historically used
 /bin/sh for two somewhat contradictory purposes:
* the system script interpreter
* as a user shell
 
 The user shell must be dynamically linked in order
 to support centralized administration.  I personally
 see no way around that.  Given that many users do
 rely on /bin/sh, it seems that /bin/sh must be
 dynamically linked.
 
It isn't necessary for the shell to be dynamically linked
(efficiency issue WRT the sparse allocations and greater
COW overheads/etc) for the shell to programmatically link in
a module for optional feature sets.  This can even be
placed under a libc call (which then wouldn't encumber
the shell unless the feature was active and increase the
footprint of generally all libc routines.)

John
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-18 Thread M. Warner Losh
In message: [EMAIL PROTECTED]
[EMAIL PROTECTED] writes:
:   It really doesn't make sense to arbitrarily cut-off a
:   discussion especially when a decision might be incorrect.

I'd say that good technical discussion about why this is wrong would
be good.  However, emotional ones should be left behind.  Except for
John's message, most of the earlier messages have been more emotional
than technical.

John, do you have any good set of benchmarks that people can run to
illustrate your point?

Warner
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-18 Thread Dan Nelson
In the last episode (Nov 18), M. Warner Losh said:
 In message: [EMAIL PROTECTED]
 [EMAIL PROTECTED] writes:
 : It really doesn't make sense to arbitrarily cut-off a discussion
 : especially when a decision might be incorrect.
 
 I'd say that good technical discussion about why this is wrong would
 be good.  However, emotional ones should be left behind.  Except for
 John's message, most of the earlier messages have been more emotional
 than technical.
 
 John, do you have any good set of benchmarks that people can run to
 illustrate your point?

Pretty much any benchmark that you can build statically and dynamically
should suffice.  I've attached a simple one that fills an array with
random numbers then qsorts them.  make compare will generate three
graphs at the end: time spent loading the executable, time spent within
the loops, and total time.  Note that both load and loop timings are
higher for the dynamic binary.  I ran it on a busy system, which is why
there are so many outliers.  Make sure you have
src/tools/tools/ministat installed someplace in your path.

Also see
http://lists.freebsd.org/pipermail/freebsd-current/2003-April/001106.html ,
where I had posted proc/pid/maps for a static and dynamic ls.

-- 
Dan Nelson
[EMAIL PROTECTED]
all: compare

REPS=50

STATICLOGS=static.total.log static.loop.log static.load.log
DYNAMICLOGS=dynamic.total.log dynamic.loop.log dynamic.load.log
LOGS=${STATICLOGS} ${DYNAMICLOGS}

CFLAGS+=-Wall

static: svd.o
${CC} -static ${CFLAGS} ${LDFLAGS} -o ${.TARGET} ${.ALLSRC} ${LDADD}

dynamic: svd.o
${CC} ${CFLAGS} ${LDFLAGS} -o ${.TARGET} ${.ALLSRC} ${LDADD}

clean:
rm -f static dynamic ${LOGS} *.o

.PHONY: run
run ${LOGS}: static dynamic
@rm -f ${LOGS}
@reps=0; while [ $$reps -lt ${REPS} ] ; do  \
  time -p ./static 21  static.loop.log | sed -ne '/real/s/real //p'  
static.total.log ; \
  time -p ./dynamic 21  dynamic.loop.log | sed -ne '/real/s/real //p'  
dynamic.total.log ; \
  echo -n . ; \
  reps=$$(($$reps+1)) ; \
done
@echo
@paste static.total.log static.loop.log | awk '{print $$1 - $$2}'  
static.load.log
@paste dynamic.total.log dynamic.loop.log | awk '{print $$1 - $$2}'  
dynamic.load.log

compare: ${LOGS}
ministat -s static.load.log dynamic.load.log 
ministat -s static.loop.log dynamic.loop.log 
ministat -s static.total.log dynamic.total.log 
#include stdio.h
#include stdlib.h
#include string.h
#include sys/time.h

#define SIZE 1024000

int comp(const void *a, const void *b)
{
return memcmp(a, b, sizeof(int));
}

int main(int argc, char *argv[])
{
int numbers[SIZE];
int i;
struct timeval tv1, tv2;

gettimeofday(tv1, NULL);

srand(5);
for (i = 0; i  SIZE; i++)
{
numbers[i] = rand();
}
qsort(numbers, SIZE, sizeof(*numbers), comp);
gettimeofday(tv2, NULL);

timersub(tv2, tv1, tv1);
printf(%ld.%06ld\n, tv1.tv_sec, tv1.tv_usec);

return 0;
}
ministat -s static.load.log dynamic.load.log
x static.load.log
+ dynamic.load.log
+--+
|   xx   + |
|   xx  ++ |
|   xx  ++ |
|   ++ |
|   ++ |
|   ++ |
|   ++ |
|   +++|
|  x+++|
|  x*  |
|  x*+ +   |
| x**+ +   ++  |
| x**+ ++  ++ +|
| *+++**+ ++   |
| xxx**+****+*++ ++*  x  +++  +|
||___M_A|  |
|   |__M__A_|  |
+--+
N   Min   MaxMedian   AvgStddev
x  96  0.001561  0.027646  0.003969  0.0049661042  0.0040137086
+  96  0.003765   0.05356   0.00824   0.010588323  0.0072195082
Difference at 95.0% confidence
0.0056 +/- 0.00165239
113.212% +/- 33.2733%

Re: Unfortunate dynamic linking for everything

2003-11-18 Thread Bruce Evans
On Tue, 18 Nov 2003, M. Warner Losh wrote:

 In message: [EMAIL PROTECTED]
 [EMAIL PROTECTED] writes:
 : It really doesn't make sense to arbitrarily cut-off a
 : discussion especially when a decision might be incorrect.

 I'd say that good technical discussion about why this is wrong would
 be good.  However, emotional ones should be left behind.  Except for
 John's message, most of the earlier messages have been more emotional
 than technical.

I used to use all dynamic linkage, but switched to all static linkage
(except for ports) when I understood John's points many year ago.  It
shouldn't be necessary to repeat the argmuments.

 John, do you have any good set of benchmarks that people can run to
 illustrate your point?

Almost any benchmark that does lots of forks or execs, or uses libraries
a lot will do.  IIRC, 5-10% of my speedups for makeworld was from building
tools static.  Makeworld is not such a good benchmark for this as it used
to be since it always builds tools static so the non-staticness of
standard binaries doesn't matter so much.  Perhaps it still matters for
/bin/sh.

Bruce
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-18 Thread dyson
M. Warner Losh said:
 In message: [EMAIL PROTECTED]
 [EMAIL PROTECTED] writes:
 : It really doesn't make sense to arbitrarily cut-off a
 : discussion especially when a decision might be incorrect.
 
 I'd say that good technical discussion about why this is wrong would
 be good.  However, emotional ones should be left behind.  Except for
 John's message, most of the earlier messages have been more emotional
 than technical.
 
 John, do you have any good set of benchmarks that people can run to
 illustrate your point?
 
I'll see if I can find some of my old benchmarks (from memory,
not disk -- lots of stuff has rotted away.).  I had hoped
to avoid doing much in the way of proof.  Pointing to the
much more complex process map along with the implication
for significantly higher overhead :-).  (The VM code generally
gets slower with more complex process maps and more memory used.
For smallish programs -- including those with a few shared libs,
changing the VM code won't help much, because the cost is built
in to the complexity of the process image.)

If set-up correctly (trying to find facts, rather than proving
a foregone conclusion), even the fork/exec
tests on LMBENCH can be informative.  The LMBENCH fork/exec benchmarks
don't really measure EVERYTHING, and there appears to be a
additional overhead beyond the old-time a.out, but they
will likely still show some of the additional overhead.  (Actually,
simple fork/exec tests of simple programs don't show all of the
additional overhead, and also bias the overhead results, but
are one data point.)

There might be a certain 'coolness' WRT dynamically linking everything,
but the overhead is certainly measurable.  If the object is to
maximally 'share', then for shells the FreeBSD VM shares maximally
without using shared libs (perhaps there is a lost know-how about
how aggressively the FreeBSD VM implements parent/child and exec
based sharing?)

I'll try to put together a few simple benchmarks -- mostly from my
defective memory.  I had recently refreshed myself on this issue
(but lost again due to disk hardware disasters and being
clobbered by vinum problems -- which I have given up on.)  Gimme
a day or so -- I have several other things in my queue.

John

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-18 Thread Scott Long
On Tue, 18 Nov 2003 [EMAIL PROTECTED] wrote:
 M. Warner Losh said:
  In message: [EMAIL PROTECTED]
  [EMAIL PROTECTED] writes:
  :   It really doesn't make sense to arbitrarily cut-off a
  :   discussion especially when a decision might be incorrect.
 
  I'd say that good technical discussion about why this is wrong would
  be good.  However, emotional ones should be left behind.  Except for
  John's message, most of the earlier messages have been more emotional
  than technical.
 
  John, do you have any good set of benchmarks that people can run to
  illustrate your point?
 
 I'll see if I can find some of my old benchmarks (from memory,
 not disk -- lots of stuff has rotted away.).  I had hoped
 to avoid doing much in the way of proof.  Pointing to the
 much more complex process map along with the implication
 for significantly higher overhead :-).  (The VM code generally
 gets slower with more complex process maps and more memory used.
 For smallish programs -- including those with a few shared libs,
 changing the VM code won't help much, because the cost is built
 in to the complexity of the process image.)

I'm glad that real arguments are finally starting to surface =-)

Our rationale for encouraging Gordon is as follows:

1.  4.x upgrade path:  As we approach 5-STABLE, a lot of users might want
to upgrade from 4-STABLE.  Historically in 4.x, the / partition has
been very modest in size.  One just simply cannot cram the bloat that
has grown in 5.x into a 4.x partition scheme.  Of course there is the
venerable 'dump - clean install - restore' scheme, but we were looking
for something a little more user-friendly.

2.  NSS support out-of-the-box:  Again, this is a user-experience issue
in that forcing NSS users to recompile world was seen as a potential
roadblock to them.

3.  Binary security updates: there is a lot of interest in providing a
binary update mechanism for doing security updates.  Having a dynamic
root means that vulnerable libraries can be updated without having to
update all of the static binaries that might use them.

4.  I've probably forgotten the other factors...

As for performance, we felt that the typical use of FreeBSD did not fall
into the category of doing forkbomb performance tests.  While there might
certainly be users that do continuously create lots of short-lived
processes, we felt that the above benefits outweighed this, but left the
door open for tailoring to this need (recompiling world with
NO_DYNAMICROOT).

If people really are concerned with the VM microbenchmarks associated with
complex process maps, then hopefully this is a challenge to look for
optimizations there.

Scott
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-18 Thread dyson
Scott Long said:
 On Tue, 18 Nov 2003 [EMAIL PROTECTED] wrote:
  M. Warner Losh said:
   In message: [EMAIL PROTECTED]
   [EMAIL PROTECTED] writes:
   : It really doesn't make sense to arbitrarily cut-off a
   : discussion especially when a decision might be incorrect.
  
   I'd say that good technical discussion about why this is wrong would
   be good.  However, emotional ones should be left behind.  Except for
   John's message, most of the earlier messages have been more emotional
   than technical.
  
   John, do you have any good set of benchmarks that people can run to
   illustrate your point?
  
  I'll see if I can find some of my old benchmarks (from memory,
  not disk -- lots of stuff has rotted away.).  I had hoped
  to avoid doing much in the way of proof.  Pointing to the
  much more complex process map along with the implication
  for significantly higher overhead :-).  (The VM code generally
  gets slower with more complex process maps and more memory used.
  For smallish programs -- including those with a few shared libs,
  changing the VM code won't help much, because the cost is built
  in to the complexity of the process image.)
 
 I'm glad that real arguments are finally starting to surface =-)
 
 Our rationale for encouraging Gordon is as follows:
 
 1.  4.x upgrade path:  As we approach 5-STABLE, a lot of users might want
 to upgrade from 4-STABLE.  Historically in 4.x, the / partition has
 been very modest in size.  One just simply cannot cram the bloat that
 has grown in 5.x into a 4.x partition scheme.  Of course there is the
 venerable 'dump - clean install - restore' scheme, but we were looking
 for something a little more user-friendly.

/bin and /sbin don't need to contain everything :-).  Adding a full
featured /rescue should also help to counterbalance this (but maybe
the upgrade would be selective?)  There are reasons for /usr/bin
and /usr/sbin (along with /usr/local, /usr/share.)

Disk space is CHEAP and grandfathering 20MB hard drives (or the
yr2000 equivalent of 4GB hard drives) is nice -- but repartitioning
them with more reasonable root allocations seems like a good
idea anyway :-).  50MB root was silly when 2-4GB disks were common.

Continuing to pay for a poorly made partitioning decision in the past
seems to be unworthy (in my opinion.)

 
 2.  NSS support out-of-the-box:  Again, this is a user-experience issue
 in that forcing NSS users to recompile world was seen as a potential
 roadblock to them.

The cool thing about properly implemented shared libs is that not everything
has to be shared.  Even the kernel supports runtime loading/addition
of modules, and the NSS sounds like a good candidate for a library
feature.  Burdening everything with the sparse .data associated with
libc seems to be a waste (but, perhaps the performance that was carefully
crafted into the codebase is no longer important?)  Really -- it is
possible that performance isnt' important anymore, and fully accepting
the ongoing loss of performance for features (without careful tradeoffs)
might be an appropriate strategy.  (In the case of the NSS stuff, it
shouldn't require everything to be built dynamic.)

 
 3.  Binary security updates: there is a lot of interest in providing a
 binary update mechanism for doing security updates.  Having a dynamic
 root means that vulnerable libraries can be updated without having to
 update all of the static binaries that might use them.

The additional hole of exploiting the system through the shared libs
is a negative tradeoff.

 
 4.  I've probably forgotten the other factors...
 
 As for performance, we felt that the typical use of FreeBSD did not fall
 into the category of doing forkbomb performance tests.

Nope - even shell execution times (e.g. long shell scripts) will see a
difference.  Prejudicial comments like 'fork-bombs' and
'microbenchmarks' aren't really fair.  Proper benchmarking is
time consuming and analysis is tricky.  System loading conditions
are difficult to control (for experiments.)

Note that alot of the lost performance is hidden by the VM code (e.g. one
of the purposes for prezeroing on SMP box -- before SMP was fine grained.)
The cost of sparse data and needless inheritance of modified pages
has performance hits that are incredibly difficult to accurately (and
honestly) measure.  Anyone can create 'preordained' benchmark results.

This is one reason why I am loathe to get into the FreeBSD performance
game again -- it is much more difficult than a few micro-benchmarks :-)
that simply measure a fork exec time.

John

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-18 Thread Colin Percival
At 17:06 18/11/2003 -0700, Scott Long wrote:
Our rationale for encouraging Gordon is as follows:

1.  4.x upgrade path:  As we approach 5-STABLE, a lot of users might want
to upgrade from 4-STABLE.  Historically in 4.x, the / partition has
been very modest in size.  One just simply cannot cram the bloat that
has grown in 5.x into a 4.x partition scheme.  Of course there is the
venerable 'dump - clean install - restore' scheme, but we were looking
for something a little more user-friendly.
  Of course, making / dynamic results in added complication of removing 
old libraries from /usr/lib, now that some of them have moved to /lib...

3.  Binary security updates: there is a lot of interest in providing a
binary update mechanism for doing security updates.  Having a dynamic
root means that vulnerable libraries can be updated without having to
update all of the static binaries that might use them.
  As far as I'm concerned, this is a non-issue.  Identifying which static 
binaries need to be replaced is now a solved problem, replacing them is 
easy, and if binary patches are used, there is effectively no impact on 
bandwidth usage either.

  On the issue of performance, however: I know people have benchmarked 
fork-bombs, but has anyone done benchmarks with moderate numbers of 
long-lived, library-intensive, processes?  It seems to me that dynamic 
linking could have caching advantages.

Colin Percival

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-18 Thread masta


[EMAIL PROTECTED] wrote:

 Scott Long said:

On Tue, 18 Nov 2003 [EMAIL PROTECTED] wrote:

M. Warner Losh said:

In message: [EMAIL PROTECTED]
[EMAIL PROTECTED] writes:
:   It really doesn't make sense to arbitrarily cut-off a
:   discussion especially when a decision might be incorrect.

I'd say that good technical discussion about why this is wrong would
be good.  However, emotional ones should be left behind.  Except for
John's message, most of the earlier messages have been more emotional
than technical.

John, do you have any good set of benchmarks that people can run to
illustrate your point?


I'll see if I can find some of my old benchmarks (from memory,
not disk -- lots of stuff has rotted away.).  I had hoped
to avoid doing much in the way of proof.  Pointing to the
much more complex process map along with the implication
for significantly higher overhead :-).  (The VM code generally
gets slower with more complex process maps and more memory used.
For smallish programs -- including those with a few shared libs,
changing the VM code won't help much, because the cost is built
in to the complexity of the process image.)

I'm glad that real arguments are finally starting to surface =-)

Our rationale for encouraging Gordon is as follows:

1.  4.x upgrade path:  As we approach 5-STABLE, a lot of users might want
to upgrade from 4-STABLE.  Historically in 4.x, the / partition has
been very modest in size.  One just simply cannot cram the bloat that
has grown in 5.x into a 4.x partition scheme.  Of course there is the
venerable 'dump - clean install - restore' scheme, but we were looking
for something a little more user-friendly.


[snip]

2.  NSS support out-of-the-box:  Again, this is a user-experience issue
in that forcing NSS users to recompile world was seen as a potential
roadblock to them.


[snip]

3.  Binary security updates: there is a lot of interest in providing a
binary update mechanism for doing security updates.  Having a dynamic
root means that vulnerable libraries can be updated without having to
update all of the static binaries that might use them.


[snip]


4.  I've probably forgotten the other factors...

As for performance, we felt that the typical use of FreeBSD did not fall
into the category of doing forkbomb performance tests.


One of ther things he might have forgot to mention is dynamic tricks
releated to PAM, which sorta falls in the same league as NSS working out
of the box. It was worth mentioning IMHO.

Besides, I see nothing preventing anybody from building their system with
static worlds, and there is nothing stopping anybody from putting /rescue
in the PATH before /bin or /sbin.

 __  __   _
|  \/  | __ _ ___| |_ __ _
| |\/| |/ _` / __| __/ _` |
| |  | | (_| \__ \ || (_| |
|_|  |_|\__,_|___/\__\__,_|

unzip ; strip ; touch ; finger ; mount ; fsck ; more ; yes ; umount ; sleep


[EMAIL PROTECTED]
http://wifibsd.org



___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-18 Thread dyson
masta said:

 One of ther things he might have forgot to mention is dynamic tricks
 releated to PAM, which sorta falls in the same league as NSS working out
 of the box. It was worth mentioning IMHO.
 
I guess that I have to remember that my own goals of 'performance'
and handling 'highest workload' for efficient use of hardware isn't
everyone's goal.

However, PAM and NSS 'tricks' really seem to be exactly that,
and certainly worthy of special builds.  However, that isn't
necessary, yet still not building everything with a shared
libc.

Note that none of this requires that libc be shared (libc and
its ilk are the worst offenders.)  Dynamic loading can certainly
be used (if you want to load a wierd, special purpose, tricky
module.)

John
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-18 Thread Scott Long
On Tue, 18 Nov 2003 [EMAIL PROTECTED] wrote:
 Scott Long said:
  On Tue, 18 Nov 2003 [EMAIL PROTECTED] wrote:
   M. Warner Losh said:
In message: [EMAIL PROTECTED]
[EMAIL PROTECTED] writes:
:   It really doesn't make sense to arbitrarily cut-off a
:   discussion especially when a decision might be incorrect.
   
I'd say that good technical discussion about why this is wrong would
be good.  However, emotional ones should be left behind.  Except for
John's message, most of the earlier messages have been more emotional
than technical.
   
John, do you have any good set of benchmarks that people can run to
illustrate your point?
   
   I'll see if I can find some of my old benchmarks (from memory,
   not disk -- lots of stuff has rotted away.).  I had hoped
   to avoid doing much in the way of proof.  Pointing to the
   much more complex process map along with the implication
   for significantly higher overhead :-).  (The VM code generally
   gets slower with more complex process maps and more memory used.
   For smallish programs -- including those with a few shared libs,
   changing the VM code won't help much, because the cost is built
   in to the complexity of the process image.)
 
  I'm glad that real arguments are finally starting to surface =-)
 
  Our rationale for encouraging Gordon is as follows:
 
  1.  4.x upgrade path:  As we approach 5-STABLE, a lot of users might want
  to upgrade from 4-STABLE.  Historically in 4.x, the / partition has
  been very modest in size.  One just simply cannot cram the bloat that
  has grown in 5.x into a 4.x partition scheme.  Of course there is the
  venerable 'dump - clean install - restore' scheme, but we were looking
  for something a little more user-friendly.
 
 /bin and /sbin don't need to contain everything :-).  Adding a full
 featured /rescue should also help to counterbalance this (but maybe
 the upgrade would be selective?)  There are reasons for /usr/bin
 and /usr/sbin (along with /usr/local, /usr/share.)

 Disk space is CHEAP and grandfathering 20MB hard drives (or the
 yr2000 equivalent of 4GB hard drives) is nice -- but repartitioning
 them with more reasonable root allocations seems like a good
 idea anyway :-).  50MB root was silly when 2-4GB disks were common.

 Continuing to pay for a poorly made partitioning decision in the past
 seems to be unworthy (in my opinion.)

/boot has grown quite large too and threatens to be unbounded in size as
times go on.  Shaving off the 30-40MB of size in /bin and /sbin can
help alleviate this, even on system formatted in 5.x partition sized.

This argument wasn't the most compelling one by itself, but it played a
part in the decision so I included it here.


 
  2.  NSS support out-of-the-box:  Again, this is a user-experience issue
  in that forcing NSS users to recompile world was seen as a potential
  roadblock to them.
 
 The cool thing about properly implemented shared libs is that not everything
 has to be shared.  Even the kernel supports runtime loading/addition
 of modules, and the NSS sounds like a good candidate for a library
 feature.  Burdening everything with the sparse .data associated with
 libc seems to be a waste (but, perhaps the performance that was carefully
 crafted into the codebase is no longer important?)  Really -- it is
 possible that performance isnt' important anymore, and fully accepting
 the ongoing loss of performance for features (without careful tradeoffs)
 might be an appropriate strategy.  (In the case of the NSS stuff, it
 shouldn't require everything to be built dynamic.)

Unfortunately, our dynamic linker prevents dlopen() and friends from
working in a static binary.  There was some talk about having NSS
lookups happen through a proxy process, but our implementation just
plain wasn't done that way, and no one has stepped forward with
alternate code.

NSS modules can affect a surprisingly large number of utilities, including
the shells.  Selectively choosing which utilities to link static vs.
shared would have resulted in quite a patchy result that probably would
not have had a significant benefit.


 
  3.  Binary security updates: there is a lot of interest in providing a
  binary update mechanism for doing security updates.  Having a dynamic
  root means that vulnerable libraries can be updated without having to
  update all of the static binaries that might use them.
 
 The additional hole of exploiting the system through the shared libs
 is a negative tradeoff.

Exploits in libraries happen though.  The LD_LIBRARY_PATH attack is an old
one that most Unixes are hopefully hardened against.


 
  4.  I've probably forgotten the other factors...
 
  As for performance, we felt that the typical use of FreeBSD did not fall
  into the category of doing forkbomb performance tests.
 
 Nope - even shell execution times (e.g. long shell scripts) will see a
 difference.  

Re: Unfortunate dynamic linking for everything

2003-11-18 Thread Garance A Drosihn
At 8:07 AM -0500 11/18/03, [EMAIL PROTECTED] wrote:
It really doesn't make sense to arbitrarily cut-off a
discussion especially when a decision might be incorrect.
All I wanted to cut off was the claim that this decision had
not been discussed publicly before.  It was also annoying that
most the recent complaints (before your message) were issues
that had been explicitly discussed and addressed before the
decision had been reached.  Many people seem to be complaining
on what they think we did, as opposed to what we actually did.
If there hadn't been a noticed increase in cost by using
all-shared-libs, then the measurements were done
incorrectly.  If the decision is made based upon allowing
for 1.5X (at least) times increase in fork/exec times, and
larger memory usage (due to sparse allocations), ...
I do remember some comments about benchmarks, and it was
true that the all-dynamic bin/sbin does come out slower.  I
don't remember if the benchmarks were ours or from NetBSD's
investigation.  However, I think we measured increase in
overall time for some set of commands, instead of increase
in the fork() routine.  Thus, the penalty observed was much
less than 50%.  I think it was under 2%, but I don't remember
the exact number.  When we're dealing with a 100% increase
in the cost of compiling something with the newer gcc, the
increase due to this change seemed pretty insignificant...
It is probably true that there are several commands where we
would see a worthwhile performance benefit by static linking,
and which have no need for things like dynamic PAM modules.
But commands which care about userids or group information
would need to stay dynamic (IMO), and I think that means all
shells.
Also, when it comes to security fixes, it would be nice for
binary upgrades if all we had to do was replace one library,
instead of replacing every command which is statically-linked
with the problem routine.
The announcement of this change may have played up disk-savings
more than it should have, because we weren't doing this to save
disk space.  It was just that the disk savings were a very nice
side-effect.  It's pretty rare that we can talk about the system
getting smaller!:-)
--
Garance Alistair Drosehn=   [EMAIL PROTECTED]
Senior Systems Programmer   or  [EMAIL PROTECTED]
Rensselaer Polytechnic Instituteor  [EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-18 Thread Garance A Drosihn
At 6:42 PM -0600 11/18/03, masta wrote:
Besides, I see nothing preventing anybody from building
their system with static worlds,
true...

and there is nothing stopping anybody from putting /rescue
in the PATH before /bin or /sbin.
Note that this will not have the same performance as the
older all-static setup, because each binary in /rescue
is a crunchgen'ed combination of *all* the commands there.
They're all hardlinked together.  Eg:
(332) # ls -l /bin/cp /rescue/cp
-r-xr-xr-x1 root  wheel   113372 Nov 16 02:29 /bin/cp*
-r-xr-xr-x  131 root  wheel  3807836 Nov 16 02:30 /rescue/cp*
I have no idea what kind of a difference that would make,
I'm just saying that it is different...  :-)
--
Garance Alistair Drosehn=   [EMAIL PROTECTED]
Senior Systems Programmer   or  [EMAIL PROTECTED]
Rensselaer Polytechnic Instituteor  [EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-18 Thread Scott Long
On Wed, 19 Nov 2003, Colin Percival wrote:
 At 17:06 18/11/2003 -0700, Scott Long wrote:
 Our rationale for encouraging Gordon is as follows:
 
 1.  4.x upgrade path:  As we approach 5-STABLE, a lot of users might want
  to upgrade from 4-STABLE.  Historically in 4.x, the / partition has
  been very modest in size.  One just simply cannot cram the bloat that
  has grown in 5.x into a 4.x partition scheme.  Of course there is the
  venerable 'dump - clean install - restore' scheme, but we were looking
  for something a little more user-friendly.

Of course, making / dynamic results in added complication of removing
 old libraries from /usr/lib, now that some of them have moved to /lib...

The magic for solving this exact problem already exists in the source
upgrade path.  Fixing this for the binary upgrade path (via sysinstall) is
still under investigation.  It is mitigated by the fact the systems that
are binary-upgraded have their library search path set to look at /lib
first.  Leaving around stale libraries is of course bad, and we will
hopefully come up with a solution soon.


 3.  Binary security updates: there is a lot of interest in providing a
  binary update mechanism for doing security updates.  Having a dynamic
  root means that vulnerable libraries can be updated without having to
  update all of the static binaries that might use them.

As far as I'm concerned, this is a non-issue.  Identifying which static
 binaries need to be replaced is now a solved problem, replacing them is
 easy, and if binary patches are used, there is effectively no impact on
 bandwidth usage either.

Bandwidth is still a concern for a lot of people, and this has the
potential to save significant bandwidth in many situations.


On the issue of performance, however: I know people have benchmarked
 fork-bombs, but has anyone done benchmarks with moderate numbers of
 long-lived, library-intensive, processes?  It seems to me that dynamic
 linking could have caching advantages.

 Colin Percival




I can't comment well here.  To my untrained mind, I would think that more
sharing of libc would help with memory pressure, but I'm just making
simple assumptions.

Scott
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-18 Thread dyson
Scott Long said:
 On Tue, 18 Nov 2003 [EMAIL PROTECTED] wrote:
  The cool thing about properly implemented shared libs is that not everything
  has to be shared.  Even the kernel supports runtime loading/addition
  of modules, and the NSS sounds like a good candidate for a library
  feature.  Burdening everything with the sparse .data associated with
  libc seems to be a waste (but, perhaps the performance that was carefully
  crafted into the codebase is no longer important?)  Really -- it is
  possible that performance isnt' important anymore, and fully accepting
  the ongoing loss of performance for features (without careful tradeoffs)
  might be an appropriate strategy.  (In the case of the NSS stuff, it
  shouldn't require everything to be built dynamic.)
 
 Unfortunately, our dynamic linker prevents dlopen() and friends from
 working in a static binary.  There was some talk about having NSS
 lookups happen through a proxy process, but our implementation just
 plain wasn't done that way, and no one has stepped forward with
 alternate code.
 
It seems like the defect is with the old limitation on the dlopen
things then.  (That limitation has been operative for years.)

Making that decision 6months ago (or whenever the decision was
made) seems to result in sidestepping the best solution (esp
when 6months is alot of time to fix such limitations.) 

Again, I cannot take the responsibility, and mostly trying to
keep 'improvements' from undoing previous work.  I am worried
(really) that there'll be a progressive loss of performance in
order to gain features that could have been implemented with
less impact.  There was ALOT of work in trying to make FreeBSD
very performant, and throwing that away with incrementalism
seems to be wasteful.  I am NOT meaning to negatively criticize,
because that is quite seductive, and I am NOT suggesting that
FreeBSD should have the Plan9 mimalist ethic. :-).

Favorite 'tricky' features will come and go -- but basic system
performance gets harder and hard to recover as time goes on!!!

I guess that all OSes have a size/performance curve where they
get bigger and fatter, SOMETIMES (not always) losing previous
attributes for new features.  Just don't let the performance loss
go too far.  IMO, I wouldn't have supported everything be built
shared if there was a POTENTIAL mitigating solution of a proper
dlopen implementation.  DG would likely have had a 'fit.'

Perhaps there should be a FreeBSD performance (and footprint)
responsibility officer before it becomes too bloated to recover? :-).

John

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-18 Thread Gordon Tetlow
On Tue, Nov 18, 2003 at 08:03:23PM -0500, [EMAIL PROTECTED] wrote:
 
 However, PAM and NSS 'tricks' really seem to be exactly that,
 and certainly worthy of special builds.  However, that isn't
 necessary, yet still not building everything with a shared
 libc.

Things like nss_ldap (which is used *heavily* at my place of employment)
are some reasons that FreeBSD doesn't make it into more places. It was
the reason why FreeBSD isn't being used here. Calling them 'tricks'
(and succumbing to the name calling you wanted to avoid) doesn't change
the fact that every major contender (IRIX, Solaris, Linux to name a few)
all support this feature set.

-gordon


pgp0.pgp
Description: PGP signature


Re: Unfortunate dynamic linking for everything

2003-11-18 Thread dyson
Gordon Tetlow said:
 On Tue, Nov 18, 2003 at 08:03:23PM -0500, [EMAIL PROTECTED] wrote:
  
  However, PAM and NSS 'tricks' really seem to be exactly that,
  and certainly worthy of special builds.  However, that isn't
  necessary, yet still not building everything with a shared
  libc.
 
 Things like nss_ldap (which is used *heavily* at my place of employment)
 are some reasons that FreeBSD doesn't make it into more places. It was
 the reason why FreeBSD isn't being used here. Calling them 'tricks'

Firstly -- I was answering back the 'tricks' comment made that you
had elided :-).  Please quote the message that set-up the context for
the usage.


 (and succumbing to the name calling you wanted to avoid) doesn't change
 the fact that every major contender (IRIX, Solaris, Linux to name a few)
 all support this feature set.
 
As discussed before, it DEFINITELY isn't necessary to dynamically link
EVERYTHING to implement your favorite feature.

Not everyone needs PAM/NSS, even though everyone needs memory management
and scarce resource allocation.  Why build in the overhead for everyone,
and make it UNNCESSARILY worse (e.g. dynamically link libc when you want
NSS or PAM?)  Of course, there was a
development resource limitation, but the decision (discussion) was
made approx 6months ago?  (Enough time to solve the problem without
a GLOBAL performance hit.)

John

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-18 Thread Matthew Dillon

:Our rationale for encouraging Gordon is as follows:
:
:1.  4.x upgrade path:  As we approach 5-STABLE, a lot of users might want
:to upgrade from 4-STABLE.  Historically in 4.x, the / partition has
:been very modest in size.  One just simply cannot cram the bloat that
:has grown in 5.x into a 4.x partition scheme.  Of course there is the
:venerable 'dump - clean install - restore' scheme, but we were looking
:for something a little more user-friendly.

This argument would apply to very old 4.x users but not to anyone who
installed it as of March 2001.  I bumped the nominal size of the
root partition to 128MB in 1.98.2.7 of sysinstall/label.c.

Prior to that Jordan had bumped the root partition size to 100MB
in 1.98.2.3 in March 2001.  It was 50MB before then, which is too small
even for 4.x.

:2.  NSS support out-of-the-box:  Again, this is a user-experience issue
:in that forcing NSS users to recompile world was seen as a potential
:roadblock to them.

Users have to recompile the world anyway, I don't think NSS is an issue.

Or to put it another way:  Nobody in their right mind should be 
contemplating upgrading a 4.x system to 5.x for production purposes.
There will simply be too much 4.x cruft left over.  Upgrading
really requires a wipe and reinstall in this instance.

I seem to recall that the main reason 5.x went to a dynamic /bin was
to work around a terribly broken PAM design.  The other issues were
just eye candy compared to the PAM problems.  Personally speaking, I
think the solution is to fix PAM instead :-)

:3.  Binary security updates: there is a lot of interest in providing a
:binary update mechanism for doing security updates.  Having a dynamic
:root means that vulnerable libraries can be updated without having to
:update all of the static binaries that might use them.

Non-issue if you have an automatic security update mechanism or script
to do the work for the user, which we don't.  Even so, /bin and /sbin are
sufficiently self-contained in the source hierarchy that recompiling and
reinstalling them is not a big deal.

I think the security argument is a red-herring because, frankly,
security problems are far more prevalient with ports and larger services
(Apache, sendmail, sshd, etc... mostly in /usr/bin and /usr/local),
and not as big an issue for /bin and /sbin.

:As for performance, we felt that the typical use of FreeBSD did not fall
:into the category of doing forkbomb performance tests.  While there might
:certainly be users that do continuously create lots of short-lived
:processes, we felt that the above benefits outweighed this, but left the
:door open for tailoring to this need (recompiling world with
:NO_DYNAMICROOT).

fork() is a non-issue (forking overhead for a dynamic executable imposes
a slight time penalty but does not really impose any resource penalty).

However, I personally believe that anything run by a shell script is
an issue in regards to performance, and anything you exec multiple
separate copies of is an issue in regards to memory overhead.  A system
running a large number of JAIL'd environments will wind up with a larger
number of swap-managed dirty pages.

But, all this aside, the reason I am decidedly against a dynamic /bin is
simply one of system recovery and safety.  I've blown up /usr so many 
times over the years that had /bin and /sbin been dynamic I would have 
been in real trouble on many occassions.

If I recall correctly, some people have proposed and/or implemented some
sort of emergency fall back or safety bin in 5.x to make up for this
issue.  I would humbly submit that this simply acknowledges that a dynamic
/bin and /sbin is a bad idea in the first place.

I do not intend to make /bin or /sbin dynamic in DragonFly by default.
I don't mind adding support for those people who want it, but I am not
going to make it the default.

-Matt
Matthew Dillon 
[EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-18 Thread Matthew Dillon
:
:As far as I'm concerned, this is a non-issue.  Identifying which static
: binaries need to be replaced is now a solved problem, replacing them is
: easy, and if binary patches are used, there is effectively no impact on
: bandwidth usage either.
:
:Bandwidth is still a concern for a lot of people, and this has the
:potential to save significant bandwidth in many situations.
:..
:Scott

I would not consider this a viable argument since binary downloads are
usually compressed.  A compressed /bin stacks up as follows (from 4.x):

-rw-r--r--  1 root  wheel  4034560 Nov 18 18:34 /tmp/x1.tar static
-rw-r--r--  1 root  wheel   849920 Nov 18 18:34 /tmp/x2.tar dynamic

-rw-r--r--  1 root  wheel  1860215 Nov 18 18:34 /tmp/x1.tgz static
-rw-r--r--  1 root  wheel   354576 Nov 18 18:34 /tmp/x2.tgz dynamic

So you are talking about 1.5 MBytes less bandwidth, which is nothing
compared to the cost of doing a full install over the net.  Yah, yah,
/sbin too... but you get the idea.

It certainly isn't enough of a difference to justify going to a
dynamic /bin and /sbin.  I'm not saying there aren't other arguments,
just that this isn't one of them.

-Matt
Matthew Dillon 
[EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-18 Thread Matthew Dillon
:/boot has grown quite large too and threatens to be unbounded in size as
:times go on.  Shaving off the 30-40MB of size in /bin and /sbin can
:help alleviate this, even on system formatted in 5.x partition sized.
:...
:This argument wasn't the most compelling one by itself, but it played a
:part in the decision so I included it here.

You aren't saving that much.  I don't have a 5.x box handy but on
a 4.x box a static /bin is only 4MB and /sbin is only 12MB.  The dynamic 
versions will save you around 12MB is my guess.

-Matt

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-18 Thread dyson
Gang,
I suspect that my position has been expressed
adequately.

Further discussion might become divisive, but
a decision that incurs the overhead of performance
or a rebuild on the default user base seems
wrong (JUST MY OPINION.)  It took ALOT of WORK
(person years) to make FreeBSD perform as well
as it does.

BOTH the add-on crew and the general user base can
have the performance and feature set without
rebuilding, but the decision was apparently made
to impose the cost of performance or rebuild and
binary maintenance on the default user base.

It makes more sense to have appropriately
upgraded the system (by the NSS project) to avoid
the performance hit by others and also provide
the feature set.  Apparently (I haven't fully
analysed this) implementing the dlopen stuff for
non-dynamic programs would have helped to mitigate
this issue.  (It might have put more burden on the
NSS/PAM/whatever addon projects, but those are
indeed addons that shouldn't take ANYTHING away
from the rest of the project.)

I am suggesting that the NSS crew and those who
are concerned about performance can BOTH have
the results that they wish for.

'All or nothing' creates divisiveness, and in these
discussions it is TOO EASY to fall into that trap.
I am not suggesting the loss of the new NSS stuff,
but also suggest that ANY loss of performance when
it can be avoided, is unwise.

My opinion is known, and hopefully the loss of
hard earned performance with person-years of work
won't happen as time goes on.  A little loss isn't
that bad, but how much loss is too much loss (esp when
not necessary?)

EOT
John

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-18 Thread Garance A Drosihn
At 6:38 PM -0800 11/18/03, Matthew Dillon wrote:
So you are talking about 1.5 MBytes less bandwidth,
which is nothing compared to the cost of doing a full
install over the net.  Yah, yah, /sbin too... but you
get the idea.
Many freebsd users (me for one) are still living on a modem,
where even one bump of 1.5 meg is a significant issue...
Remember that the issue we're talking about is security
updates, not full system upgrades.  Everyone would want
the security updates, even if they're on a slow link.
--
Garance Alistair Drosehn=   [EMAIL PROTECTED]
Senior Systems Programmer   or  [EMAIL PROTECTED]
Rensselaer Polytechnic Instituteor  [EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Unfortunate dynamic linking for everything

2003-11-18 Thread Robert Watson
On Tue, 18 Nov 2003 [EMAIL PROTECTED] wrote:

 There might be a certain 'coolness' WRT dynamically linking everything,
 but the overhead is certainly measurable.  If the object is to maximally
 'share', then for shells the FreeBSD VM shares maximally without using
 shared libs (perhaps there is a lost know-how about how aggressively the
 FreeBSD VM implements parent/child and exec based sharing?)
 
 I'll try to put together a few simple benchmarks -- mostly from my
 defective memory.  I had recently refreshed myself on this issue (but
 lost again due to disk hardware disasters and being clobbered by vinum
 problems -- which I have given up on.)  Gimme a day or so -- I have
 several other things in my queue. 

I guess one of the key observations to make here is that the vast majority
of applications in FreeBSD have been dynamically linked for years.  The
only thing that has changed recently is a few binaries in /bin and /sbin. 
Of these binaries, the vast majority are never run for most systems, are
run once during boot, or extremely infrequently in response to an
administrative action where minor differences in latency will be
unnoticeable.  This would include applications like ping, mount_*,
fsirand, newfs, swapon, kldload, chflags, rcorder, quotacheck, etc.  The
once during boot case is interesting in the aggregate, but most of the
binaries in question should probably have been linked dynamically long ago
simply because there's no real benefit to making them statically linked.

So I think this leaves three interesting cases: 

(1) Shells, which are run for extended periods of time, and which are
often run in a large number (propotional to #users logged in, or
#windows open).  I'm not to interested in this argument simply because
the applications most people are interested in using FreeBSD for are
already dynamically linked: Apache, databases, perl, XWindows, KDE,
...  The vast majority of long-lived processes are already dynamically
linked.

(2) Shells again, because they will be fork()d and exec()d frequently
during heavily scripted activities, such as system boot, periodic
events, large make jobs, etc.  And presumably the only shell of
interest is sh, although some of the supporting non-builtin binaries
may also be of interest. 

(3) Other binaries, such as mount_*, etc, which aren't run very often, but
when they are run, will be run in aggregate with many other similar
binaries, such as during system boot.  The cost of one binary going
dynamic is presumably extremely small, but if you suddenly make many
binaries dynamic, it's presumably much more noticeable. 

Some macrobenchmark results that would probably be interesting to see
(some of which I know have already been looked at as part of this
discussion): 

- With a move to rcNG (/etc/rc.d), our boot process is now probably quite
  a bit more sensitive to the net performance change from dynamic linking. 
  This would be at least one benchmark that would be interesting to look
  at (and I understand has been looked at by those exploring dynamic
  linking). 

- The impact on large multi-part build tasks, such as buildworld, is also
  interesting.  Turning on process accounting shows that 'sh' is run by
  make(1) once for each task it spawns off during the build.  A
  macrobenchmark would be helpful to look at whether there's a positive or
  negative impact.

Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
[EMAIL PROTECTED]  Network Associates Laboratories


___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to [EMAIL PROTECTED]


  1   2   >