Re: svn commit: r341803 - head/libexec/rc

[Simon J. Gerraty]

Just caught the tail of this thread so sorry for
chiming in from the peanut gallery...

blah | while read x; do ...; done

behaves very differently to the for loop variant in that the body of the
loop runs in a sub-shell and thus cannot affect the outer scope.
In many cases that's exactly what you want.
Sometimes though, it isn't.

Conrad Meyer  wrote:
> On Tue, Dec 11, 2018 at 2:42 PM Devin Teske  wrote:
> > In that case, would it be appropriate to say that:
> >
> > blah | while read x; do ...; done
> >
> > Is always more efficiently written as:
> >
> > IFS=$'\n'
> > for x in $( blah ); do ...; done
> 
> I don't know.  The suggestion came from jilles@, who is much more
> familiar with sh(1) than I am.
> 
> My understanding is that it's important that 'set -o noglob' is set,
> or else 'blah' lines that include globs may be evaluated against the
> filesystem.  There is also a caveat if 'blah' is the 'set' command, or
> similar, in that IFS' own value itself will be split across multiple
> for loop iteration 'x' values ("IFS='", "'").
> 
> I would hesitate to say "always" given my limited understanding of the
> shell, but it might be true.
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"

Re: svn commit: r341803 - head/libexec/rc

[Conrad Meyer]

On Tue, Dec 11, 2018 at 2:42 PM Devin Teske  wrote:
> In that case, would it be appropriate to say that:
>
> blah | while read x; do ...; done
>
> Is always more efficiently written as:
>
> IFS=$'\n'
> for x in $( blah ); do ...; done

I don't know.  The suggestion came from jilles@, who is much more
familiar with sh(1) than I am.

My understanding is that it's important that 'set -o noglob' is set,
or else 'blah' lines that include globs may be evaluated against the
filesystem.  There is also a caveat if 'blah' is the 'set' command, or
similar, in that IFS' own value itself will be split across multiple
for loop iteration 'x' values ("IFS='", "'").

I would hesitate to say "always" given my limited understanding of the
shell, but it might be true.

Best,
Conrad
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"

Re: svn commit: r341803 - head/libexec/rc

[Devin Teske]

> On Dec 11, 2018, at 1:54 PM, Conrad Meyer  wrote:
> 
> On Tue, Dec 11, 2018 at 12:35 PM Devin Teske  wrote:
>>> On Dec 11, 2018, at 11:57 AM, Conrad Meyer  wrote:
>>> Is there any interest in a tee(2)-like syscall?
>>> 
>> 
>> Linux has vmsplice(2). I know jmg@ also expressed interest in having a
>> vmsplice in FreeBSD.
> 
> Sure; they're related.  See also splice(2).  But tee(2) is probably
> the one that would be most useful here.
> 
>> As for sh not being able to read more than a single byte at a time because
>> it could be reading from a pipe, what if it read into a buffer and returned
>> a line from the buffer. A subsequent read would return more data from the
>> buffer, ad nauseam until the buffer runs out -- in which case another chunk
>> is read to augment the data.
>> 
>> This buffer could be expunged when stdin collapses (e.g., when the sub-
>> shell completes.
> 
> Yeah, this is basically what "buffered input" means.  An example of
> the problem with the naive solution is the scenario Warner pasted a
> few emails ago.  Take:
> 
>foo | (read bar; baz)
> 
> (Where 'baz' is not a built-in.)  To implement this correctly with
> buffered 'read,' you have to somehow flush any input in the buffer
> beyond the end of the first line to the pipe that will be baz's stdin,
> as well as with the remaining contents of the pipe from foo (which
> may be indefinite).  That is what I suggested above as (A).
> 
> One big caveat is that you basically have to spawn a thread or process
> to keep feeding the input from the first pipe to the magical implicit
> pipe.  This overhead can be avoided readily in most scenarios if only
> the 'read' command is buffered.  Also, the described (read bar; baz)
> sub-program is a fairly odd construction, and the feeding isn't needed
> if no non-builtin programs after 'read' will access stdin.
> 
> If it helps paint a more concrete picture, imagine 'foo' as 'yes' and
> 'baz' as 'pv > /dev/null' or something (i.e., indefinite data source,
> indefinite data sink).

Thanks, this definitely illustrates the trouble.

In that case, would it be appropriate to say that:

blah | while read x; do ...; done

Is always more efficiently written as:

IFS=$'\n'
for x in $( blah ); do ...; done

?
-- 
Devin
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"

Re: svn commit: r341803 - head/libexec/rc

[Conrad Meyer]

On Tue, Dec 11, 2018 at 12:35 PM Devin Teske  wrote:
> > On Dec 11, 2018, at 11:57 AM, Conrad Meyer  wrote:
> > Is there any interest in a tee(2)-like syscall?
> >
>
> Linux has vmsplice(2). I know jmg@ also expressed interest in having a
> vmsplice in FreeBSD.

Sure; they're related.  See also splice(2).  But tee(2) is probably
the one that would be most useful here.

> As for sh not being able to read more than a single byte at a time because
> it could be reading from a pipe, what if it read into a buffer and returned
> a line from the buffer. A subsequent read would return more data from the
> buffer, ad nauseam until the buffer runs out -- in which case another chunk
> is read to augment the data.
>
> This buffer could be expunged when stdin collapses (e.g., when the sub-
> shell completes.

Yeah, this is basically what "buffered input" means.  An example of
the problem with the naive solution is the scenario Warner pasted a
few emails ago.  Take:

foo | (read bar; baz)

(Where 'baz' is not a built-in.)  To implement this correctly with
buffered 'read,' you have to somehow flush any input in the buffer
beyond the end of the first line to the pipe that will be baz's stdin,
 as well as with the remaining contents of the pipe from foo (which
may be indefinite).  That is what I suggested above as (A).

One big caveat is that you basically have to spawn a thread or process
to keep feeding the input from the first pipe to the magical implicit
pipe.  This overhead can be avoided readily in most scenarios if only
the 'read' command is buffered.  Also, the described (read bar; baz)
sub-program is a fairly odd construction, and the feeding isn't needed
if no non-builtin programs after 'read' will access stdin.

If it helps paint a more concrete picture, imagine 'foo' as 'yes' and
'baz' as 'pv > /dev/null' or something (i.e., indefinite data source,
indefinite data sink).

Best,
Conrad
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"

Re: svn commit: r341803 - head/libexec/rc

[Poul-Henning Kamp]

In message <20181212071210.l...@besplex.bde.org>, Bruce Evans writes:

>But software bloat is now outrunning CPU speed increases.  Some bandwidths
>for reading 1 byte at a time run today on the same 2GHz CPU i386 UP hardware
>
>linux-2.1.128 kernel built in 1998: 2500k/sec
>linux-2.4.0t8 kernel built in 2000: 1720k/sec
>linux-2.6.10  kernel built in 2004: 1540k/sec
>FreeBSD-4 kernel built in 2007:  680k/sec
>FreeBSD-~5.2  kernel built in 2018:  700k/sec
>FreeBSD-11kernel built in 2018:  720k/sec (SMP kernel)
>FreeBSD-pre12 kernel built in 2018:  540k/sec (SMP kernel)
>FreeBSD-13kernel built in 2018:  170k/sec (SMP kernel)

It is not just software bloat, it is also caused by the deeper and
deeper pile of kludges between what goes for a "CPU" these days and
what counts as "RAM".

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"

Re: svn commit: r341803 - head/libexec/rc

[Bruce Evans]

On Tue, 11 Dec 2018, John Baldwin wrote:


On 12/11/18 9:40 AM, Devin Teske wrote:

...
Thank you for the background which was lost by the time I got to the phab.

I can't help but ask though,...

If it was noticed that read(2) processes the stream one byte at a time,
why not just optimize read(2)?

I'm afraid of the prospect of having to hunt down every instance of while-read,
but if we can fix the underlying read(2) inefficiency then we make while-read 
OK.


It's a system call.  A CPU emulator has to do a lot of work for a system call
because it involves two mode switches (user -> kernel and back again).  You
can't "fix" that as it's just a part of the CPU architecture.  There's a reason
that stdio uses buffering by default, it's because system calls have overhead.
The 'read' builtin in sh can't use buffering, so it is always going to be
inefficient.


Syscalls have always been well known to be slow, but slowness is relative.
CPUs are thousands of times slower that in 1980, so systems should be able
to crunch through a few MB of data read 1 byte at a time fast enough that
no one notices the slowness, even when emulated.

But software bloat is now outrunning CPU speed increases.  Some bandwidths
for reading 1 byte at a time run today on the same 2GHz CPU i386 UP hardware

linux-2.1.128 kernel built in 1998: 2500k/sec
linux-2.4.0t8 kernel built in 2000: 1720k/sec
linux-2.6.10  kernel built in 2004: 1540k/sec
FreeBSD-4 kernel built in 2007:  680k/sec
FreeBSD-~5.2  kernel built in 2018:  700k/sec
FreeBSD-11kernel built in 2018:  720k/sec (SMP kernel)
FreeBSD-pre12 kernel built in 2018:  540k/sec (SMP kernel)
FreeBSD-13kernel built in 2018:  170k/sec (SMP kernel)

This is with all recent security-related pessimizations like ibrs turned off,
except the main one for i386 is using separate address spaces for the kernel
and userland and this cannot be turned off.  This is what gives most of the
recent slowdown factor of more than 3.  This slowdown factor is close to
3 for large block sizes, since read() is so slow that it takes about the
same time for all block sizes below a few hundred bytes.

Network bandwidth has similar slowdowns for small packets starting in
FreeBSD-4 (the old Linuxes have low enough syscall overhead for the NIC
to saturate at 640 kpps before the CPU or 1 Gbps ethernet saturates).
amd64 doesn't have the 3-fold slowdown from separate address spaces.

Optimizing syscalls is not very important, but it is convenient for
applications with bad buffering to not be very slow, and it is annoying
to lose benchmarks by a factor of 2 in 1998 and 10 now.

Bruce
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"

Re: svn commit: r341803 - head/libexec/rc

[Devin Teske]

> On Dec 11, 2018, at 11:57 AM, Conrad Meyer  wrote:
> 
> On Tue, Dec 11, 2018 at 10:04 AM Warner Losh  wrote:
>> On Tue, Dec 11, 2018, 9:55 AM John Baldwin >> 
>>> The 'read' builtin in sh can't use buffering, so it is always going to be 
>>> slow
>> 
>> It can't use it because of pipes. The example from the parts of this that 
>> was on IRC was basically:
>> 
>> foo | (read bar; baz)
>> 
>> Which reads one line into the bar variable and then sends the rest to the 
>> bar command.
> 
> It can't trivially, but it's not impossible.  sh could play games and
> buffer its own use of stdin, and then open a fresh pipe for stdin of
> subsequent non-builtins, writing out unused portions of the buffer.[A]
> 
> Some other alternatives that would require kernel support but are
> things we've talked about doing in the kernel before anyway:
> 
> * If we had something like eBPF programs attached to IO, maybe sh's
> read built-in could push a small eBPF program into the kernel that
> determined how many bytes could be read from the pipe in a single
> syscall without reading too far.  It's fairly trivial.  Simply
> returning a number of bytes up to and including the first '\n' would
> be a fine, if sometimes conservative amount.  (Input lines can be
> continued with a trailing backslash, except in -r mode, but as a
> first-cut approximation, reading-until-newline is probably good
> enough.)[B]
> 
> * Heck, even just a read_until_newline(2) syscall would work and
> probably be more broadly useful than just sh(1).  I don't think it
> passes the sniff test — not general enough, and probably not something
> you want beginners stumbling across instead of fgets(3) — but it'd be
> fine, and there are other pipe-abusing programs that care about
> reading ASCII text lines without overconsuming input than just
> sh(1).[C]
> 
> * If we had something like Linux's tee(2) system call (which is as it
> sounds — tee(1) for pipes), sh(1)'s read built-in could tee(2) for
> buffering without impacting stdin, and read(2) stdin only when it knew
> how many bytes were consumed (or when the pipe buffer became full).[D]
> 
> I suspect (C) would be the easiest to implement correctly, followed by
> (D).  (B) is requires some architectural design and bikeshedding and
> the details on the kernel side are tricky.  (A) would be a little
> tricky and probably require extensive changes to sh(1) itself, which
> is a risk to the base system.  But it would not impact the kernel.
> 
> Is there any interest in a tee(2)-like syscall?
> 

Linux has vmsplice(2). I know jmg@ also expressed interest in having a
vmsplice in FreeBSD.

As for sh not being able to read more than a single byte at a time because
it could be reading from a pipe, what if it read into a buffer and returned
a line from the buffer. A subsequent read would return more data from the
buffer, ad nauseam until the buffer runs out -- in which case another chunk
is read to augment the data.

This buffer could be expunged when stdin collapses (e.g., when the sub-
shell completes.
-- 
Devin
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"

Re: svn commit: r341803 - head/libexec/rc

[Conrad Meyer]

On Tue, Dec 11, 2018 at 10:04 AM Warner Losh  wrote:
> On Tue, Dec 11, 2018, 9:55 AM John Baldwin >
>> The 'read' builtin in sh can't use buffering, so it is always going to be 
>> slow
>
> It can't use it because of pipes. The example from the parts of this that was 
> on IRC was basically:
>
> foo | (read bar; baz)
>
> Which reads one line into the bar variable and then sends the rest to the bar 
> command.

It can't trivially, but it's not impossible.  sh could play games and
buffer its own use of stdin, and then open a fresh pipe for stdin of
subsequent non-builtins, writing out unused portions of the buffer.[A]

Some other alternatives that would require kernel support but are
things we've talked about doing in the kernel before anyway:

* If we had something like eBPF programs attached to IO, maybe sh's
read built-in could push a small eBPF program into the kernel that
determined how many bytes could be read from the pipe in a single
syscall without reading too far.  It's fairly trivial.  Simply
returning a number of bytes up to and including the first '\n' would
be a fine, if sometimes conservative amount.  (Input lines can be
continued with a trailing backslash, except in -r mode, but as a
first-cut approximation, reading-until-newline is probably good
enough.)[B]

* Heck, even just a read_until_newline(2) syscall would work and
probably be more broadly useful than just sh(1).  I don't think it
passes the sniff test — not general enough, and probably not something
you want beginners stumbling across instead of fgets(3) — but it'd be
fine, and there are other pipe-abusing programs that care about
reading ASCII text lines without overconsuming input than just
sh(1).[C]

* If we had something like Linux's tee(2) system call (which is as it
sounds — tee(1) for pipes), sh(1)'s read built-in could tee(2) for
buffering without impacting stdin, and read(2) stdin only when it knew
how many bytes were consumed (or when the pipe buffer became full).[D]

I suspect (C) would be the easiest to implement correctly, followed by
(D).  (B) is requires some architectural design and bikeshedding and
the details on the kernel side are tricky.  (A) would be a little
tricky and probably require extensive changes to sh(1) itself, which
is a risk to the base system.  But it would not impact the kernel.

Is there any interest in a tee(2)-like syscall?

Thanks,
Conrad
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"

Re: svn commit: r341803 - head/libexec/rc

[Conrad Meyer]

On Tue, Dec 11, 2018 at 9:24 AM John Baldwin  wrote:
> list_vars is rarely used outside of
> 'netif', so it probably doesn't make a measurable difference on bare metal.

To clarify: It likely doesn't make a measurable difference *to boot
times* on bare metal, but even on amd64 bare metal the time of
list_vars itself is readily measurable; the 71% reduction quoted in
the commit log was measured on amd64.

Best,
Conrad
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"

Re: svn commit: r341803 - head/libexec/rc

[Warner Losh]

On Tue, Dec 11, 2018, 9:55 AM John Baldwin  The 'read' builtin in sh can't use buffering, so it is always going to be
> slow
>

It can't use it because of pipes. The example from the parts of this that
was on IRC was basically:

foo | (read bar; baz)

Which reads one line into the bar variable and then sends the rest to the
bar command.

Warner

>
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"

Re: svn commit: r341803 - head/libexec/rc

[John Baldwin]

On 12/11/18 9:40 AM, Devin Teske wrote:
> 
> 
>> On Dec 11, 2018, at 9:23 AM, John Baldwin > > wrote:
>>
>> On 12/10/18 5:38 PM, Conrad Meyer wrote:
>>> Author: cem
>>> Date: Tue Dec 11 01:38:50 2018
>>> New Revision: 341803
>>> URL: https://svnweb.freebsd.org/changeset/base/341803
>>>
>>> Log:
>>>  rc.subr: Implement list_vars without using 'read'
>>>
>>>  'read' pessimistically read(2)s one byte at a time, which can be quite
>>>  silly for large environments in slow emulators.
>>>
>>>  In my boring user environment, truss shows that the number of read()
>>>  syscalls to source rc.subr and invoke list_vars is reduced by something 
>>> like
>>>  3400 to 60.  ministat(1) shows a significant time difference of about -71%
>>>  for my environment.
>>>
>>>  Suggested by:jilles
>>>  Discussed with:dteske, jhb, jilles
>>>  Differential Revision:https://reviews.freebsd.org/D18481
>>
>> For some background, one my colleagues reported that it was taking hours in
>> (an admittedly slow) CPU simulator to get through '/etc/rc.d/netif start'.
>> I ended up running that script under truss in a RISC-V qemu machine.  The
>> entire run took 212 seconds (truss did slow it down quite a bit).  Of that
>> 212 seconds, the read side of each list_vars invocation took ~25.5 seconds,
>> and with lo0 and vtnet0 there were 8 list_vars invocations, so 204 out of
>> the 212 seconds were spent in the single-byte read() syscalls in 'while 
>> read'.
>>
>> Even on qemu without truss during bootup 'netif start' took a couple of
>> seconds (long enough to get 2-3 Ctrl-T's in) before this change and is now
>> similar to bare metal with the change.  list_vars is rarely used outside of
>> 'netif', so it probably doesn't make a measurable difference on bare metal.
>>
> 
> Thank you for the background which was lost by the time I got to the phab.
> 
> I can't help but ask though,...
> 
> If it was noticed that read(2) processes the stream one byte at a time,
> why not just optimize read(2)?
> 
> I'm afraid of the prospect of having to hunt down every instance of 
> while-read,
> but if we can fix the underlying read(2) inefficiency then we make while-read 
> OK.

It's a system call.  A CPU emulator has to do a lot of work for a system call
because it involves two mode switches (user -> kernel and back again).  You
can't "fix" that as it's just a part of the CPU architecture.  There's a reason
that stdio uses buffering by default, it's because system calls have overhead.
The 'read' builtin in sh can't use buffering, so it is always going to be
inefficient.

-- 
John Baldwin


___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"

Re: svn commit: r341803 - head/libexec/rc

[Devin Teske]

> On Dec 11, 2018, at 9:23 AM, John Baldwin  wrote:
> 
> On 12/10/18 5:38 PM, Conrad Meyer wrote:
>> Author: cem
>> Date: Tue Dec 11 01:38:50 2018
>> New Revision: 341803
>> URL: https://svnweb.freebsd.org/changeset/base/341803
>> 
>> Log:
>>  rc.subr: Implement list_vars without using 'read'
>> 
>>  'read' pessimistically read(2)s one byte at a time, which can be quite
>>  silly for large environments in slow emulators.
>> 
>>  In my boring user environment, truss shows that the number of read()
>>  syscalls to source rc.subr and invoke list_vars is reduced by something like
>>  3400 to 60.  ministat(1) shows a significant time difference of about -71%
>>  for my environment.
>> 
>>  Suggested by:   jilles
>>  Discussed with: dteske, jhb, jilles
>>  Differential Revision:  https://reviews.freebsd.org/D18481
> 
> For some background, one my colleagues reported that it was taking hours in
> (an admittedly slow) CPU simulator to get through '/etc/rc.d/netif start'.
> I ended up running that script under truss in a RISC-V qemu machine.  The
> entire run took 212 seconds (truss did slow it down quite a bit).  Of that
> 212 seconds, the read side of each list_vars invocation took ~25.5 seconds,
> and with lo0 and vtnet0 there were 8 list_vars invocations, so 204 out of
> the 212 seconds were spent in the single-byte read() syscalls in 'while read'.
> 
> Even on qemu without truss during bootup 'netif start' took a couple of
> seconds (long enough to get 2-3 Ctrl-T's in) before this change and is now
> similar to bare metal with the change.  list_vars is rarely used outside of
> 'netif', so it probably doesn't make a measurable difference on bare metal.
> 

Thank you for the background which was lost by the time I got to the phab.

I can't help but ask though,...

If it was noticed that read(2) processes the stream one byte at a time,
why not just optimize read(2)?

I'm afraid of the prospect of having to hunt down every instance of while-read,
but if we can fix the underlying read(2) inefficiency then we make while-read 
OK.
-- 
Devin
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"

Re: svn commit: r341803 - head/libexec/rc

[John Baldwin]

On 12/10/18 5:38 PM, Conrad Meyer wrote:
> Author: cem
> Date: Tue Dec 11 01:38:50 2018
> New Revision: 341803
> URL: https://svnweb.freebsd.org/changeset/base/341803
> 
> Log:
>   rc.subr: Implement list_vars without using 'read'
>   
>   'read' pessimistically read(2)s one byte at a time, which can be quite
>   silly for large environments in slow emulators.
>   
>   In my boring user environment, truss shows that the number of read()
>   syscalls to source rc.subr and invoke list_vars is reduced by something like
>   3400 to 60.  ministat(1) shows a significant time difference of about -71%
>   for my environment.
>   
>   Suggested by:   jilles
>   Discussed with: dteske, jhb, jilles
>   Differential Revision:  https://reviews.freebsd.org/D18481

For some background, one my colleagues reported that it was taking hours in
(an admittedly slow) CPU simulator to get through '/etc/rc.d/netif start'.
I ended up running that script under truss in a RISC-V qemu machine.  The
entire run took 212 seconds (truss did slow it down quite a bit).  Of that
212 seconds, the read side of each list_vars invocation took ~25.5 seconds,
and with lo0 and vtnet0 there were 8 list_vars invocations, so 204 out of
the 212 seconds were spent in the single-byte read() syscalls in 'while read'.

Even on qemu without truss during bootup 'netif start' took a couple of
seconds (long enough to get 2-3 Ctrl-T's in) before this change and is now
similar to bare metal with the change.  list_vars is rarely used outside of
'netif', so it probably doesn't make a measurable difference on bare metal.

-- 
John Baldwin


___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"