On Fri, Nov 19, 2021 at 08:40:37PM -0700, Theo de Raadt wrote:
> Scott Cheloha <[email protected]> wrote:
>
> > On Fri, Nov 19, 2021 at 07:42:18PM -0700, Theo de Raadt wrote:
> > > +#include <sys/param.h> /* for MAXBSIZE */
> > >
> > > No way, that is non-POSIX namespace. We are going in precisely the
> > > opposite
> > > direction, eliminating this non-portability from the tree.
> > >
> > > No biblical scrolls have it written "all programs must use buffer sizes
> > > decided by a system header file".
> > >
> > > If you want 64*1024 in this specific program then just say 64*1024.
> >
> > Is there a nicer way to pick a "reasonable" buffer size when we just
> > want to move as many bytes as possible on a given platform without
> > hogging the machine?
>
> Pick a number.
>
> Use a number.
>
> [...]
>
> The right buffer size should be chosen for in a specific program. It is
> a number which is small enough to not hog the system, yet large enough
> to gain reasonable benefits versus the cost of servicing/faulting
> memory. Obviously the number chosen will change during the era, unlike
> MAXBSIZE and BUFSIZ which are going on 30 years old and were chosen by
> someone sharing a vax 11/780 (or even smaller machine).
The point of diminishing returns on my machine is 128K.
I ran this synthetic benchmark on doubling buffer sizes from 8K up
through 1M:
for i in $(jot 100); do
dd if=/dev/zero bs=1M count=1K 2>/dev/null \
| nanotime tee /dev/null > /dev/null 2>>time.$size
done
On an otherwise quiet machine I got these results:
x time.8192
+ time.16384
* time.32768
% time.65536
# time.131072
@ time.262144
& time.524288
$ time.1048576
N Min Max Median Avg Stddev
x 100 0.94371683 1.0566744 0.95342365 0.97797553 0.031421145
+ 100 0.61271914 0.64374531 0.63532658 0.6346069 0.0055198003
Difference at 99.5% confidence
-0.343369 +/- 0.00985781
-35.1101% +/- 1.00798%
(Student's t, pooled s = 0.0225583)
* 100 0.41367684 0.44712464 0.43615004 0.43494362 0.0071550676
Difference at 99.5% confidence
-0.543032 +/- 0.00995768
-55.5261% +/- 1.01819%
(Student's t, pooled s = 0.0227869)
% 100 0.26071834 0.28996026 0.28033773 0.27985476 0.0057852679
Difference at 99.5% confidence
-0.698121 +/- 0.00987233
-71.3843% +/- 1.00947%
(Student's t, pooled s = 0.0225916)
# 100 0.24090659 0.26264406 0.25455999 0.25441678 0.0044142139
Difference at 99.5% confidence
-0.723559 +/- 0.00980448
-73.9854% +/- 1.00253%
(Student's t, pooled s = 0.0224363)
@ 100 0.24297249 0.26462487 0.25686952 0.25593464 0.0047320572
Difference at 99.5% confidence
-0.722041 +/- 0.00981862
-73.8302% +/- 1.00397%
(Student's t, pooled s = 0.0224687)
& 100 0.24499786 0.26534237 0.25667553 0.2562827 0.0039443348
Difference at 99.5% confidence
-0.721693 +/- 0.00978533
-73.7946% +/- 1.00057%
(Student's t, pooled s = 0.0223925)
$ 100 0.24157916 0.2657958 0.25527962 0.25451313 0.0049337784
Difference at 99.5% confidence
-0.723462 +/- 0.0098281
-73.9755% +/- 1.00494%
(Student's t, pooled s = 0.0224903)
Beyond 128K you are buying little to nothing with the additional
memory:
x time.131072
+ time.262144
* time.524288
% time.1048576
N Min Max Median Avg Stddev
x 100 0.24090659 0.26264406 0.25455999 0.25441678 0.0044142139
+ 100 0.24297249 0.26462487 0.25686952 0.25593464 0.0047320572
No difference proven at 99.5% confidence
* 100 0.24499786 0.26534237 0.25667553 0.2562827 0.0039443348
Difference at 99.5% confidence
0.00186592 +/- 0.00182919
0.73341% +/- 0.718975%
(Student's t, pooled s = 0.00418587)
% 100 0.24157916 0.2657958 0.25527962 0.25451313 0.0049337784
No difference proven at 99.5% confidence
I think the .73% improvement at 512KB is a fluke. Even if it isn't,
quadruple the memory for less than 1% improvement isn't worth it.
So, is 128K ok? Any objections?
Index: tee.c
===================================================================
RCS file: /cvs/src/usr.bin/tee/tee.c,v
retrieving revision 1.13
diff -u -p -r1.13 tee.c
--- tee.c 21 Nov 2021 16:15:43 -0000 1.13
+++ tee.c 22 Nov 2021 00:04:24 -0000
@@ -43,6 +43,8 @@
#include <string.h>
#include <unistd.h>
+#define SIZE (128 * 1024)
+
struct list {
SLIST_ENTRY(list) next;
int fd;
@@ -67,9 +69,10 @@ main(int argc, char *argv[])
{
struct list *p;
int fd;
+ size_t size;
ssize_t n, rval, wval;
int append, ch, exitval;
- char buf[8192];
+ char *buf;
if (pledge("stdio wpath cpath", NULL) == -1)
err(1, "pledge");
@@ -109,6 +112,10 @@ main(int argc, char *argv[])
if (pledge("stdio", NULL) == -1)
err(1, "pledge");
+ size = SIZE;
+ buf = malloc(size);
+ if (buf == NULL)
+ err(1, NULL);
while ((rval = read(STDIN_FILENO, buf, sizeof(buf))) > 0) {
SLIST_FOREACH(p, &head, next) {
for (n = 0; n < rval; n += wval) {
@@ -121,6 +128,7 @@ main(int argc, char *argv[])
}
}
}
+ free(buf);
if (rval == -1) {
warn("read");
exitval = 1;
