Pádraig Brady wrote: > > You wouldn't want multiple threads/processes fighting over > the disk head so you would do something like: > > find /disk1 | xargs md5sum & find /disk2 | xargs md5sum > > Note if we're piping/redirecting the output of the above > then we must be careful to line buffer the output from md5sum > so that it's not interspersed. Hmm I wonder should > we linebuffer the output from *sum by default.
In the attached patch, I've changed the default buffering to line buffered to address the above issue. For standard size files there is a 2% performance drop. cheers, Pádraig. p.s. I'll look at bypassing stdio on input to see if I can get at least the 2% back
>From 0db7057c6256d9cd25e988b3fe23e97a0e30f717 Mon Sep 17 00:00:00 2001 From: =?utf-8?q?P=C3=A1draig=20Brady?= <[email protected]> Date: Tue, 20 Oct 2009 19:19:58 +0100 Subject: [PATCH] md5sum, sha*sum, sum: line buffer the outputted checksums * src/md5sum.c (main): Set stdout to line buffered mode to ensure parallel running instances don't intersperse their output. This adds 5% to the run time in the worst case of many zero length files, or 2% with standard file sizes. * src/sum.c (main): Likewise. --- src/md5sum.c | 6 ++++-- src/sum.c | 4 ++++ 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/src/md5sum.c b/src/md5sum.c index aa2a144..b7db03e 100644 --- a/src/md5sum.c +++ b/src/md5sum.c @@ -513,7 +513,6 @@ digest_check (const char *checkfile_name) if (!status_only) { printf (_("%s: FAILED open or read\n"), filename); - fflush (stdout); } } else @@ -539,7 +538,6 @@ digest_check (const char *checkfile_name) printf ("%s: %s\n", filename, _("FAILED")); else if (!quiet) printf ("%s: %s\n", filename, _("OK")); - fflush (stdout); } } } @@ -619,6 +617,10 @@ main (int argc, char **argv) atexit (close_stdout); + /* Line buffer stdout to ensure lines are written atomically and immediately + so that processes running in parallel do not intersperse their output. */ + setvbuf (stdout, NULL, _IOLBF, 0); + while ((opt = getopt_long (argc, argv, "bctw", long_options, NULL)) != -1) switch (opt) { diff --git a/src/sum.c b/src/sum.c index 91d7f34..f0e0cc0 100644 --- a/src/sum.c +++ b/src/sum.c @@ -233,6 +233,10 @@ main (int argc, char **argv) atexit (close_stdout); + /* Line buffer stdout to ensure lines are written atomically and immediately + so that processes running in parallel do not intersperse their output. */ + setvbuf (stdout, NULL, _IOLBF, 0); + have_read_stdin = false; while ((optc = getopt_long (argc, argv, "rs", longopts, NULL)) != -1) -- 1.6.2.5
