Re: [RFC] Force stdio output streams to line-buffered mode
On Saturday, February 19, 2011 1:50:43 pm Jeremie Le Hen wrote: Hi, I've been annoyed multiple time when running a command such like iostat -x 1 | grep -v ad10 | cat -n The problem stems from two factors: - grep's stdio sees that its stdout is not a terminal, so stdout is full buffered and not line-buffered; - iostat produces output too slowly so the aforementioned buffer takes numerous seconds to be filled and flushed to the last command. This problems is not specific to FreeBSD, it is actually a consequence of POSIX specification. I've checked this on Solaris and Linux. I've attached a small patch for stdio, so if the environment variable STDIO_IOLBF is set, the output streams will be line-oriented by default. iostat -x 1 | env STDIO_IOLBF=1 grep -v ad10 | cat -n Before send it as a PR, I would like to hear your comments about this, especially: - the variable name (no bikeshed please, I just ask this if there is a naming convention I'm not aware of); - the documentation: I've put a hint in stdio(3) manpage and put the full explanation in setvbuf(3). Many people would find this useful I think. -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [RFC] Force stdio output streams to line-buffered mode
On Sat, Feb 19, 2011 at 02:37:29PM -0600, Matthew D. Fuller wrote: On Sat, Feb 19, 2011 at 07:50:43PM +0100 I heard the voice of Jeremie Le Hen, and lo! it spake thus: I've attached a small patch for stdio, so if the environment variable STDIO_IOLBF is set, the output streams will be line-oriented by default. iostat -x 1 | env STDIO_IOLBF=1 grep -v ad10 | cat -n I've no real comment on anything else (sounds like an interesting hack, whatever else), but just for this particular case, you know that grep has a --line-buffered arg, right? Yes indeed, my example wasn't especially smart :). Actually, I often stumble on this problem with an awk script I use to columize output. -- Jeremie Le Hen Humans are born free and equal. But some are more equal than others. Coluche ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
[RFC] Force stdio output streams to line-buffered mode
Hi,
I've been annoyed multiple time when running a command such like
iostat -x 1 | grep -v ad10 | cat -n
The problem stems from two factors:
- grep's stdio sees that its stdout is not a terminal, so stdout is
full buffered and not line-buffered;
- iostat produces output too slowly so the aforementioned buffer takes
numerous seconds to be filled and flushed to the last command.
This problems is not specific to FreeBSD, it is actually a consequence
of POSIX specification. I've checked this on Solaris and Linux.
I've attached a small patch for stdio, so if the environment variable
STDIO_IOLBF is set, the output streams will be line-oriented by default.
iostat -x 1 | env STDIO_IOLBF=1 grep -v ad10 | cat -n
Before send it as a PR, I would like to hear your comments about this,
especially:
- the variable name (no bikeshed please, I just ask this if there is a
naming convention I'm not aware of);
- the documentation: I've put a hint in stdio(3) manpage and put the
full explanation in setvbuf(3).
Thanks.
Regards,
--
Jeremie Le Hen
Humans are born free and equal. But some are more equal than others.
Coluche
diff -rup /usr/src.orig/lib/libc/stdio/makebuf.c /usr/src/lib/libc/stdio/makebuf.c
--- /usr/src.orig/lib/libc/stdio/makebuf.c 2009-08-03 10:13:06.0 +0200
+++ /usr/src/lib/libc/stdio/makebuf.c 2011-02-19 19:09:56.0 +0100
@@ -59,6 +59,7 @@ __smakebuf(fp)
FILE *fp;
{
void *p;
+ char *bmode;
int flags;
size_t size;
int couldbetty;
@@ -79,7 +80,8 @@ __smakebuf(fp)
flags |= __SMBF;
fp-_bf._base = fp-_p = p;
fp-_bf._size = size;
- if (couldbetty isatty(fp-_file))
+ if (((bmode = getenv(STDIO_IOLBF)) bmode[0] != '\0') ||
+ (couldbetty isatty(fp-_file)))
flags |= __SLBF;
fp-_flags |= flags;
}
diff -rup /usr/src.orig/lib/libc/stdio/setbuf.3 /usr/src/lib/libc/stdio/setbuf.3
--- /usr/src.orig/lib/libc/stdio/setbuf.3 2009-08-03 10:13:06.0 +0200
+++ /usr/src/lib/libc/stdio/setbuf.3 2011-02-19 19:09:13.0 +0100
@@ -79,7 +79,9 @@ and an optimally-sized buffer is obtaine
If a stream refers to a terminal
(as
.Dv stdout
-normally does) it is line buffered.
+normally does), or the environment variable
+.Ev STDIO_IOLBF
+is set, it is line buffered.
The standard error stream
.Dv stderr
is always unbuffered.
@@ -176,6 +178,12 @@ The
function returns what the equivalent
.Fn setvbuf
would have returned.
+.Sh ENVIRONMENT
+.Bl -tag -width .Ev STDIO_IOLBF
+If the environment variable
+.Ev STDIO_IOLBF
+is set, output streams will be line-buffered by default
+even when not referring to a terminal.
.Sh SEE ALSO
.Xr fclose 3 ,
.Xr fopen 3 ,
diff -rup /usr/src.orig/lib/libc/stdio/stdio.3 /usr/src/lib/libc/stdio/stdio.3
--- /usr/src.orig/lib/libc/stdio/stdio.3 2009-08-03 10:13:06.0 +0200
+++ /usr/src/lib/libc/stdio/stdio.3 2011-02-19 12:56:00.0 +0100
@@ -137,7 +137,8 @@ an interactive or
.Dq terminal
device, as determined by the
.Xr isatty 3
-function.
+function (this can be overriden with an environment variable, see
+.Xr setvbuf 3 ) .
In fact,
.Em all
freshly-opened streams that refer to terminal devices
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [RFC] Force stdio output streams to line-buffered mode
On Sat, Feb 19, 2011 at 07:50:43PM +0100 I heard the voice of Jeremie Le Hen, and lo! it spake thus: I've attached a small patch for stdio, so if the environment variable STDIO_IOLBF is set, the output streams will be line-oriented by default. iostat -x 1 | env STDIO_IOLBF=1 grep -v ad10 | cat -n I've no real comment on anything else (sounds like an interesting hack, whatever else), but just for this particular case, you know that grep has a --line-buffered arg, right? -- Matthew Fuller (MF4839) | fulle...@over-yonder.net Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
