Re: uniq on open streams
At Wed, 24 Apr 2019 09:25:58 +0200, Andreas Krey wrote:
Subject: Re: uniq on open streams
>
> On Tue, 23 Apr 2019 11:50:23 +, Greg A. Woods wrote:
> ...
> >
> > Thinking "Oh, so it should work on FreeBSD too...", I opened a window to
> > my recently installed FreeBSD-12.0 machine, and after rewriting the Lua
> > code into AWK, I was surprised to see absolutely no output at all!
>
> stdout buffering is a thing. awk may not flush stdout, esp. when
> stdout is not a terminal. So uniq may not get anything.
>
> (echo a; echo b; echo c; sleep 3) | uniq -c
>
> is test case enough, and doesn't have the stdout buffer issue.
Awk does seem to flush stdout even when it feeds a pipe (and
incidentally has an "fflush" function just in case), but I've actually
ended up using:
{ echo a; echo b; echo c; read junk; } | uniq-apple
(Awk has "getline" too, but the shell variant is easier to type!)
Note, as this hints the Apple version (text_cmds-99) of FreeBSD's uniq
will build and run fine on NetBSD (with -D__APPLE__), and it does show
the 'c' line immediately.
However there's been so much churn (and what seems like entirely
unnecessary and mis-guided localization) both before and after that
version that I don't think it's useful to look at, though it does give a
hint as to how the algorithm might be changed for certain combinations
of command-line options to achieve the effect of printing each line
immediately when it differs from the line before (instead of only
printing the previous line, possibly with a count, when the current line
differs or EOF has been reached).
I don't see that changing this behaviour would ever be terribly useful
though since as you mentioned, stdout buffering is a thing (i.e. seeing
the output immediately only works if uniq's stdout is the tty. If it's
not the end of the pipeline then it will buffer its output too.
--
Greg A. Woods
+1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpMBrGO6AQez.pgp
Description: OpenPGP Digital Signature
Re: uniq on open streams
On Tue, 23 Apr 2019 11:50:23 +, Greg A. Woods wrote: ... > > Thinking "Oh, so it should work on FreeBSD too...", I opened a window to > my recently installed FreeBSD-12.0 machine, and after rewriting the Lua > code into AWK, I was surprised to see absolutely no output at all! stdout buffering is a thing. awk may not flush stdout, esp. when stdout is not a terminal. So uniq may not get anything. (echo a; echo b; echo c; sleep 3) | uniq -c is test case enough, and doesn't have the stdout buffer issue. - Andreas -- "Totally trivial. Famous last words." From: Linus Torvalds Date: Fri, 22 Jan 2010 07:29:21 -0800
Re: uniq on open streams
At Tue, 23 Apr 2019 13:45:40 +0200, Andreas Krey wrote:
Subject: Re: uniq on open streams
>
> On Tue, 23 Apr 2019 05:51:19 +, JP wrote:
> > I had a need to run uniq on an open stream. It doesn't seem to print
> > the most recent line.
> >
> > $ lua -e 'print("a");print("b");print("c"); repeat until false' |uniq
> > a
> > b
> >
> > ^ should print the c as well, no?
>
> Yes, but. 'uniq -c' can only print the 'c' line once it gets a different
> input line
> or EOF, and I'd bet that the code doesn't try to behave differently when run
> without -c.
So, this surprised me, and so I tried this command-line in the first
open window I could paste it into, and was surprised to see the 'c'!
It turns out I had pasted into a macOS window.
Thinking "Oh, so it should work on FreeBSD too...", I opened a window to
my recently installed FreeBSD-12.0 machine, and after rewriting the Lua
code into AWK, I was surprised to see absolutely no output at all!
Then I made the mistake of looking at the recent FreeBSD code. Oy!
Such churn! I've no idea why it doesn't output anything.
So I checked opensource.apple.com. Their version (in "text_cmds") is
somewhat behind the most recent FreeBSD churn, but is none the less
derived from an earlier version of the FreeBSD code.
I then looked at the OpenSolaris version (and the UNIX SysVr4 code from
which it came), and I thought at first glance that it might spit out the
'c' too, but alas it does not.
Neither does OpenBSD's.
Also, what about lack of a trailing newline on the last line when the
line before has the same content? Is the newline part of the line?
Some (FreeBSD and SysVr4/OpenSolaris) would say not.
--
Greg A. Woods
+1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpNGxrlrEzPd.pgp
Description: OpenPGP Digital Signature
Re: uniq on open streams
Andreas Krey wrote: > [...] also: > >netbsd$ uniq --help >uniq: uniq: No such file or directory Thanks, this one should be fixed now. -uwe
Re: uniq on open streams
oh i gotcha, the -c; i need this thing on open streams though
i guess it's trivial to implement what i need - i did so with lua in a
couple line
On Tue, Apr 23, 2019 at 7:51 AM JP wrote:
>
> it should print the line the first time it sees it and not print the
> repeats, in my opinion
>
> On Tue, Apr 23, 2019 at 11:45 AM Andreas Krey wrote:
> >
> > On Tue, 23 Apr 2019 05:51:19 +, JP wrote:
> > > I had a need to run uniq on an open stream. It doesn't seem to print
> > > the most recent line.
> > >
> > > $ lua -e 'print("a");print("b");print("c"); repeat until false' |uniq
> > > a
> > > b
> > >
> > > ^ should print the c as well, no?
> >
> > Yes, but. 'uniq -c' can only print the 'c' line once it gets a different
> > input line
> > or EOF, and I'd bet that the code doesn't try to behave differently when
> > run without -c.
> >
> > (Behaviour is different from GNU coreutils uniq; also:
> >
> >netbsd$ uniq --help
> >uniq: uniq: No such file or directory
> >
> > )
> >
> > - Andreas
> >
> > --
> > "Totally trivial. Famous last words."
> > From: Linus Torvalds
> > Date: Fri, 22 Jan 2010 07:29:21 -0800
Re: uniq on open streams
it should print the line the first time it sees it and not print the
repeats, in my opinion
On Tue, Apr 23, 2019 at 11:45 AM Andreas Krey wrote:
>
> On Tue, 23 Apr 2019 05:51:19 +, JP wrote:
> > I had a need to run uniq on an open stream. It doesn't seem to print
> > the most recent line.
> >
> > $ lua -e 'print("a");print("b");print("c"); repeat until false' |uniq
> > a
> > b
> >
> > ^ should print the c as well, no?
>
> Yes, but. 'uniq -c' can only print the 'c' line once it gets a different
> input line
> or EOF, and I'd bet that the code doesn't try to behave differently when run
> without -c.
>
> (Behaviour is different from GNU coreutils uniq; also:
>
>netbsd$ uniq --help
>uniq: uniq: No such file or directory
>
> )
>
> - Andreas
>
> --
> "Totally trivial. Famous last words."
> From: Linus Torvalds
> Date: Fri, 22 Jan 2010 07:29:21 -0800
Re: uniq on open streams
On Tue, 23 Apr 2019 05:51:19 +, JP wrote:
> I had a need to run uniq on an open stream. It doesn't seem to print
> the most recent line.
>
> $ lua -e 'print("a");print("b");print("c"); repeat until false' |uniq
> a
> b
>
> ^ should print the c as well, no?
Yes, but. 'uniq -c' can only print the 'c' line once it gets a different input
line
or EOF, and I'd bet that the code doesn't try to behave differently when run
without -c.
(Behaviour is different from GNU coreutils uniq; also:
netbsd$ uniq --help
uniq: uniq: No such file or directory
)
- Andreas
--
"Totally trivial. Famous last words."
From: Linus Torvalds
Date: Fri, 22 Jan 2010 07:29:21 -0800
uniq on open streams
I had a need to run uniq on an open stream. It doesn't seem to print
the most recent line.
$ lua -e 'print("a");print("b");print("c"); repeat until false' |uniq
a
b
^ should print the c as well, no?
