Re: I/O semantics of pipe and FIFO.

[Don Cragun]

While it may be true that one or more systems have this "feature",
the script you have provided does not demonstrate that behavior.
(And, I am not aware of any system that does behave the way you
have described.)

Since a writer is blocked from writing into a FIFO if the FIFO is
not open for reading, both of your asynchronous echo commands will
block until dd opens the FIFO for reading.  When the dd starts,
one of the echo commands will be unblocked and write 4 bytes into
the FIFO.  There is then a race as to whether the 2nd echo will
write another 4 bytes into the FIFO or dd will read 4 bytes from
the FIFO.  In the example you have shown, dd reads the contents
of the FIFO before the 2nd echo wakes up and writes the other 4
bytes into the FIFO and then dd reads those 4 bytes as a second
partial read.

With a more complicated test that opens the FIFO for reading
(without actually reading any data from the FIFO) before
executing the echo commands, then runs the echo commands and
then runs the dd command (before the other reader closes the
FIFO), we can see that the output from the two echo commands is
seen by dd in 1 read from the FIFO.  Unfortunately, at this
point we have dd hung waiting for another writer, to allow its
blocked read to continue, but adding another echo takes care of
that.  If a read from a FIFO returned at most the data written by
a single write system call, the dd in the following script would
show 3 partial reads instead of two.

opener.c:
=
#include 
#include 
#include 
#include 
#include 
#include 
#include 

int
main(int argc, char *argv[]) {
int fd;

fd = open(argv[1], O_RDONLY);
if(fd == -1) {
fprintf(stderr, "Can't open '%s' (%s)\n", argv[1], 
strerror(errno));
exit(1);
}
sleep(30);
printf("opener done.\n");
}

shell script:
#!/bin/ksh
[ ! -p myfifo ] && mkfifo myfifo
ls -l myfifo
date
./opener myfifo&
sleep 1
date
/bin/echo abc > myfifo
date
/bin/echo 123 > myfifo
date
dd if=myfifo&
date
sleep 40
date
/bin/echo XYZ > myfifo
date

Output produced by running the above shell script on macOS Sierra version 
10.12.3:
==
prw-r--r--  1 dwc  staff  0 Mar  4 22:16 myfifo
Sat Mar  4 22:16:38 PST 2017
Sat Mar  4 22:16:39 PST 2017
Sat Mar  4 22:16:39 PST 2017
Sat Mar  4 22:16:39 PST 2017
Sat Mar  4 22:16:39 PST 2017
opener done.
Sat Mar  4 22:17:19 PST 2017
abc
123
XYZ
0+2 records in
0+1 records out
12 bytes transferred in 0.002135 secs (5621 bytes/sec)
Sat Mar  4 22:17:19 PST 2017

 - Don


> On Mar 4, 2017, at 6:25 PM, Danny Niu  wrote:
> 
> Found it: http://man.cat-v.org/unix_8th/2/write
> 
> Should I file an editorial bug on the rationale section of the write() 
> interface, to note that some systems have this feature? Since we've already 
> mentioned the special case of making a zero-length write request. 
> 
> From: Danny Niu 
> Sent: Sunday, March 5, 2017 10:05
> To: Stephane Chazelas
> Cc: Austin Group Mailing List
> Subject: Re: I/O semantics of pipe and FIFO.
> 
> Just so you guys can reproduce what I was talking about, here's essentially 
> what I did:
> 
> #!/bin/sh
> 
> cd ~
> mkfifo myfifo
> /bin/echo 123 >myfifo & /bin/echo abc >myfifo &
> sleep 1
> dd if=myfifo #dd defaults to block size of 512 in this case.
> 
> output from dd (stdout+stderr):
> 123
> abc
> 0+2 records in
> 0+1 records out
> 8 bytes transferred in 0.004737 secs
> 
> Not that I use /bin/echo instead of just echo to ensure two separate write 
> system calls are made.
> 
> From: Stephane Chazelas 
> Sent: Sunday, March 5, 2017 05:48
> To: Danny Niu
> Cc: Austin Group Mailing List
> Subject: Re: I/O semantics of pipe and FIFO.
> 
> 2017-03-04 13:14:08 +, Danny Niu:
>> Hi all.
>> 
>> I couldn't remember where I saw it saying, that when reading
>> from a pipe or a FIFO, the read syscall returns the content of
>> at most one write call. It's a bit similar to the
>> message-nondiscard semantics of dear old STREAM.
>> 
>> Currently, I'm reading through the text to find out a bit
>> more, and I appreciate a bit of pointer on this.
> [...]
> 
> (echo x; echo y) | (sleep 1; dd count=1 2> /dev/null)
> 
> outputs both x and y in all of Linux, FreeBSD and Solaris in my
> tests.
> 
> That a read wouldn't read what's currently in the pipe would be
> quite surprising.
> 
> I also wouldn't expect pipes to store the writes as individual
> separate message but use one buffer.
> 
> In:
> 
> (
>  dd bs=4 count=1 if=/dev/zero 2> /dev/null
>  echo first through >&2
>  dd bs=4 count=1 if=/dev/zero 2> /dev/null
>  echo second through >&2
> ) | (sleep 1; dd bs=10 count=1 2> /dev/null) | wc -c
> 
> That is where the second write blocks because the pipe is full,
> the reading dd still reads both writes 

[1003.1(2008)/Issue 7 0001124]: Mention different IO semantics of pipe and FIFO.

[Austin Group Bug Tracker]

The following issue has been UPDATED. 
== 
http://austingroupbugs.net/view.php?id=1124 
== 
Reported By:dannyniu
Assigned To:ajosey
== 
Project:1003.1(2008)/Issue 7
Issue ID:   1124
Category:   System Interfaces
Type:   Enhancement Request
Severity:   Editorial
Priority:   normal
Status: Under Review
Name:   DannyNiu/NJF 
Organization:
User Reference:  
Section:System Interfaces - write() 
Page Number:Informative section 
Line Number:Rationales 
Interp Status:  --- 
Final Accepted Text: 
== 
Date Submitted: 2017-03-05 02:46 UTC
Last Modified:  2017-03-05 06:10 UTC
== 
Summary:Mention different IO semantics of pipe and FIFO.
== 

Issue History 
Date ModifiedUsername   FieldChange   
== 
2017-03-05 02:46 dannyniu   New Issue
2017-03-05 02:46 dannyniu   Status   New => Under Review 
2017-03-05 02:46 dannyniu   Assigned To   => ajosey  
2017-03-05 02:46 dannyniu   Name  => DannyNiu/NJF
2017-03-05 02:46 dannyniu   Section   => System Interfaces -
write()
2017-03-05 02:46 dannyniu   Page Number   => Informative section
2017-03-05 02:46 dannyniu   Line Number   => Rationales  
2017-03-05 06:10 dannyniu   Desired Action Updated   
==

[1003.1(2008)/Issue 7 0001124]: Mention different IO semantics of pipe and FIFO.

[Austin Group Bug Tracker]

The following issue has been SUBMITTED. 
== 
http://austingroupbugs.net/view.php?id=1124 
== 
Reported By:dannyniu
Assigned To:ajosey
== 
Project:1003.1(2008)/Issue 7
Issue ID:   1124
Category:   System Interfaces
Type:   Enhancement Request
Severity:   Editorial
Priority:   normal
Status: Under Review
Name:   DannyNiu/NJF 
Organization:
User Reference:  
Section:System Interfaces - write() 
Page Number:Informative section 
Line Number:Rationales 
Interp Status:  --- 
Final Accepted Text: 
== 
Date Submitted: 2017-03-05 02:46 UTC
Last Modified:  2017-03-05 02:46 UTC
== 
Summary:Mention different IO semantics of pipe and FIFO.
Description: 
In the rationale section for the write() function, we've taken note of the
fact that, some systems allows a write of length zero to indicate
end-of-file to the other end of a pipe. 

In my opinion, this feature is the by-product of some systems using the
message-nondiscard IO semantics for pipe, such as 8th edition we've noted
in the text. And I think it's reasonable, that we further note such
semantics, by mentioning on such systems, read() from a pipe, will return
the content of at most one write() call. 
Desired Action: 
Add to after the text:

Also, some existing systems ... permit a write of zero bytes ... indicates
a successful write of an end-of-file indication. 

The following: 

On such systems, a read() from a pipe or a FIFO, returns at most the
content of one write() call. 
== 

Issue History 
Date ModifiedUsername   FieldChange   
== 
2017-03-05 02:46 dannyniu   New Issue
2017-03-05 02:46 dannyniu   Status   New => Under Review 
2017-03-05 02:46 dannyniu   Assigned To   => ajosey  
2017-03-05 02:46 dannyniu   Name  => DannyNiu/NJF
2017-03-05 02:46 dannyniu   Section   => System Interfaces -
write()
2017-03-05 02:46 dannyniu   Page Number   => Informative section
2017-03-05 02:46 dannyniu   Line Number   => Rationales  
==

Re: I/O semantics of pipe and FIFO.

[Danny Niu]

Found it: http://man.cat-v.org/unix_8th/2/write

Should I file an editorial bug on the rationale section of the write() 
interface, to note that some systems have this feature? Since we've already 
mentioned the special case of making a zero-length write request. 

From: Danny Niu 
Sent: Sunday, March 5, 2017 10:05
To: Stephane Chazelas
Cc: Austin Group Mailing List
Subject: Re: I/O semantics of pipe and FIFO.

Just so you guys can reproduce what I was talking about, here's essentially 
what I did:

#!/bin/sh

cd ~
mkfifo myfifo
/bin/echo 123 >myfifo & /bin/echo abc >myfifo &
sleep 1
dd if=myfifo #dd defaults to block size of 512 in this case.

output from dd (stdout+stderr):
123
abc
0+2 records in
0+1 records out
8 bytes transferred in 0.004737 secs

Not that I use /bin/echo instead of just echo to ensure two separate write 
system calls are made.

From: Stephane Chazelas 
Sent: Sunday, March 5, 2017 05:48
To: Danny Niu
Cc: Austin Group Mailing List
Subject: Re: I/O semantics of pipe and FIFO.

2017-03-04 13:14:08 +, Danny Niu:
> Hi all.
>
> I couldn't remember where I saw it saying, that when reading
> from a pipe or a FIFO, the read syscall returns the content of
> at most one write call. It's a bit similar to the
> message-nondiscard semantics of dear old STREAM.
>
> Currently, I'm reading through the text to find out a bit
> more, and I appreciate a bit of pointer on this.
[...]

(echo x; echo y) | (sleep 1; dd count=1 2> /dev/null)

outputs both x and y in all of Linux, FreeBSD and Solaris in my
tests.

That a read wouldn't read what's currently in the pipe would be
quite surprising.

I also wouldn't expect pipes to store the writes as individual
separate message but use one buffer.

In:

(
  dd bs=4 count=1 if=/dev/zero 2> /dev/null
  echo first through >&2
  dd bs=4 count=1 if=/dev/zero 2> /dev/null
  echo second through >&2
) | (sleep 1; dd bs=10 count=1 2> /dev/null) | wc -c

That is where the second write blocks because the pipe is full,
the reading dd still reads both writes in Linux and Solaris in
my tests (on Solaris (10 on amd64 at least), reduce to 2
instead of 4 or both writes would block).

On FreeBSD, I get only the first write (using 8000 followed by
1 for instance).

FreeBSD is also the only one of the three where

dd bs=100 count=1 if=/dev/zero | dd bs=100 count=1 | wc -c

Doesn't output 100. The others schedule both processes back
and forth during their write() and read() system call while the
pipe is being filled and emptied several times.

--
Stephane

[1003.1(2016)/Issue7+TC2 0001122]: POSIX should include gettext() and friends

[Austin Group Bug Tracker]

A NOTE has been added to this issue. 
== 
http://austingroupbugs.net/view.php?id=1122 
== 
Reported By:joerg
Assigned To:
== 
Project:1003.1(2016)/Issue7+TC2
Issue ID:   1122
Category:   System Interfaces
Type:   Enhancement Request
Severity:   Editorial
Priority:   normal
Status: New
Name:   Jörg Schilling 
Organization:
User Reference:  
Section:3 + 4 
Page Number:1102...and others 
Line Number:somewhere in section 3 and 4 
Interp Status:  --- 
Final Accepted Text: 
== 
Date Submitted: 2017-02-28 16:51 UTC
Last Modified:  2017-03-05 00:12 UTC
== 
Summary:POSIX should include gettext() and friends
== 

-- 
 (0003608) keld (reporter) - 2017-03-05 00:12
 http://austingroupbugs.net/view.php?id=1122#c3608 
-- 
Case conversion and collation is handled by the locale, which also chooses
which message catalogue to use. Quite elaborate schemes for case
conversions and also collation can be specified with current glibc (ISO
30112) functionality, and that is indeed in widespread use - building on
ISO 14651 for the collation.

Where is the need for u8"" or u"" support? Gettext handles localization of
a program, where the strings to be translated are just given to the API as
native strings, which are character encoding independent, and you can port
the program to another environment with another internal encoding. 

I believe \U and \u escapes are available as normal native string
processing.

For UTF-16 or other 16-bit handling, I believe the normal libc APIS treat
16 bits as one byte, but I am not sure. I believe IOS is using UTF-16 so
there must be a lot of experience of porting normal standard C/C++/POSIX
utilities to that environment. Does anybody know how this is done? 

Issue History 
Date ModifiedUsername   FieldChange   
== 
2017-02-28 16:51 joerg  New Issue
2017-02-28 16:51 joerg  Name  => Jörg Schilling 
2017-02-28 16:51 joerg  Section   => 3 + 4   
2017-02-28 16:51 joerg  Page Number   => 1102...and others
2017-02-28 16:51 joerg  Line Number   => somewhere in
section 3 and 4
2017-03-01 16:05 steffenNote Added: 0003575  
2017-03-01 16:54 shware_systems Note Added: 0003576  
2017-03-01 17:10 joerg  Note Added: 0003577  
2017-03-01 17:10 joerg  Note Edited: 0003577 
2017-03-01 17:11 steffenNote Added: 0003578  
2017-03-01 17:13 steffenNote Added: 0003579  
2017-03-01 17:23 joerg  Note Added: 0003580  
2017-03-01 17:27 joerg  Note Edited: 0003580 
2017-03-01 18:09 joerg  Note Edited: 0003580 
2017-03-01 22:37 steffenNote Added: 0003581  
2017-03-02 06:39 shware_systems Note Added: 0003582  
2017-03-02 09:09 joerg  Note Added: 0003583  
2017-03-02 09:40 keld   Note Added: 0003584  
2017-03-02 09:56 keld   Note Added: 0003585  
2017-03-02 14:41 steffenNote Added: 0003586  
2017-03-02 17:01 keld   Note Added: 0003587  
2017-03-03 04:42 shware_systems Note Added: 0003588  
2017-03-03 16:44 joerg  Note Added: 0003596  
2017-03-03 21:21 EdSchouten Note Added: 0003603  
2017-03-05 00:12 keld   Note Added: 0003608  
==

Re: I/O semantics of pipe and FIFO.

[Stephane Chazelas]

2017-03-04 13:14:08 +, Danny Niu:
> Hi all. 
> 
> I couldn't remember where I saw it saying, that when reading
> from a pipe or a FIFO, the read syscall returns the content of
> at most one write call. It's a bit similar to the
> message-nondiscard semantics of dear old STREAM. 
> 
> Currently, I'm reading through the text to find out a bit
> more, and I appreciate a bit of pointer on this. 
[...]

(echo x; echo y) | (sleep 1; dd count=1 2> /dev/null)

outputs both x and y in all of Linux, FreeBSD and Solaris in my
tests.

That a read wouldn't read what's currently in the pipe would be
quite surprising.

I also wouldn't expect pipes to store the writes as individual
separate message but use one buffer.

In:

(
  dd bs=4 count=1 if=/dev/zero 2> /dev/null
  echo first through >&2
  dd bs=4 count=1 if=/dev/zero 2> /dev/null
  echo second through >&2
) | (sleep 1; dd bs=10 count=1 2> /dev/null) | wc -c

That is where the second write blocks because the pipe is full,
the reading dd still reads both writes in Linux and Solaris in
my tests (on Solaris (10 on amd64 at least), reduce to 2
instead of 4 or both writes would block).

On FreeBSD, I get only the first write (using 8000 followed by
1 for instance).

FreeBSD is also the only one of the three where

dd bs=100 count=1 if=/dev/zero | dd bs=100 count=1 | wc -c

Doesn't output 100. The others schedule both processes back
and forth during their write() and read() system call while the
pipe is being filled and emptied several times.

-- 
Stephane

Re: I/O semantics of pipe and FIFO.

[Danny Niu]

From: Danny Niu 
Sent: Saturday, March 4, 2017 22:43
To: Shware Systems
Subject: Re: I/O semantics of pipe and FIFO.

LSB deferred to SUSv4 on this, but I found Linux v4.09 manpage section 2 read 
mentioning it might return less when reading from pipe, tty, etc.

I did an experiment earlier this day on Darwin, by /bin/echo 'ing two very 
short strings into a fifo (nearly) simultaneously, then dd it out a bit later, 
and I got two partial reads.

From: Shware Systems 
Sent: Saturday, March 4, 2017 21:50
To: danny...@hotmail.com
Subject: RE: I/O semantics of pipe and FIFO.

syscall is more Linux, so you may have been reading LSB docs. For POSIX it 
should read n bytes, or the size used with setvbuf, that I see.

On Saturday, March 4, 2017 Danny Niu  wrote:

Hi all.

I couldn't remember where I saw it saying, that when reading from a pipe or a 
FIFO, the read syscall returns the content of at most one write call. It's a 
bit similar to the message-nondiscard semantics of dear old STREAM.

Currently, I'm reading through the text to find out a bit more, and I 
appreciate a bit of pointer on this.

Thanks.

I/O semantics of pipe and FIFO.

[Danny Niu]

Hi all. 

I couldn't remember where I saw it saying, that when reading from a pipe or a 
FIFO, the read syscall returns the content of at most one write call. It's a 
bit similar to the message-nondiscard semantics of dear old STREAM. 

Currently, I'm reading through the text to find out a bit more, and I 
appreciate a bit of pointer on this. 

Thanks.