Re: I/O semantics of pipe and FIFO.

[Alfred Perlstein]

I've seen that bug, but I think our bug was more that dd would exit if it got 
back a short read from input. So if you did something like this (maybe not 
exactly, but close):

dd bs=1m | dd bs=10m > f.out

You might get only 1mb in f.out. This had to do with skipping disklabels and 
other such schenanigans. 

Sent from my iPhone

> On Mar 8, 2017, at 4:09 PM, Devin Teske  wrote:
> 
> Problem we had found was:
> 
> Executing dd with a closed stdout and stderr would cause the summary messages 
> printed at the end to go into the destination output file.
> 
> For example,
> 
> dd if=/dev/zero of=/tmp/foo bs=1m count=1
> 
> Works fine, but the following:
> 
> dd if=/dev/zero of=/tmp/foo bs=1m count=1 >&- 2>&-
> 
> Will cause the summary statistics of dd to appear in /tmp/foo instead of on 
> the console.
> 
> The issue is that the summary statistics are send to fd1, which if you close 
> down stdout and stdin, fd1 is actually the output file since it got the 
> lowest file descriptor available when open(2) was called on the output file.
> 
> This was never fixed because it was deemed “silly developer, don’t close 
> stdout and stderr before invoking dd”.
> 
> The argument has been made by Jilles T. that it is generally a bad idea to 
> close down any of the standard file descriptors because it cannot be 
> predicted how a particular UNIX utility will react (e.g., in the case of dd, 
> causing a simple printf(3) to go to an unexpected location).
> — 
> Devin
> 
> 
>> On Mar 4, 2017, at 8:12 PM, Alfred Perlstein  wrote:
>> 
>> Devin and I found this when we worked together.  I think it was due to some 
>> situation in dd(1) where short reads would exit pre-maturely, however I may 
>> be mis-remembering.  Devin, do you recall the specifics?
>> 
>> 
>>> On 3/4/17 7:44 PM, Julian Elischer wrote:
>>> 
>>> an interesting point to discuss? is our behaviour in this test right?
>>>  from: "austin-group mailng list (posix standard discussion)"
>>> 
>>> -- rest of email is quoted ---
>>> On 5/3/17 5:48 am, Stephane Chazelas wrote:
>>> 
>>> 2017-03-04 13:14:08 +, Danny Niu:
 Hi all.
 
 I couldn't remember where I saw it saying, that when reading
 from a pipe or a FIFO, the read syscall returns the content of
 at most one write call. It's a bit similar to the
 message-nondiscard semantics of dear old STREAM.
 
 Currently, I'm reading through the text to find out a bit
 more, and I appreciate a bit of pointer on this.
>>> [...]
>>> 
>>> (echo x; echo y) | (sleep 1; dd count=1 2> /dev/null)
>>> 
>>> outputs both x and y in all of Linux, FreeBSD and Solaris in my
>>> tests.
>>> 
>>> That a read wouldn't read what's currently in the pipe would be
>>> quite surprising.
>>> 
>>> I also wouldn't expect pipes to store the writes as individual
>>> separate message but use one buffer.
>>> 
>>> In:
>>> 
>>> (
>>> dd bs=4 count=1 if=/dev/zero 2> /dev/null
>>> echo first through >&2
>>> dd bs=4 count=1 if=/dev/zero 2> /dev/null
>>> echo second through >&2
>>> ) | (sleep 1; dd bs=10 count=1 2> /dev/null) | wc -c
>>> 
>>> That is where the second write blocks because the pipe is full,
>>> the reading dd still reads both writes in Linux and Solaris in
>>> my tests (on Solaris (10 on amd64 at least), reduce to 2
>>> instead of 4 or both writes would block).
>>> 
>>> On FreeBSD, I get only the first write (using 8000 followed by
>>> 1 for instance).
>>> 
>>> FreeBSD is also the only one of the three where
>>> 
>>> dd bs=100 count=1 if=/dev/zero | dd bs=100 count=1 | wc -c
>>> 
>>> Doesn't output 100. The others schedule both processes back
>>> and forth during their write() and read() system call while the
>>> pipe is being filled and emptied several times.
>>> 
>> 
> 

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: I/O semantics of pipe and FIFO.

[Don Lewis]

On  8 Mar, Devin Teske wrote:
> Problem we had found was:
> 
> Executing dd with a closed stdout and stderr would cause the summary
> messages printed at the end to go into the destination output file.
> 
> For example,
> 
> dd if=/dev/zero of=/tmp/foo bs=1m count=1
> 
> Works fine, but the following:
> 
> dd if=/dev/zero of=/tmp/foo bs=1m count=1 >&- 2>&-
> 
> Will cause the summary statistics of dd to appear in /tmp/foo instead
> of on the console.
> 
> The issue is that the summary statistics are send to fd1, which if you
> close down stdout and stdin, fd1 is actually the output file since it
> got the lowest file descriptor available when open(2) was called on
> the output file.
> 
> This was never fixed because it was deemed “silly developer, don’t
> close stdout and stderr before invoking dd”.
> 
> The argument has been made by Jilles T. that it is generally a bad
> idea to close down any of the standard file descriptors because it
> cannot be predicted how a particular UNIX utility will react (e.g., in
> the case of dd, causing a simple printf(3) to go to an unexpected
> location).

Shades of:

- From our Harris VOS system.
   JOBCNTRL ER  2211 : IT'S NOT NICE TO FOOL POP!
   >he 2211
   YOU JUST TRIED TO FAKE-OUT MOTHER NATURE, AND SHE CAUGHT YOU!  SUPER-
   VULCAN NOW HAS YOUR NAME ON HIS ENEMY LIST, AND YOU CAN BE CERTAIN THAT
   FUTURE ATTEMPTS TO RESOURCE LFN 0,3,OR 6 WILL RESULT IN YOUR BEING
   ABORTED, SPINDLED, MANGLED, FOLDED, PUNCHED, DELETED, AND DEALLOCATED.

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: I/O semantics of pipe and FIFO.

[Devin Teske]

Problem we had found was:

Executing dd with a closed stdout and stderr would cause the summary messages 
printed at the end to go into the destination output file.

For example,

dd if=/dev/zero of=/tmp/foo bs=1m count=1

Works fine, but the following:

dd if=/dev/zero of=/tmp/foo bs=1m count=1 >&- 2>&-

Will cause the summary statistics of dd to appear in /tmp/foo instead of on the 
console.

The issue is that the summary statistics are send to fd1, which if you close 
down stdout and stdin, fd1 is actually the output file since it got the lowest 
file descriptor available when open(2) was called on the output file.

This was never fixed because it was deemed “silly developer, don’t close stdout 
and stderr before invoking dd”.

The argument has been made by Jilles T. that it is generally a bad idea to 
close down any of the standard file descriptors because it cannot be predicted 
how a particular UNIX utility will react (e.g., in the case of dd, causing a 
simple printf(3) to go to an unexpected location).
— 
Devin


> On Mar 4, 2017, at 8:12 PM, Alfred Perlstein  wrote:
> 
> Devin and I found this when we worked together.  I think it was due to some 
> situation in dd(1) where short reads would exit pre-maturely, however I may 
> be mis-remembering.  Devin, do you recall the specifics?
> 
> 
> On 3/4/17 7:44 PM, Julian Elischer wrote:
>> 
>> an interesting point to discuss? is our behaviour in this test right?
>>   from: "austin-group mailng list (posix standard discussion)"
>> 
>> -- rest of email is quoted ---
>> On 5/3/17 5:48 am, Stephane Chazelas wrote:
>> 
>> 2017-03-04 13:14:08 +, Danny Niu:
>>> Hi all.
>>> 
>>> I couldn't remember where I saw it saying, that when reading
>>> from a pipe or a FIFO, the read syscall returns the content of
>>> at most one write call. It's a bit similar to the
>>> message-nondiscard semantics of dear old STREAM.
>>> 
>>> Currently, I'm reading through the text to find out a bit
>>> more, and I appreciate a bit of pointer on this.
>> [...]
>> 
>> (echo x; echo y) | (sleep 1; dd count=1 2> /dev/null)
>> 
>> outputs both x and y in all of Linux, FreeBSD and Solaris in my
>> tests.
>> 
>> That a read wouldn't read what's currently in the pipe would be
>> quite surprising.
>> 
>> I also wouldn't expect pipes to store the writes as individual
>> separate message but use one buffer.
>> 
>> In:
>> 
>> (
>>  dd bs=4 count=1 if=/dev/zero 2> /dev/null
>>  echo first through >&2
>>  dd bs=4 count=1 if=/dev/zero 2> /dev/null
>>  echo second through >&2
>> ) | (sleep 1; dd bs=10 count=1 2> /dev/null) | wc -c
>> 
>> That is where the second write blocks because the pipe is full,
>> the reading dd still reads both writes in Linux and Solaris in
>> my tests (on Solaris (10 on amd64 at least), reduce to 2
>> instead of 4 or both writes would block).
>> 
>> On FreeBSD, I get only the first write (using 8000 followed by
>> 1 for instance).
>> 
>> FreeBSD is also the only one of the three where
>> 
>> dd bs=100 count=1 if=/dev/zero | dd bs=100 count=1 | wc -c
>> 
>> Doesn't output 100. The others schedule both processes back
>> and forth during their write() and read() system call while the
>> pipe is being filled and emptied several times.
>> 
> 

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Fwd: Re: I/O semantics of pipe and FIFO.

[Alfred Perlstein]

Devin and I found this when we worked together.  I think it was due to 
some situation in dd(1) where short reads would exit pre-maturely, 
however I may be mis-remembering.  Devin, do you recall the specifics?



On 3/4/17 7:44 PM, Julian Elischer wrote:


an interesting point to discuss? is our behaviour in this test right?
   from: "austin-group mailng list (posix standard discussion)"

-- rest of email is quoted ---
On 5/3/17 5:48 am, Stephane Chazelas wrote:

2017-03-04 13:14:08 +, Danny Niu:

Hi all.

I couldn't remember where I saw it saying, that when reading
from a pipe or a FIFO, the read syscall returns the content of
at most one write call. It's a bit similar to the
message-nondiscard semantics of dear old STREAM.

Currently, I'm reading through the text to find out a bit
more, and I appreciate a bit of pointer on this.

[...]

(echo x; echo y) | (sleep 1; dd count=1 2> /dev/null)

outputs both x and y in all of Linux, FreeBSD and Solaris in my
tests.

That a read wouldn't read what's currently in the pipe would be
quite surprising.

I also wouldn't expect pipes to store the writes as individual
separate message but use one buffer.

In:

(
  dd bs=4 count=1 if=/dev/zero 2> /dev/null
  echo first through >&2
  dd bs=4 count=1 if=/dev/zero 2> /dev/null
  echo second through >&2
) | (sleep 1; dd bs=10 count=1 2> /dev/null) | wc -c

That is where the second write blocks because the pipe is full,
the reading dd still reads both writes in Linux and Solaris in
my tests (on Solaris (10 on amd64 at least), reduce to 2
instead of 4 or both writes would block).

On FreeBSD, I get only the first write (using 8000 followed by
1 for instance).

FreeBSD is also the only one of the three where

dd bs=100 count=1 if=/dev/zero | dd bs=100 count=1 | wc -c

Doesn't output 100. The others schedule both processes back
and forth during their write() and read() system call while the
pipe is being filled and emptied several times.



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Fwd: Re: I/O semantics of pipe and FIFO.

[Julian Elischer]

an interesting point to discuss? is our behaviour in this test right?
   from: "austin-group mailng list (posix standard discussion)"

-- rest of email is quoted ---
On 5/3/17 5:48 am, Stephane Chazelas wrote:

2017-03-04 13:14:08 +, Danny Niu:

Hi all.

I couldn't remember where I saw it saying, that when reading
from a pipe or a FIFO, the read syscall returns the content of
at most one write call. It's a bit similar to the
message-nondiscard semantics of dear old STREAM.

Currently, I'm reading through the text to find out a bit
more, and I appreciate a bit of pointer on this.

[...]

(echo x; echo y) | (sleep 1; dd count=1 2> /dev/null)

outputs both x and y in all of Linux, FreeBSD and Solaris in my
tests.

That a read wouldn't read what's currently in the pipe would be
quite surprising.

I also wouldn't expect pipes to store the writes as individual
separate message but use one buffer.

In:

(
  dd bs=4 count=1 if=/dev/zero 2> /dev/null
  echo first through >&2
  dd bs=4 count=1 if=/dev/zero 2> /dev/null
  echo second through >&2
) | (sleep 1; dd bs=10 count=1 2> /dev/null) | wc -c

That is where the second write blocks because the pipe is full,
the reading dd still reads both writes in Linux and Solaris in
my tests (on Solaris (10 on amd64 at least), reduce to 2
instead of 4 or both writes would block).

On FreeBSD, I get only the first write (using 8000 followed by
1 for instance).

FreeBSD is also the only one of the three where

dd bs=100 count=1 if=/dev/zero | dd bs=100 count=1 | wc -c

Doesn't output 100. The others schedule both processes back
and forth during their write() and read() system call while the
pipe is being filled and emptied several times.

--
Stephane

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"