Re: I/O semantics of pipe and FIFO.
I've seen that bug, but I think our bug was more that dd would exit if it got back a short read from input. So if you did something like this (maybe not exactly, but close): dd bs=1m | dd bs=10m > f.out You might get only 1mb in f.out. This had to do with skipping disklabels and other such schenanigans. Sent from my iPhone > On Mar 8, 2017, at 4:09 PM, Devin Teske wrote: > > Problem we had found was: > > Executing dd with a closed stdout and stderr would cause the summary messages > printed at the end to go into the destination output file. > > For example, > > dd if=/dev/zero of=/tmp/foo bs=1m count=1 > > Works fine, but the following: > > dd if=/dev/zero of=/tmp/foo bs=1m count=1 >&- 2>&- > > Will cause the summary statistics of dd to appear in /tmp/foo instead of on > the console. > > The issue is that the summary statistics are send to fd1, which if you close > down stdout and stdin, fd1 is actually the output file since it got the > lowest file descriptor available when open(2) was called on the output file. > > This was never fixed because it was deemed “silly developer, don’t close > stdout and stderr before invoking dd”. > > The argument has been made by Jilles T. that it is generally a bad idea to > close down any of the standard file descriptors because it cannot be > predicted how a particular UNIX utility will react (e.g., in the case of dd, > causing a simple printf(3) to go to an unexpected location). > — > Devin > > >> On Mar 4, 2017, at 8:12 PM, Alfred Perlstein wrote: >> >> Devin and I found this when we worked together. I think it was due to some >> situation in dd(1) where short reads would exit pre-maturely, however I may >> be mis-remembering. Devin, do you recall the specifics? >> >> >>> On 3/4/17 7:44 PM, Julian Elischer wrote: >>> >>> an interesting point to discuss? is our behaviour in this test right? >>> from: "austin-group mailng list (posix standard discussion)" >>> >>> -- rest of email is quoted --- >>> On 5/3/17 5:48 am, Stephane Chazelas wrote: >>> >>> 2017-03-04 13:14:08 +, Danny Niu: Hi all. I couldn't remember where I saw it saying, that when reading from a pipe or a FIFO, the read syscall returns the content of at most one write call. It's a bit similar to the message-nondiscard semantics of dear old STREAM. Currently, I'm reading through the text to find out a bit more, and I appreciate a bit of pointer on this. >>> [...] >>> >>> (echo x; echo y) | (sleep 1; dd count=1 2> /dev/null) >>> >>> outputs both x and y in all of Linux, FreeBSD and Solaris in my >>> tests. >>> >>> That a read wouldn't read what's currently in the pipe would be >>> quite surprising. >>> >>> I also wouldn't expect pipes to store the writes as individual >>> separate message but use one buffer. >>> >>> In: >>> >>> ( >>> dd bs=4 count=1 if=/dev/zero 2> /dev/null >>> echo first through >&2 >>> dd bs=4 count=1 if=/dev/zero 2> /dev/null >>> echo second through >&2 >>> ) | (sleep 1; dd bs=10 count=1 2> /dev/null) | wc -c >>> >>> That is where the second write blocks because the pipe is full, >>> the reading dd still reads both writes in Linux and Solaris in >>> my tests (on Solaris (10 on amd64 at least), reduce to 2 >>> instead of 4 or both writes would block). >>> >>> On FreeBSD, I get only the first write (using 8000 followed by >>> 1 for instance). >>> >>> FreeBSD is also the only one of the three where >>> >>> dd bs=100 count=1 if=/dev/zero | dd bs=100 count=1 | wc -c >>> >>> Doesn't output 100. The others schedule both processes back >>> and forth during their write() and read() system call while the >>> pipe is being filled and emptied several times. >>> >> > ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: I/O semantics of pipe and FIFO.
On 8 Mar, Devin Teske wrote: > Problem we had found was: > > Executing dd with a closed stdout and stderr would cause the summary > messages printed at the end to go into the destination output file. > > For example, > > dd if=/dev/zero of=/tmp/foo bs=1m count=1 > > Works fine, but the following: > > dd if=/dev/zero of=/tmp/foo bs=1m count=1 >&- 2>&- > > Will cause the summary statistics of dd to appear in /tmp/foo instead > of on the console. > > The issue is that the summary statistics are send to fd1, which if you > close down stdout and stdin, fd1 is actually the output file since it > got the lowest file descriptor available when open(2) was called on > the output file. > > This was never fixed because it was deemed “silly developer, don’t > close stdout and stderr before invoking dd”. > > The argument has been made by Jilles T. that it is generally a bad > idea to close down any of the standard file descriptors because it > cannot be predicted how a particular UNIX utility will react (e.g., in > the case of dd, causing a simple printf(3) to go to an unexpected > location). Shades of: - From our Harris VOS system. JOBCNTRL ER 2211 : IT'S NOT NICE TO FOOL POP! >he 2211 YOU JUST TRIED TO FAKE-OUT MOTHER NATURE, AND SHE CAUGHT YOU! SUPER- VULCAN NOW HAS YOUR NAME ON HIS ENEMY LIST, AND YOU CAN BE CERTAIN THAT FUTURE ATTEMPTS TO RESOURCE LFN 0,3,OR 6 WILL RESULT IN YOUR BEING ABORTED, SPINDLED, MANGLED, FOLDED, PUNCHED, DELETED, AND DEALLOCATED. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: I/O semantics of pipe and FIFO.
Problem we had found was: Executing dd with a closed stdout and stderr would cause the summary messages printed at the end to go into the destination output file. For example, dd if=/dev/zero of=/tmp/foo bs=1m count=1 Works fine, but the following: dd if=/dev/zero of=/tmp/foo bs=1m count=1 >&- 2>&- Will cause the summary statistics of dd to appear in /tmp/foo instead of on the console. The issue is that the summary statistics are send to fd1, which if you close down stdout and stdin, fd1 is actually the output file since it got the lowest file descriptor available when open(2) was called on the output file. This was never fixed because it was deemed “silly developer, don’t close stdout and stderr before invoking dd”. The argument has been made by Jilles T. that it is generally a bad idea to close down any of the standard file descriptors because it cannot be predicted how a particular UNIX utility will react (e.g., in the case of dd, causing a simple printf(3) to go to an unexpected location). — Devin > On Mar 4, 2017, at 8:12 PM, Alfred Perlstein wrote: > > Devin and I found this when we worked together. I think it was due to some > situation in dd(1) where short reads would exit pre-maturely, however I may > be mis-remembering. Devin, do you recall the specifics? > > > On 3/4/17 7:44 PM, Julian Elischer wrote: >> >> an interesting point to discuss? is our behaviour in this test right? >> from: "austin-group mailng list (posix standard discussion)" >> >> -- rest of email is quoted --- >> On 5/3/17 5:48 am, Stephane Chazelas wrote: >> >> 2017-03-04 13:14:08 +, Danny Niu: >>> Hi all. >>> >>> I couldn't remember where I saw it saying, that when reading >>> from a pipe or a FIFO, the read syscall returns the content of >>> at most one write call. It's a bit similar to the >>> message-nondiscard semantics of dear old STREAM. >>> >>> Currently, I'm reading through the text to find out a bit >>> more, and I appreciate a bit of pointer on this. >> [...] >> >> (echo x; echo y) | (sleep 1; dd count=1 2> /dev/null) >> >> outputs both x and y in all of Linux, FreeBSD and Solaris in my >> tests. >> >> That a read wouldn't read what's currently in the pipe would be >> quite surprising. >> >> I also wouldn't expect pipes to store the writes as individual >> separate message but use one buffer. >> >> In: >> >> ( >> dd bs=4 count=1 if=/dev/zero 2> /dev/null >> echo first through >&2 >> dd bs=4 count=1 if=/dev/zero 2> /dev/null >> echo second through >&2 >> ) | (sleep 1; dd bs=10 count=1 2> /dev/null) | wc -c >> >> That is where the second write blocks because the pipe is full, >> the reading dd still reads both writes in Linux and Solaris in >> my tests (on Solaris (10 on amd64 at least), reduce to 2 >> instead of 4 or both writes would block). >> >> On FreeBSD, I get only the first write (using 8000 followed by >> 1 for instance). >> >> FreeBSD is also the only one of the three where >> >> dd bs=100 count=1 if=/dev/zero | dd bs=100 count=1 | wc -c >> >> Doesn't output 100. The others schedule both processes back >> and forth during their write() and read() system call while the >> pipe is being filled and emptied several times. >> > ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Fwd: Re: I/O semantics of pipe and FIFO.
Devin and I found this when we worked together. I think it was due to some situation in dd(1) where short reads would exit pre-maturely, however I may be mis-remembering. Devin, do you recall the specifics? On 3/4/17 7:44 PM, Julian Elischer wrote: an interesting point to discuss? is our behaviour in this test right? from: "austin-group mailng list (posix standard discussion)" -- rest of email is quoted --- On 5/3/17 5:48 am, Stephane Chazelas wrote: 2017-03-04 13:14:08 +, Danny Niu: Hi all. I couldn't remember where I saw it saying, that when reading from a pipe or a FIFO, the read syscall returns the content of at most one write call. It's a bit similar to the message-nondiscard semantics of dear old STREAM. Currently, I'm reading through the text to find out a bit more, and I appreciate a bit of pointer on this. [...] (echo x; echo y) | (sleep 1; dd count=1 2> /dev/null) outputs both x and y in all of Linux, FreeBSD and Solaris in my tests. That a read wouldn't read what's currently in the pipe would be quite surprising. I also wouldn't expect pipes to store the writes as individual separate message but use one buffer. In: ( dd bs=4 count=1 if=/dev/zero 2> /dev/null echo first through >&2 dd bs=4 count=1 if=/dev/zero 2> /dev/null echo second through >&2 ) | (sleep 1; dd bs=10 count=1 2> /dev/null) | wc -c That is where the second write blocks because the pipe is full, the reading dd still reads both writes in Linux and Solaris in my tests (on Solaris (10 on amd64 at least), reduce to 2 instead of 4 or both writes would block). On FreeBSD, I get only the first write (using 8000 followed by 1 for instance). FreeBSD is also the only one of the three where dd bs=100 count=1 if=/dev/zero | dd bs=100 count=1 | wc -c Doesn't output 100. The others schedule both processes back and forth during their write() and read() system call while the pipe is being filled and emptied several times. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Fwd: Re: I/O semantics of pipe and FIFO.
an interesting point to discuss? is our behaviour in this test right? from: "austin-group mailng list (posix standard discussion)" -- rest of email is quoted --- On 5/3/17 5:48 am, Stephane Chazelas wrote: 2017-03-04 13:14:08 +, Danny Niu: Hi all. I couldn't remember where I saw it saying, that when reading from a pipe or a FIFO, the read syscall returns the content of at most one write call. It's a bit similar to the message-nondiscard semantics of dear old STREAM. Currently, I'm reading through the text to find out a bit more, and I appreciate a bit of pointer on this. [...] (echo x; echo y) | (sleep 1; dd count=1 2> /dev/null) outputs both x and y in all of Linux, FreeBSD and Solaris in my tests. That a read wouldn't read what's currently in the pipe would be quite surprising. I also wouldn't expect pipes to store the writes as individual separate message but use one buffer. In: ( dd bs=4 count=1 if=/dev/zero 2> /dev/null echo first through >&2 dd bs=4 count=1 if=/dev/zero 2> /dev/null echo second through >&2 ) | (sleep 1; dd bs=10 count=1 2> /dev/null) | wc -c That is where the second write blocks because the pipe is full, the reading dd still reads both writes in Linux and Solaris in my tests (on Solaris (10 on amd64 at least), reduce to 2 instead of 4 or both writes would block). On FreeBSD, I get only the first write (using 8000 followed by 1 for instance). FreeBSD is also the only one of the three where dd bs=100 count=1 if=/dev/zero | dd bs=100 count=1 | wc -c Doesn't output 100. The others schedule both processes back and forth during their write() and read() system call while the pipe is being filled and emptied several times. -- Stephane ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"