Chet Ramey wrote:
On 10/16/15 7:52 PM, Linda Walsh wrote:
As I mentioned, my initial take on implementation was
using standard pipes instead of named pipes (not having read
or perhaps having glossed over the 'named pipes' aspect).
I think you're missing that process substitution is a word expansion
that is defined to expand to a filename. When it uses /dev/fd, it
uses pipes and exposes that pipe to the process as a filename in
/dev/fd. Named pipes are an alternative for systems that don't support
/dev/fd.
-----
??? I've never seen a usage where it expands to a filename and
is treated as such.
Are you meaning:
readarray foo </etc/passwd (being a read from filename case).
vs.
readarray foo < <(cat /etc/passwd), and in this case, "<(...)"
is creating a process ... that puts the input file on some "/dev/fd/xx",
and that readarray (or read)
is then "handling that" as though it were a normal file?
That's what you are meaning by process substitution?...
I wasn't understanding the literalness of the wording...(?)
But read or readarray, read from "file handle"s, not filenames --
as in:
"cat /etc/passwd |readarray foo"
The only conceptual difference between that and
readarray foo < <(cat /etc/passwd) is whether or not the readarray
is done in the parent or the child, .. i.e.
from a semantic point of view, how is:
readarray foo < <(cat /etc/passwd)
different from
shopt -s lastpipe
cat /etc/passwd |readarray foo
Is there something in the semantics that would require they
be implemented differently?
While using pipes for here-documents is possible, the original file-based
implementation (and I mean original -- it's almost 28 years old now) is
simple and works. Inserting another process just to keep writing data to
a pipe (many here documents are larger than the OS pipe buffer size), deal
with write errors and SIGPIPE asynchronously, and add a few pipe file
descriptors to manage, seems like more trouble than benefit. Files are
simple, can handle arbitrary amounts of data, and don't require an
additional process or bookkeeping.
----
But they do require a tempstore that wasn't present in a system-start
bash script I was writing -- kept having problems with the different
bash communication methods all needing some external-fs names to be
present to handle what could have been done in memory.
BTW, Dealing with the SIGPIPE is why I kept the output pipe open in
the reader process, and use a child-sig handler to close the specific
pipe -- I.e. when I wrote similar code in perl, I kept track of child
processes->pipe pairs, so my foreground process could close the
"writer-pipe" only after the reader-pipe had everything. Alternatively,
Could use a pair of pipes, and parent could send a message to child
process for more data, or close the pipe) -- a handshake, as it were.
As far as may docs being larger than the buffer size -- shouldn't be
an issue unless you are trying 2-way communication -- but that has lots
of warnings about each reader waiting for output from the other as
a writer and the need for asynch/non-blocking design.
Writing to a tmp file requires enough room for the entire tmp doc on
disk. My tmp right now only has 4.6Available. If some heredoc tried to
encode the contents of a video, or Blue-ray data disk, and then
something tried to unpack it to tmp first, it would die --- but with
pipes, you don't have to send the whole thing at once -- that requires
basically full space on sender and receiver or 2x the HEREDOC space. If
it could hold it in memory... my linux box has 2x48G (2 NUMA nodes) or
~96G + ~8G Swap. So I'd say it really is case dependent -- ideally if
bash was going to be enhanced in this area, it allow the user to set the
defaults to use processes/pipes or tmp files.
The problem I had with bash & tmps, was that bash scripts started
execution off of 'root' before a separate FS for /tmp was mounted. The
root partition has even less space available on it than /tmp. So the
use of a /tmp file when /tmp wasn't really available yet was an issue.
here-strings are similar, and there's no reason to have here strings use
a different mechanism than here documents.
There was also a backwards compatibility issue. Since historical shells
had used temp files for here-documents, there was a set of applications
that assumed stdin was seekable as long as it wasn't a terminal, and it was
important to support them.
----
Oh? What POSIX compatible applications?
Yes. You are missing the fundamental nature of process substitution:
a word expansion that results in a filename.
---
hopefully my q's at top were more on track?
If you think that you can improve the implementation of here
documents and here strings by using a pipe instead of a temp
file, by all means take a shot at it. You might want to get
ahold of some old Unix `shar' archives to test your implementation
with very large here-documents.
----
Or, I might read those same shar achives today directly
into memory -- depends on system resources.
