Re: How do I compose pipes?
On Friday, 29 January 2021 at 03:49:38 UTC, Ali Çehreli wrote:
On 1/28/21 3:45 PM, Anthony wrote:
> void end(AccumulatorPipe acc) {
> auto pids = acc.pids ~ P.spawnShell("cat", acc.stdout);
>
> foreach (pid; pids) {
> P.wait(pid);
> }
> }
> ```
>
>
> So now I can do something like:
> ```
> run("find source -name '*.d'")
> .pipe("entr ./make.d tests")
> .end(),
Cool but there should be one improvement because I don't think
end() is guaranteed to be executed in that code, which may
leave zombie processes around. From 'man waitpid':
"A child that terminates, but has not been waited for becomes
a "zombie".
Which is relayed to std.process documentation as "to avoid
child processes becoming "zombies"".
Ali
I take it you're referring to missing scope guards like in your
code
`scope (exit) wait(lsPid);`
Yeah, that is a tricky one. I can't think of a way to have a nice
interface that also closes the pids on exit since scope guards
are handled on function exit in this case.
Perhaps thats just the nature of the problem though.
I'll take a look at what scriptlike does
https://github.com/Abscissa/scriptlike#script-style-shell-commands
Re: How do I compose pipes?
On 1/28/21 3:45 PM, Anthony wrote:
> void end(AccumulatorPipe acc) {
> auto pids = acc.pids ~ P.spawnShell("cat", acc.stdout);
>
> foreach (pid; pids) {
> P.wait(pid);
> }
> }
> ```
>
>
> So now I can do something like:
> ```
> run("find source -name '*.d'")
> .pipe("entr ./make.d tests")
> .end(),
Cool but there should be one improvement because I don't think end() is
guaranteed to be executed in that code, which may leave zombie processes
around. From 'man waitpid':
"A child that terminates, but has not been waited for becomes a "zombie".
Which is relayed to std.process documentation as "to avoid child
processes becoming "zombies"".
Ali
Re: How do I compose pipes?
On Thursday, 28 January 2021 at 17:18:46 UTC, Ali Çehreli wrote:
On 1/28/21 2:16 AM, Anthony wrote:
> auto p = pipeProcess("ls");
> auto q = pipeProcess("cat", stdin = p.stdout); //it would be
good to do
That would work if `cat` received the *contents* of the files
(and with a "-" command line switch). Since `ls` produces file
names, you would have to make the complete `cat` command line
from `ls`'s output.
> Do I need to manually extract the output from
pipes.stdin.readln
Seems to be so for the `ls | cat` case. But the following `find
| grep` example shows how two ends of pipes can be connected:
import std.stdio;
import std.process;
import std.range;
// BONUS: Enable one of the following lines to enjoy an issue.
// version = bonus_bug;
// version = bonus_bug_but_this_works;
void main() {
// Writes to 'a':
auto a = pipe();
auto lsPid = spawnProcess([ "find", "."], stdin, a.writeEnd);
scope (exit) wait(lsPid);
// Reads from 'a', writes to 'b':
auto b = pipe();
auto catPid = spawnProcess([ "grep", "-e", `\.d$` ],
a.readEnd, b.writeEnd);
scope (exit) wait(catPid);
version (bonus_bug) {
// Fails with the following error.
//
// "/usr/include/dmd/phobos/std/typecons.d(6540): Error:
// `"Attempted to access an uninitialized payload."`"
writefln!"Some of the D source files under the current
directory:\n%-( %s\n%)"(
b.readEnd.byLine);
} else version (bonus_bug_but_this_works) {
// Note .take at the end:
writefln!"Some of the D source files under the current
directory:\n%-( %s\n%)"(
b.readEnd.byLine.take(1000));
} else {
// The results are read from 'b':
writeln(b.readEnd.byLine);
}
}
I've discovered a strange issue, which can be observed by
uncommenting the 'version = bonus_bug;' line above. But comment
that one back in and uncomment the next line, now it works. (?)
Ali
Thanks Ali.
I was messing around and below seems to work well enough for me.
```
struct AccumulatorPipe {
Pid[] pids;
File stdin;
File stdout;
}
AccumulatorPipe run(string cmd) {
AccumulatorPipe acc;
auto p = P.pipeShell(cmd, P.Redirect.stdout);
acc.pids ~= p.pid;
acc.stdout = p.stdout;
return acc;
}
AccumulatorPipe pipe(AccumulatorPipe acc0, string cmd) {
AccumulatorPipe acc;
Pipe p = P.pipe();
acc.stdin = p.writeEnd;
acc.stdout = p.readEnd;
auto pid = P.spawnShell(cmd, acc0.stdout, acc.stdin);
acc.pids = acc0.pids ~ pid;
return acc;
}
void end(AccumulatorPipe acc) {
auto pids = acc.pids ~ P.spawnShell("cat", acc.stdout);
foreach (pid; pids) {
P.wait(pid);
}
}
```
So now I can do something like:
```
run("find source -name '*.d'")
.pipe("entr ./make.d tests")
.end(),
```
That would work if `cat` received the *contents* of the files
(and with a "-" command line switch)
I was actually trying to use cat to just spit out the filenames
of the directory as a test.
But I see what you mean.
Re: How do I compose pipes?
On 1/28/21 2:16 AM, Anthony wrote:
> auto p = pipeProcess("ls");
> auto q = pipeProcess("cat", stdin = p.stdout); //it would be good to do
That would work if `cat` received the *contents* of the files (and with
a "-" command line switch). Since `ls` produces file names, you would
have to make the complete `cat` command line from `ls`'s output.
> Do I need to manually extract the output from pipes.stdin.readln
Seems to be so for the `ls | cat` case. But the following `find | grep`
example shows how two ends of pipes can be connected:
import std.stdio;
import std.process;
import std.range;
// BONUS: Enable one of the following lines to enjoy an issue.
// version = bonus_bug;
// version = bonus_bug_but_this_works;
void main() {
// Writes to 'a':
auto a = pipe();
auto lsPid = spawnProcess([ "find", "."], stdin, a.writeEnd);
scope (exit) wait(lsPid);
// Reads from 'a', writes to 'b':
auto b = pipe();
auto catPid = spawnProcess([ "grep", "-e", `\.d$` ], a.readEnd,
b.writeEnd);
scope (exit) wait(catPid);
version (bonus_bug) {
// Fails with the following error.
//
// "/usr/include/dmd/phobos/std/typecons.d(6540): Error:
// `"Attempted to access an uninitialized payload."`"
writefln!"Some of the D source files under the current
directory:\n%-( %s\n%)"(
b.readEnd.byLine);
} else version (bonus_bug_but_this_works) {
// Note .take at the end:
writefln!"Some of the D source files under the current
directory:\n%-( %s\n%)"(
b.readEnd.byLine.take(1000));
} else {
// The results are read from 'b':
writeln(b.readEnd.byLine);
}
}
I've discovered a strange issue, which can be observed by uncommenting
the 'version = bonus_bug;' line above. But comment that one back in and
uncomment the next line, now it works. (?)
Ali
How do I compose pipes?
This post https://dlang.org/library/std/process/pipe.html
mentions:
Pipes can, for example, be used for interprocess communication
by spawning a new process and passing one end of the pipe to the
child, while the parent uses the other end. (See also
pipeProcess and pipeShell for an easier way of doing this.)
```
auto p = pipe();
auto outFile = File("D downloads.txt", "w");
auto cpid = spawnProcess(["curl",
"http://dlang.org/download.html;],
stdin, p.writeEnd);
scope(exit) wait(cpid);
auto gpid = spawnProcess(["grep", "-o", `http://\S*\.zip`],
p.readEnd, outFile);
scope(exit) wait(gpid);
```
How am I able to compose the pipes from pipeProcess if I can't
pass in an existing pipe into the function. Eg.
```
auto p = pipeProcess("ls");
auto q = pipeProcess("cat", stdin = p.stdout); //it would be good
to do this or something like it
```
Do I need to manually extract the output from pipes.stdin.readln
and place it in pipes.stdout.writeln?
Or is there an easier way to do this?
Thanks
