Re: How to get output of piped process?

[Jesse Phillips via Digitalmars-d-learn]

On Saturday, 6 March 2021 at 21:20:30 UTC, kdevel wrote:


```pipechain.d
import std.stdio;
import std.process;
import std.conv;
import std.array;
import std.range;
import std.algorithm;

int main (string [] args)
{
   auto p = pipe ();
   auto proc1 = spawnProcess (["cat"], stdin, p.writeEnd);
   auto q = pipe ();
   auto proc2 = spawnProcess (["cat"], p.readEnd, q.writeEnd);
   auto os = appender!string;
   q.readEnd.byChunk (4096).copy (os);
   auto res2 = wait (proc2);
   auto res1 = wait (proc1);
   stderr.writeln ("res1 = ", res1, ", res2 = ", res2);
   write (os[]);
   stderr.writeln ("fin");
   return 0;
}
```

AFAICS this is immune to SIGPIPE. Do I miss anything?


I probably missed that from the documentation and was trying to 
make it work with pipeProcess.

Re: How to get output of piped process?

[kdevel via Digitalmars-d-learn]

On Saturday, 6 March 2021 at 01:53:15 UTC, Jesse Phillips wrote:
[...]

I think this post is going to answer your need.

https://dev.to/jessekphillips/piping-process-output-1cai

I haven't read all the replies, so maybe you have it working 
and this will benefit someone else.


If I understand your code correctly you have a program "reverse" 
writing
output to another program "check". I am wondering why you have 
two threads
shoveling data between pipes. In the example below data are read 
from stdin
by the first process and is piped to another one which is finally 
piped

into the program and copied into a string appender.

```pipechain.d
import std.stdio;
import std.process;
import std.conv;
import std.array;
import std.range;
import std.algorithm;

int main (string [] args)
{
   auto p = pipe ();
   auto proc1 = spawnProcess (["cat"], stdin, p.writeEnd);
   auto q = pipe ();
   auto proc2 = spawnProcess (["cat"], p.readEnd, q.writeEnd);
   auto os = appender!string;
   q.readEnd.byChunk (4096).copy (os);
   auto res2 = wait (proc2);
   auto res1 = wait (proc1);
   stderr.writeln ("res1 = ", res1, ", res2 = ", res2);
   write (os[]);
   stderr.writeln ("fin");
   return 0;
}
```

AFAICS this is immune to SIGPIPE. Do I miss anything?

Re: How to get output of piped process?

[Imperatorn via Digitalmars-d-learn]

On Wednesday, 3 March 2021 at 20:43:54 UTC, Danny Arends wrote:

On Monday, 22 February 2021 at 14:52:22 UTC, frame wrote:
On Monday, 22 February 2021 at 13:23:40 UTC, Danny Arends 
wrote:



https://github.com/DannyArends/DaNode/blob/master/danode/process.d

Danny


This example shows how easy it is to implement a non-blocking 
stream. Phobos knows this for sockets but not for pipes?


Sockets seem to be more OS independent and are way more mature 
in Phobos.


Pipes seem to have been added as an afterthought in std.process 
and std.stdio


I had to add code for windows to deal with 
non-blocking/buffering pipes, Linux uses fcntl/fileno to enable 
non-blocking


No idea why non-blocking pipes aren't in Phobos, but pipes 
should not be an afterthought but a first class citizen imho


True, we should add this ☀️

Re: How to get output of piped process?

[Jesse Phillips via Digitalmars-d-learn]

On Wednesday, 17 February 2021 at 06:58:55 UTC, Jedi wrote:
I an using pipeShell, I have redirected stdout, stderr, and 
stdin.


I am trying to read from the output and display it in my app. I 
have followed this code almost exactly except I use try wait 
and flush because the app is continuously updating the output. 
(it outputs a progress text on the same line and I'm trying to 
poll it to report to the user)




I think this post is going to answer your need.

https://dev.to/jessekphillips/piping-process-output-1cai

I haven't read all the replies, so maybe you have it working and 
this will benefit someone else.

Re: How to get output of piped process?

[Danny Arends via Digitalmars-d-learn]

On Monday, 22 February 2021 at 14:52:22 UTC, frame wrote:

On Monday, 22 February 2021 at 13:23:40 UTC, Danny Arends wrote:


https://github.com/DannyArends/DaNode/blob/master/danode/process.d

Danny


This example shows how easy it is to implement a non-blocking 
stream. Phobos knows this for sockets but not for pipes?


Sockets seem to be more OS independent and are way more mature in 
Phobos.


Pipes seem to have been added as an afterthought in std.process 
and std.stdio


I had to add code for windows to deal with non-blocking/buffering 
pipes, Linux uses fcntl/fileno to enable non-blocking


No idea why non-blocking pipes aren't in Phobos, but pipes should 
not be an afterthought but a first class citizen imho

Re: How to get output of piped process?

[Danny Arends via Digitalmars-d-learn]

On Tuesday, 23 February 2021 at 10:07:03 UTC, Imperatorn wrote:

On Monday, 22 February 2021 at 13:23:40 UTC, Danny Arends wrote:

On Friday, 19 February 2021 at 15:39:25 UTC, kdevel wrote:

[...]


Perhaps a bit late, but this is how I deal with pipes and 
spawnShell.

Read one byte at a time from stdout and stderr:

https://github.com/DannyArends/DaNode/blob/master/danode/process.d

Danny


Interesting, do you have any benchmarks for DaNode?


I used to run Apache bench on the code to make sure it was as 
fast as possible, for static files it is pretty performant since 
it buffers the files, and serves them directly from memory.


The main overhead comes from the external process booting up, 
rdmd is nice since it only does the compile once then reuses the 
compiled binary for external scripts. PHP and such are always hit 
with the additional overhead.


Feel free to run your own tests, the development branch has the 
latest version with some additional bug fixes not yet available 
in the master branch, and feedback is welcome and can be posted 
as an issue in Github

Re: How to get output of piped process?

[Danny Arends via Digitalmars-d-learn]

On Thursday, 25 February 2021 at 15:28:25 UTC, kdevel wrote:

On Monday, 22 February 2021 at 13:23:40 UTC, Danny Arends wrote:

On Friday, 19 February 2021 at 15:39:25 UTC, kdevel wrote:


[...]

Fortunately the D runtime /does/ take care and it throws---if 
the signal

is ignored beforehand. I filed issue 21649.


[...]


Perhaps a bit late,


It's never too late.™ :-)


but this is how I deal with pipes and spawnShell.
Read one byte at a time from stdout and stderr:

https://github.com/DannyArends/DaNode/blob/master/danode/process.d


Is this immune to SIGPIPE and is this design able to serve 
infinite

streams?


No I have linked up a signal handler to just ignore sigpipe, the 
web server closes connections after not seeing a valid output 
from the script for 5min (e.g. no header)



BTW: Why does run use spawnShell and not spawnProcess (would
save one File object).


I tried different approaches, this one worked for me™ and I just 
went with it. I need the stdin (for the get/post/cookies) stdout 
(php/d/brainf*ck script output) and stderr for the error stream 
from he external script




If the design is not intended to serve infinite streams I would
suggest to open two temporary files "out" and "err", delete 
them,

and let the child process write stdout/stderr into those files.


I used to do that, but it generated a lot of temporary files, and 
I would need to parse in the files after the process is done to 
serve the output to the client. Using pipes is cleaner code wise, 
since I can just stream back the output to the client (e.g. in 
keepalive connections)



IFAICS this avoid threads, sleep, pipe and reading with fgetc.

Re: How to get output of piped process?

[kdevel via Digitalmars-d-learn]

On Monday, 22 February 2021 at 13:23:40 UTC, Danny Arends wrote:

On Friday, 19 February 2021 at 15:39:25 UTC, kdevel wrote:


[...]

Fortunately the D runtime /does/ take care and it throws---if 
the signal

is ignored beforehand. I filed issue 21649.


[...]


Perhaps a bit late,


It's never too late.™ :-)


but this is how I deal with pipes and spawnShell.
Read one byte at a time from stdout and stderr:

https://github.com/DannyArends/DaNode/blob/master/danode/process.d


Is this immune to SIGPIPE and is this design able to serve 
infinite
streams? BTW: Why does run use spawnShell and not spawnProcess 
(would

save one File object).

If the design is not intended to serve infinite streams I would
suggest to open two temporary files "out" and "err", delete them,
and let the child process write stdout/stderr into those files.
IFAICS this avoid threads, sleep, pipe and reading with fgetc.

Re: How to get output of piped process?

[Imperatorn via Digitalmars-d-learn]

On Monday, 22 February 2021 at 13:23:40 UTC, Danny Arends wrote:

On Friday, 19 February 2021 at 15:39:25 UTC, kdevel wrote:

[...]


Perhaps a bit late, but this is how I deal with pipes and 
spawnShell.

Read one byte at a time from stdout and stderr:

https://github.com/DannyArends/DaNode/blob/master/danode/process.d

Danny


Interesting, do you have any benchmarks for DaNode?

Re: How to get output of piped process?

[frame via Digitalmars-d-learn]

On Monday, 22 February 2021 at 13:23:40 UTC, Danny Arends wrote:


https://github.com/DannyArends/DaNode/blob/master/danode/process.d

Danny


This example shows how easy it is to implement a non-blocking 
stream. Phobos knows this for sockets but not for pipes?

Re: How to get output of piped process?

[Danny Arends via Digitalmars-d-learn]

On Friday, 19 February 2021 at 15:39:25 UTC, kdevel wrote:
On Friday, 19 February 2021 at 13:42:46 UTC, Steven 
Schveighoffer wrote:


[...]


[...]


Sure.


[...]


As application programmer I don't want to check any error codes.
Thankfully I don't have to in D. There is a nice off-topic 
example

from the linux kernel [1] what happens when people do not check
return values [1].


[...]


Fortunately the D runtime /does/ take care and it throws---if 
the signal

is ignored beforehand. I filed issue 21649.

[1] 
move RLIMIT_NPROC check from set_user() to 
do_execve_common()


Perhaps a bit late, but this is how I deal with pipes and 
spawnShell.

Read one byte at a time from stdout and stderr:

https://github.com/DannyArends/DaNode/blob/master/danode/process.d

Danny

Re: How to get output of piped process?

[kdevel via Digitalmars-d-learn]

On Friday, 19 February 2021 at 13:42:46 UTC, Steven Schveighoffer 
wrote:


[...]

ignoring SIGPIPE is a process-wide thing, and so it's not 
appropriate for Phobos to make that decision for you. But it's 
trivial to ignore it.


Sure.

I've never been a fan of SIGPIPE. If you look around on the 
Internet, you'll find that most people agree that the reasoning 
for SIGPIPE is to fix poor programming (i.e. ignoring of error 
codes).


As application programmer I don't want to check any error codes.
Thankfully I don't have to in D. There is a nice off-topic example
from the linux kernel [1] what happens when people do not check
return values [1].

But it doesn't give you any good way to handle it. A SIGPIPE 
can be due to any pipe being written, it doesn't tell you which 
one. In order to know which one caused it, well, you have to 
look at the error code of the call!


Fortunately the D runtime /does/ take care and it throws---if the 
signal

is ignored beforehand. I filed issue 21649.

[1] 
move RLIMIT_NPROC check from set_user() to do_execve_common()

Re: How to get output of piped process?

[Steven Schveighoffer via Digitalmars-d-learn]

On 2/19/21 5:41 AM, kdevel wrote:

On Friday, 19 February 2021 at 08:37:50 UTC, Imperatorn wrote:

Does your iopipe handle... Pipes? 😀


BTW: What about SIGPIPE?

In an experimental code I have this

    :
    fout.rawWrite (buf);
    fout.rawWrite ("\n");
    writeln ("flushing");
    fout.flush ();    // (a)
    enforce (! fout.eof, "eof on write to child");    // (b)
    writeln ("reading from pipe");
    :

fout actually is the child's stdin. Sometimes between (a) and (b)
a SIGPIPE occurs and terminates the process (exit code 141). That
prevents the proper handling of eof.

Why isn't SIGPIPE blocked or handled by default?


ignoring SIGPIPE is a process-wide thing, and so it's not appropriate 
for Phobos to make that decision for you. But it's trivial to ignore it.


I've never been a fan of SIGPIPE. If you look around on the Internet, 
you'll find that most people agree that the reasoning for SIGPIPE is to 
fix poor programming (i.e. ignoring of error codes). But it doesn't give 
you any good way to handle it. A SIGPIPE can be due to any pipe being 
written, it doesn't tell you which one. In order to know which one 
caused it, well, you have to look at the error code of the call!


The end result is -- it makes poor programming the standard. If you 
ignore SIGPIPE for a child process, then if that process' operation 
depends on SIGPIPE killing it, then you have screwed over that child 
process, or rather, exposed the lack of error checking in the child process.


-Steve

Re: How to get output of piped process?

[Steven Schveighoffer via Digitalmars-d-learn]

On 2/19/21 3:37 AM, Imperatorn wrote:

On Thursday, 18 February 2021 at 17:27:48 UTC, Steven Schveighoffer wrote:

On 2/18/21 4:40 AM, frame wrote:

[...]


It's just the way it is. Everything in Phobos is a C FILE * (wrapped 
in a File). You need to use alternative i/o systems to get the 
information.


[...]


Does your iopipe handle... Pipes? 😀

Like someone wrote, can it? 🤔


Not yet. Important to remember that iopipe does not do actual i/o, it's 
a library to add *buffering* to i/o.


The std.io (which is the intended source for iopipe) library does not 
support pipes at the moment. But I plan to add them.


https://github.com/MartinNowak/io/issues/24

-Steve

Re: How to get output of piped process?

[kdevel via Digitalmars-d-learn]

On Friday, 19 February 2021 at 08:37:50 UTC, Imperatorn wrote:

Does your iopipe handle... Pipes? 😀


BTW: What about SIGPIPE?

In an experimental code I have this

   :
   fout.rawWrite (buf);
   fout.rawWrite ("\n");
   writeln ("flushing");
   fout.flush ();// (a)
   enforce (! fout.eof, "eof on write to child");// (b)
   writeln ("reading from pipe");
   :

fout actually is the child's stdin. Sometimes between (a) and (b)
a SIGPIPE occurs and terminates the process (exit code 141). That
prevents the proper handling of eof.

Why isn't SIGPIPE blocked or handled by default?

Re: How to get output of piped process?

[Imperatorn via Digitalmars-d-learn]

On Thursday, 18 February 2021 at 17:27:48 UTC, Steven 
Schveighoffer wrote:

On 2/18/21 4:40 AM, frame wrote:

[...]


It's just the way it is. Everything in Phobos is a C FILE * 
(wrapped in a File). You need to use alternative i/o systems to 
get the information.


[...]


Does your iopipe handle... Pipes? 😀

Like someone wrote, can it? 🤔

Re: How to get output of piped process?

[frame via Digitalmars-d-learn]

On Thursday, 18 February 2021 at 17:27:48 UTC, Steven 
Schveighoffer wrote:


readln will block. eof doesn't tell you that there is no data 
in the pipe, it just says whether the pipe has been closed.


Of course, I must have been thinking of another language - I 
should take a coffee before posting.


Is there a d-library that can handle process/named pipes on 
Windows and Linux? I did not found any.

Re: How to get output of piped process?

[Steven Schveighoffer via Digitalmars-d-learn]

On 2/18/21 4:40 AM, frame wrote:

On Thursday, 18 February 2021 at 06:04:13 UTC, Jedi wrote:

Unfortunately, std.process wraps all the pipes in File structs, so 
you have almost no good mechanisms to properly read the data.



WTF?


It's just the way it is. Everything in Phobos is a C FILE * (wrapped in 
a File). You need to use alternative i/o systems to get the information.


you can I believe get the file descriptor out of the File, which should 
help with better mechanisms. Try `fileno` and `windowsHandle`.




I'm wonder about this message. You can always use readln() and eof() on 
such kind of streams. byLine() is just not the best option here.


readln will block. eof doesn't tell you that there is no data in the 
pipe, it just says whether the pipe has been closed.


What you need is non-blocking ways to check "does this pipe have more 
output for me?" If you read the implementation of execute, it gets 
around this by redirecting stderr to stdout (so you can wait on just one 
pipe).


If you need them separated, then you need to do something more 
asynchronous. And Phobos does not wrap that, you have to use OS primitives.


-Steve

Re: How to get output of piped process?

[frame via Digitalmars-d-learn]

On Thursday, 18 February 2021 at 06:04:13 UTC, Jedi wrote:

Unfortunately, std.process wraps all the pipes in File 
structs, so you have almost no good mechanisms to properly 
read the data.



WTF?


-Steve


I'm wonder about this message. You can always use readln() and 
eof() on such kind of streams. byLine() is just not the best 
option here.

Re: How to get output of piped process?

[Jedi via Digitalmars-d-learn]

On Wednesday, 17 February 2021 at 14:36:58 UTC, Steven 
Schveighoffer wrote:

On 2/17/21 1:58 AM, Jedi wrote:
I an using pipeShell, I have redirected stdout, stderr, and 
stdin.


I am trying to read from the output and display it in my app. 
I have followed this code almost exactly except I use try wait 
and flush because the app is continuously updating the output. 
(it outputs a progress text on the same line and I'm trying to 
poll it to report to the user)



auto pipes = pipeProcess("my_application", Redirect.stdout | 
Redirect.stderr);

scope(exit) wait(pipes.pid);

// Store lines of output.
string[] output;
foreach (line; pipes.stdout.byLine) output ~= line.idup;

// Store lines of errors.
string[] errors;
foreach (line; pipes.stderr.byLine) errors ~= line.idup;


My code

auto p = pipeShell(`app.exe "`~f.name~`"`, Redirect.stdout | 
Redirect.stdin | Redirect.stderr);



     while(!tryWait(p.pid).terminated)
     {
     string[] output;
     foreach (line; p.stdout.byLine)


You need to be careful here -- this will wait until stdout is 
*closed*.



     {
     output ~= line.idup;
     writeln(line);
     }

     string[] errors;
     foreach (line; p.stderr.byLine)
     {
     errors ~= line.idup;
     writeln("Err:"~line);
     }


Same thing here.


     }

wait(p.pid);

None of this works though. What is strange is that when I 
close out the debugger the app starts working(no console 
output but I able to see that it is doing something) but is 
very slow.


auto p = executeShell(`app.exe "`~f.name~`"`);

Does work, except I have no output or input. I have another 
app that I do the exact same code and I can get the output and 
parse it, but this is after the app terminates. I imagine the 
issue here is that I'm trying to get the output while the app 
is running.


Without knowing the pattern of what your app is outputting, 
it's hard to tell what will happen.


The most common problem with people dealing with piped output 
is not reading data off the pipe, which then makes the child 
process hang trying to write to the pipe, because the buffer is 
full.


For instance, if your process outputs tons of stuff to stderr, 
you will hang, because you are waiting for stdout to be closed 
first before you read anything from stderr, the child process 
fills up stderr pipe, and is put to sleep waiting for it to be 
writable, never closing stdout.


Unfortunately, std.process wraps all the pipes in File structs, 
so you have almost no good mechanisms to properly read the data.



WTF?


-Steve


Seriously, I can't simply get the output in real time? Come on, 
that is lame, Surely D can do better than that? How hard is it to 
get a buffer? Is there any hacks? How can one communicate with an 
app using std io if one can't actually communicate until the app 
is closed? It makes no sense.


But note that even executeShell doesn't display the output of the 
app.exe so it is more than just pipeShell.


The app just outputs text, just like almost every other text. One 
shouldn't have to know any pattern, that defeats the purpose. I 
should just be able to get the output of the app.exe, and also if 
the app is requesting input. This isn't rocket science but it 
seems someone wants to turn it in to it?


When the app.exe is running it just prints stuff out, every once 
in a while it might ask for input(e.g., to overwrite the file if 
it exists, but I can get around that by checking in D)... but 
ultimately I just want to consolidate the output it gives so I 
need access to it BEFORE the app closes. The app.exe processes 
files, takes some time to do so so if I have to wait to display 
anything nothing will be displayed for a long time.

Re: How to get output of piped process?

[Steven Schveighoffer via Digitalmars-d-learn]

On 2/17/21 1:58 AM, Jedi wrote:

I an using pipeShell, I have redirected stdout, stderr, and stdin.

I am trying to read from the output and display it in my app. I have 
followed this code almost exactly except I use try wait and flush 
because the app is continuously updating the output. (it outputs a 
progress text on the same line and I'm trying to poll it to report to 
the user)



auto pipes = pipeProcess("my_application", Redirect.stdout | 
Redirect.stderr);

scope(exit) wait(pipes.pid);

// Store lines of output.
string[] output;
foreach (line; pipes.stdout.byLine) output ~= line.idup;

// Store lines of errors.
string[] errors;
foreach (line; pipes.stderr.byLine) errors ~= line.idup;


My code

auto p = pipeShell(`app.exe "`~f.name~`"`, Redirect.stdout | 
Redirect.stdin | Redirect.stderr);



     while(!tryWait(p.pid).terminated)
     {
     string[] output;
     foreach (line; p.stdout.byLine)


You need to be careful here -- this will wait until stdout is *closed*.


     {
     output ~= line.idup;
     writeln(line);
     }

     string[] errors;
     foreach (line; p.stderr.byLine)
     {
     errors ~= line.idup;
     writeln("Err:"~line);
     }


Same thing here.


     }

wait(p.pid);

None of this works though. What is strange is that when I close out the 
debugger the app starts working(no console output but I able to see that 
it is doing something) but is very slow.


auto p = executeShell(`app.exe "`~f.name~`"`);

Does work, except I have no output or input. I have another app that I 
do the exact same code and I can get the output and parse it, but this 
is after the app terminates. I imagine the issue here is that I'm trying 
to get the output while the app is running.


Without knowing the pattern of what your app is outputting, it's hard to 
tell what will happen.


The most common problem with people dealing with piped output is not 
reading data off the pipe, which then makes the child process hang 
trying to write to the pipe, because the buffer is full.


For instance, if your process outputs tons of stuff to stderr, you will 
hang, because you are waiting for stdout to be closed first before you 
read anything from stderr, the child process fills up stderr pipe, and 
is put to sleep waiting for it to be writable, never closing stdout.


Unfortunately, std.process wraps all the pipes in File structs, so you 
have almost no good mechanisms to properly read the data.


-Steve

How to get output of piped process?

[Jedi via Digitalmars-d-learn]

I an using pipeShell, I have redirected stdout, stderr, and stdin.

I am trying to read from the output and display it in my app. I 
have followed this code almost exactly except I use try wait and 
flush because the app is continuously updating the output. (it 
outputs a progress text on the same line and I'm trying to poll 
it to report to the user)



auto pipes = pipeProcess("my_application", Redirect.stdout | 
Redirect.stderr);

scope(exit) wait(pipes.pid);

// Store lines of output.
string[] output;
foreach (line; pipes.stdout.byLine) output ~= line.idup;

// Store lines of errors.
string[] errors;
foreach (line; pipes.stderr.byLine) errors ~= line.idup;


My code

auto p = pipeShell(`app.exe "`~f.name~`"`, Redirect.stdout | 
Redirect.stdin | Redirect.stderr);



while(!tryWait(p.pid).terminated)
{
string[] output;
foreach (line; p.stdout.byLine)
{
output ~= line.idup;
writeln(line);
}

string[] errors;
foreach (line; p.stderr.byLine)
{
errors ~= line.idup;
writeln("Err:"~line);
}
}

wait(p.pid);

None of this works though. What is strange is that when I close 
out the debugger the app starts working(no console output but I 
able to see that it is doing something) but is very slow.


auto p = executeShell(`app.exe "`~f.name~`"`);

Does work, except I have no output or input. I have another app 
that I do the exact same code and I can get the output and parse 
it, but this is after the app terminates. I imagine the issue 
here is that I'm trying to get the output while the app is 
running.



I want to be able to get the output so I can reduce much of the 
clutter and give a progress report. I am ok with simply hooking 
up the in and out of the console of the app to mine just as if I 
ran app.exe directly.