Re: [Python-Dev] Fwd: subprocess.Popen(.... stdout=IGNORE, ...)
On Tue, 13 Jun 2006, Martin Blais wrote:
Hi all. Now let's see if I remember something about my module...
In the subprocess module, by default the files handles in the child
are inherited from the parent. To ignore a child's output, I can use
the stdout or stderr options to send the output to a pipe::
p = Popen(command, stdout=PIPE, stderr=PIPE)
However, this is sensitive to the buffer deadlock problem, where for
example the buffer for stderr might become full and a deadlock occurs
because the child is blocked on writing to stderr and the parent is
blocked on reading from stdout or waiting for the child to finish.
For example, using this command will cause deadlock::
call('cat /boot/vmlinuz'.split(), stdout=PIPE, stderr=PIPE)
Yes, the call convenience function is basically for the case when you are
not interested in redirection.
Popen.communicate() implements a solution using either select() or
multiple threads (under Windows) to read from the pipes, and returns
the strings as a result. It works out like this::
p = Popen(command, stdout=PIPE, stderr=PIPE)
output, errors = p.communicate()
if p.returncode != 0:
?
Now, as a user of the subprocess module, sometimes I just want to
call some child process and simply ignore its output, and to do so I
am forced to use communicate() as above and wastefully capture and
ignore the strings. This is actually quite a common use case. Just
run something, and check the return code.
Yes, this is a common case, and using communicate() is indeed overkill and
wasteful.
Right now, in order to do
this without polluting the parent's output, you cannot use the call()
convenience (or is there another way?).
A workaround that works under UNIX is to do this::
FNULL = open('/dev/null', 'w')
returncode = call(command, stdout=FNULL, stderr=FNULL)
Yes, this works. You can also do:
returncode = subprocess.call(command, stdout=open('/dev/null', 'w'),
stderr=subprocess.STDOUT)
Some feedback requested, I'd like to know what you think:
1. Would it not be nice to add a IGNORE constant to subprocess.py
that would do this automatically?, i.e. ::
returncode = call(command, stdout=IGNORE, stderr=IGNORE)
Rather than capture and accumulate the output, it would find an
appropriate OS-specific way to ignore the output (the /dev/null file
above works well under UNIX, how would you do this under Windows?
I'm sure we can find something.)
I have a vague feeling of that this has been discussed before, but I
cannot find a tracker for this. I guess an IGNORE constant would be nice.
Using open('/dev/null', 'w') is only a few more characters to type, but as
you say, it's not platform independent.
So, feel free to submit a patch or a Feature Request Tracker.
2. call() should be modified to not be sensitive to the deadlock
problem, since its interface provides no way to return the
contents of the output. The IGNORE value provides a possible
solution for this.
How do you suggest the call() should be modified? I'm not really sure it
can do more without being more complicated. Being simple is the main
purpose of call().
3. With the /dev/null file solution, the following code actually
works without deadlock, because stderr is never blocked on writing
to /dev/null::
p = Popen(command, stdout=PIPE, stderr=IGNORE)
text = p.stdout.read()
retcode = p.wait()
Any idea how this idiom could be supported using a more portable
solution (i.e. how would I make this idiom under Windows, is there
some equivalent to /dev/null)?
Yes, as Terry Reedy points out, NUL: can be used.
Regards,
/Peter Åstrand [EMAIL PROTECTED]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Draft PEP to make file objects support non-blocking mode.
On Mon, 21 Mar 2005, Donovan Baarda wrote: I don't agree with that. There's no need to use non-blocking I/O when using select(), and in fact things are less confusing if you don't. You would think that... and the fact that select, popen2 etc all use file objects encourage you to think that. However, this is a trap that can catch you out badly. Check the attached python scripts that demonstrate the problem. This is no trap. When select() indicates that you can write or read, it means that you can write or read at least one byte. The .read() and .write() file methods, however, always writes and reads *everything*. These works, basically, just like fread()/fwrite(). The only ways to ensure that a select process does not block like this, without using non-blocking mode, are; 1) use a buffer size of 1 in the select process. 2) understand the child process's read/write behaviour and adjust the selector process accordingly... ie by making the buffer sizes just right for the child process, 3) Use os.read / os.write. The read method's current behaviour needs to be documented, so its actual behaviour can be used to differentiate between an empty non-blocking read, and EOF. This means recording that IOError(EAGAIN) is raised for an empty non-blocking read. Isn't that unix-specific? The file object is supposed to provide a more or less platform-independent interface, I thought. I think the fread/fwrite and read/write behaviour is posix standard and possibly C standard stuff... so it _should_ be the same on other platforms. Sorry if I've misunderstood your point, but fread()/fwrite() does not return EAGAIN. /Peter Åstrand [EMAIL PROTECTED] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Draft PEP to make file objects support non-blocking mode.
On Mon, 21 Mar 2005, Donovan Baarda wrote: The only ways to ensure that a select process does not block like this, without using non-blocking mode, are; 3) Use os.read / os.write. [...] but os.read / os.write will block too. No. Try it... replace the file read/writes in selector.py. They will only do partial reads if the file is put into non-blocking mode. I've just tried it; I replaced: data = o.read(BUFF_SIZE) with: data = os.read(o.fileno(), BUFF_SIZE) Works for me without any hangs. Another example is the subprocess module, which does not use non-blocking mode in any way. (If you are using pipes, however, you shouldn't write more than PIPE_BUF bytes in each write.) I think the fread/fwrite and read/write behaviour is posix standard and possibly C standard stuff... so it _should_ be the same on other platforms. Sorry if I've misunderstood your point, but fread()/fwrite() does not return EAGAIN. no, fread()/fwrite() will return 0 if nothing was read/written, and ferror() will return EAGAIN to indicated that it was a would block condition at least I think it does... the man page simply says ferror() returns a non-zero value. fread() should loop internally on EAGAIN, in blocking mode. /Peter Åstrand [EMAIL PROTECTED] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding any() and all()
On Fri, 11 Mar 2005, Paul Moore wrote: Not sure this is pertinent but anyway: any and all are often used as variable names. all especially often and then almost always as a list of something. It would not be good to add all to the list of words to watch out for. Also, all is usually thought of as a list of Using any and all as variables hides the builtins, but doesn't disallow their use elsewhere. Personally, though, I wouldn't use any or all as variable names, so that's a style issue. Even though you can use them as variables (and shadow the builtins), you will still get warnings from pychecker. The code will also be harder to read: When you see all in the middle of some code, you don't know if it's referring to the builtin or a variable. Personally, I think Python has too many builtins already. /Peter Åstrand [EMAIL PROTECTED] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
