I'm bumping this PEP again in hopes of getting some feedback.
Thanks,
Eric
On Tue, Sep 8, 2009 at 23:52, Eric Pruitt <eric.pru...@gmail.com> wrote:
> PEP: 3145
> Title: Asynchronous I/O For subprocess.Popen
> Author: (James) Eric Pruitt, Charles R. McCreary, Josiah Carlson
> Type: Standards Track
> Content-Type: text/plain
> Created: 04-Aug-2009
> Python-Version: 3.2
>
> Abstract:
>
> In its present form, the subprocess.Popen implementation is prone to
> dead-locking and blocking of the parent Python script while waiting on data
> from the child process.
>
> Motivation:
>
> A search for "python asynchronous subprocess" will turn up numerous
> accounts of people wanting to execute a child process and communicate with
> it from time to time reading only the data that is available instead of
> blocking to wait for the program to produce data [1] [2] [3]. The current
> behavior of the subprocess module is that when a user sends or receives
> data via the stdin, stderr and stdout file objects, dead locks are common
> and documented [4] [5]. While communicate can be used to alleviate some of
> the buffering issues, it will still cause the parent process to block while
> attempting to read data when none is available to be read from the child
> process.
>
> Rationale:
>
> There is a documented need for asynchronous, non-blocking functionality in
> subprocess.Popen [6] [7] [2] [3]. Inclusion of the code would improve the
> utility of the Python standard library that can be used on Unix based and
> Windows builds of Python. Practically every I/O object in Python has a
> file-like wrapper of some sort. Sockets already act as such and for
> strings there is StringIO. Popen can be made to act like a file by simply
> using the methods attached the the subprocess.Popen.stderr, stdout and
> stdin file-like objects. But when using the read and write methods of
> those options, you do not have the benefit of asynchronous I/O. In the
> proposed solution the wrapper wraps the asynchronous methods to mimic a
> file object.
>
> Reference Implementation:
>
> I have been maintaining a Google Code repository that contains all of my
> changes including tests and documentation [9] as well as blog detailing
> the problems I have come across in the development process [10].
>
> I have been working on implementing non-blocking asynchronous I/O in the
> subprocess.Popen module as well as a wrapper class for subprocess.Popen
> that makes it so that an executed process can take the place of a file by
> duplicating all of the methods and attributes that file objects have.
>
> There are two base functions that have been added to the subprocess.Popen
> class: Popen.send and Popen._recv, each with two separate implementations,
> one for Windows and one for Unix based systems. The Windows
> implementation uses ctypes to access the functions needed to control pipes
> in the kernel 32 DLL in an asynchronous manner. On Unix based systems,
> the Python interface for file control serves the same purpose. The
> different implementations of Popen.send and Popen._recv have identical
> arguments to make code that uses these functions work across multiple
> platforms.
>
> When calling the Popen._recv function, it requires the pipe name be
> passed as an argument so there exists the Popen.recv function that passes
> selects stdout as the pipe for Popen._recv by default. Popen.recv_err
> selects stderr as the pipe by default. "Popen.recv" and "Popen.recv_err"
> are much easier to read and understand than "Popen._recv('stdout' ..." and
> "Popen._recv('stderr' ..." respectively.
>
> Since the Popen._recv function does not wait on data to be produced
> before returning a value, it may return empty bytes. Popen.asyncread
> handles this issue by returning all data read over a given time
> interval.
>
> The ProcessIOWrapper class uses the asyncread and asyncwrite functions to
> allow a process to act like a file so that there are no blocking issues
> that can arise from using the stdout and stdin file objects produced from
> a subprocess.Popen call.
>
>
> References:
>
> [1] [ python-Feature Requests-1191964 ] asynchronous Subprocess
> http://mail.python.org/pipermail/python-bugs-list/2006-December/
> 036524.html
>
> [2] Daily Life in an Ivory Basement : /feb-07/problems-with-subprocess
> http://ivory.idyll.org/blog/feb-07/problems-with-subprocess
>
> [3] How can I run an external command asynchronously from Python? - Stack
> Overflow
> http://stackoverflow.com/questions/636561/how-can-i-run-an-external-
> command-asynchronously-from-python
>
> [4] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation
> http://docs.python.org/library/subprocess.html#subprocess.Popen.wait
>
> [5] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation
> http://docs.python.org/library/subprocess.html#subprocess.Popen.kill
>
> [6] Issue 1191964: asynchronous Subprocess - Python tracker
> http://bugs.python.org/issue1191964
>
> [7] Module to allow Asynchronous subprocess use on Windows and Posix
> platforms - ActiveState Code
> http://code.activestate.com/recipes/440554/
>
> [8] subprocess.rst - subprocdev - Project Hosting on Google Code
>
> http://code.google.com/p/subprocdev/source/browse/doc/subprocess.rst?spec=svn2c925e935cad0166d5da85e37c742d8e7f609de5&r=2c925e935cad0166d5da85e37c742d8e7f609de5#437
>
> [9] subprocdev - Project Hosting on Google Code
> http://code.google.com/p/subprocdev
>
> [10] Python Subprocess Dev
> http://subdev.blogspot.com/
>
> Copyright:
>
> This P.E.P. is licensed under the Open Publication License;
> http://www.opencontent.org/openpub/.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
