[issue10482] subprocess and deadlock avoidance

[Mark Lawrence]

Change by Mark Lawrence :


--
nosy:  -BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10482] subprocess and deadlock avoidance

[Martin Panter]

Martin Panter added the comment:

Related: Issue 1260171, essentially proposing streaming readers and writers for 
communicate() instead of fixed buffers, but without using OS threads.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10482
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10482] subprocess and deadlock avoidance

[Martin Panter]

Changes by Martin Panter vadmium...@gmail.com:


--
nosy: +vadmium

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10482
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10482] subprocess and deadlock avoidance

[Akira Li]

Changes by Akira Li 4kir4...@gmail.com:


--
nosy: +akira

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10482
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10482] subprocess and deadlock avoidance

[Mark Lawrence]

Mark Lawrence added the comment:

@Glenn can you provide a formal patch so we can take this forward?

--
nosy: +BreamoreBoy
versions: +Python 3.5 -Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10482
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10482] subprocess and deadlock avoidance

[Christian Heimes]

Changes by Christian Heimes li...@cheimes.de:


--
stage:  - needs patch
versions: +Python 3.4 -Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10482
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10482] subprocess and deadlock avoidance

[Ross Lagerwall]

Changes by Ross Lagerwall rosslagerw...@gmail.com:


--
nosy: +rosslagerwall

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10482
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10482] subprocess and deadlock avoidance

[Jon Brandvein]

Changes by Jon Brandvein jon.brandv...@gmail.com:


--
nosy: +brandj

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10482
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10482] subprocess and deadlock avoidance

[Glenn Linderman]

Glenn Linderman v+pyt...@g.nevcal.com added the comment:

Looking at the code the way I've used it in my modified server.py:

stderr = []
stderr_thread = threading.Thread(target=self._readerthread,
 args=(p.stderr, stderr))
stderr_thread.daemon = True
stderr_thread.start()

self.log_message(writer: %s % str( nbytes ))
stdin_thread = threading.Thread(target=self._writerthread,
args=(self.rfile, p.stdin, nbytes))
stdin_thread.daemon = True
stdin_thread.start()

and later

stderr_thread.join()
stdin_thread.join()

p.stderr.close()
p.stdout.close()

if stderr:
stderr = stderr[ 0 ].decode(UTF-8)

It seems like this sort of code (possibly passing in the encoding) could be 
bundled back inside subprocess (I borrowed it from there).

It also seems from recent discussion on npopdev that the cheat-sheet how to 
replace other sys and os popen functions would be better done as wrapper 
functions for the various cases.  Someone pointed out that the hard cases 
probably aren't cross-platform, but that currently the easy cases all get 
harder when using subprocess than when using the deprecated facilities.  They 
shouldn't.  The names may need to be a bit more verbose to separate the various 
use cases, but each use case should remain at least as simple as the prior 
function.

So perhaps instead of just  subprocess.PIPE  to select particular handling for 
stdin, stdout, and stderr, subprocess should implement some varieties to handle 
attaching  different types of reader and writer threads to the handles... of 
course, parameters need to come along for the ride too: maybe the the 
additional variations would be object references with parameters supplied, 
instead of just a manifest constant like .PIPE.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10482
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10482] subprocess and deadlock avoidance

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

Or various incarnations of functools.partial applied to subprocess.Popen.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10482
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10482] subprocess and deadlock avoidance

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

The general idea is sound. My work colleagues have certainly had to implement 
their own reader/writer thread equivalents to keep subprocess from blocking.

It makes sense to provide more robust public support for such techniques in 
process itself.

--
nosy: +ncoghlan
versions: +Python 3.3 -Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10482
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10482] subprocess and deadlock avoidance

[Glenn Linderman]

Glenn Linderman v+pyt...@g.nevcal.com added the comment:

Here's an updated _writerthread idea that handles more cases:

def _writerthread(self, fhr, fhw, length=None):
if length is None:
flag = True
while flag:
buf = fhr.read( 512 )
fhw.write( buf )
if len( buf ) == 0:
flag = False
else:
while length  0:
buf = fhr.read( min( 512, length ))
fhw.write( buf )
length -= len( buf )
# throw away additional data [see bug #427345]
while select.select([fhr._sock], [], [], 0)[0]:
if not fhr._sock.recv(1):
break
fhw.close()

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10482
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10482] subprocess and deadlock avoidance

[Glenn Linderman]

Glenn Linderman v+pyt...@g.nevcal.com added the comment:

Sorry, left some extraneous code in the last message, here is the right code:

def _writerthread(self, fhr, fhw, length=None):
if length is None:
flag = True
while flag:
buf = fhr.read( 512 )
fhw.write( buf )
if len( buf ) == 0:
flag = False
else:
while length  0:
buf = fhr.read( min( 512, length ))
fhw.write( buf )
length -= len( buf )
fhw.close()

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10482
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10482] subprocess and deadlock avoidance

[Glenn Linderman]

Glenn Linderman v+pyt...@g.nevcal.com added the comment:

So I've experimented a bit, and it looks like simply exposing ._readerthread as 
an external API would handle the buffered case for stdout or stderr.  For 
http.server CGI scripts, I think it is fine to buffer stderr, as it should not 
be a high-volume channel... but not both stderr and stdout, as stdout can be 
huge.  And not stdin, because it can be huge also.

For stdin, something like the following might work nicely for some cases, 
including http.server (with revisions):

def _writerthread(self, fhr, fhw, length):
while length  0:
buf = fhr.read( min( 8196, length ))
fhw.write( buf )
length -= len( buf )
fhw.close()

When the stdin data is buffered, but the application wishes to be stdout 
centric instead of stdin centric (like the current ._communicate code), a 
variation could be made replacing fhr by a data buffer, and writing it 
gradually (or fully) to the pipe, but from a secondary thread.

Happily, this sort of code (the above is extracted from a test version of 
http.server) can be implemented in the server, but would be more usefully 
provided by subprocess, in my opinion.

To include the above code inside subprocess would just be a matter of tweaking 
references to class members instead of parameters.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10482
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10482] subprocess and deadlock avoidance

[Glenn Linderman]

New submission from Glenn Linderman v+pyt...@g.nevcal.com:

.communicate is a nice API for programs that produce little output, and can be 
buffered.  While that may cover a wide range of uses, it doesn't cover 
launching CGI programs, such as is done in http.server.  Now there are nice 
warnings about that issue in the http.server documentation.

However, while .communicate has the building blocks to provide more general 
solutions, it doesn't expose them to the user, nor does it separate them into 
building blocks, rather it is a monolith inside ._communicate.

For example, it would be nice to have an API that would capture a stream using 
a thread which could be used for either stdout or stdin, and is what 
._communicate does under the covers for both of them.

It would also be nice to have an API that would pump a bytestream to .stdin as 
a background thread.  ._communicate doesn't provide that one, but uses the 
foreground thread for that.  And, it requires that it be fully buffered.  It 
would be most useful for http.server if this API could connect a file handle 
and an optional maximum read count to .stdin, yet do it in a background thread.

That would leave the foreground thread able to process stdout.  It is correct 
(but not what http.server presently does, but I'll be entering that enhancement 
request soon) for http.server to read the first line from the CGI program, 
transform it, add a few more headers, and send that to the browser, and then 
hook up .stdout to the browser (shutil.copyfileobj can be used for the rest of 
the stream).  However, there is no deadlock free way of achieving this sort of 
solution, capturing the stderr to be logged, not needing to buffer a 
potentially large file upload, and transforming the stdout, with the facilities 
currently provided by subprocess.  Breaking apart some of the existing building 
blocks, and adding an additional one for .stdin processing would allow a real 
http.server implementation, as well as being more general for other complex 
uses.

You see, for http.server, the stdin

--
components: Library (Lib)
messages: 121871
nosy: v+python
priority: normal
severity: normal
status: open
title: subprocess and deadlock avoidance
type: feature request
versions: Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10482
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com