[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell

[wabu]

New submission from wabu:

using `p = create_subprocess_exec(..., stdout=subprocess.PIPE, limit=...)`, 
p.stdout has not transport set, so the underlying protocol is unable to pause 
the reading of the transport, resulting in high memory usage when slowly 
consuming input from p.stdout, even if the limit parameter is passed. 

A workaround is to set the transport manually after creating the subprocess:
`p.stdout.set_transport(p._transport.get_pipe_transport(1))`, but this should 
happen inside the create_subprocess call.

--
components: asyncio
messages: 229763
nosy: gvanrossum, haypo, wabu, yselivanov
priority: normal
severity: normal
status: open
title: memory leak: no transport for pipes by create_subprocess_exec/shell
versions: Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22685
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell

[wabu]

wabu added the comment:

Sorry for the confusion, yes i do the yield from. The stdout stream for the 
process is actually producing data as it should. The subprocess produces a high 
amount of data (pbzip2), but is only consumed slowly. 

Normally when the buffer limit is reached for a stream reader, it calls 
pause_reading on the transport inside the feed_data method (see 
https://code.google.com/p/tulip/source/browse/asyncio/streams.py#365),
but here this is not happening, as the returned reader has no transport set 
(p.stdout._transport == None). So it fills up all the memory.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22685
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell

[wabu]

wabu added the comment:

Here's a more complete example

@coroutine
put_data(filename, queue, chunksize=16000):
pbzip2 = yield from asyncio.create_subprocess_exec(
 'pbzip2', '-cd', filename, 
 stdout=asyncio.subprocess.PIPE, limit=self.chunksize*2)

while not pbzip2.stdout.at_eof():
data = yield from pbzip2.stdout.read(chunksize)
yield from queue.put(data)

adding the workaround after createing the stream fixes the issue:
pbzip2.stdout.set_transport(pbzip2._transport.get_pipe_transport(1))

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22685
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell

[wabu]

wabu added the comment:

thanks a lot, the fix works!

On 21.10.2014 22:16, Guido van Rossum wrote:
 
 Guido van Rossum added the comment:
 
 Can you confirm that this patch fixes the problem (without you needing the 
 workaround in your own code)?
 
 --
 keywords: +patch
 Added file: http://bugs.python.org/file36989/fix22685.patch
 
 ___
 Python tracker rep...@bugs.python.org
 http://bugs.python.org/issue22685
 ___


--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22685
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell

[wabu]

wabu added the comment:

On 21.10.2014 22:41, Guido van Rossum wrote:
 Guido van Rossum added the comment:
 
 Victor, do you think this needs a unittest? It seems kind of difficult to
 test for whether memory fills up (the machine may get wedged if it does :-).

You could setup a the subprocess with
asyncio.async(asyncio.create_subprocess_exec(...)) and then let the
asyncio loop run for a limited time with
loop.run_until_complete(asyncio.sleep(.1)), watching carefully for
higher memory usage after each sleep.

But still it's difficult to create a reliable unit-test with this ...

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22685
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell

[wabu]

wabu added the comment:

thanks for the fixes 'n' integration

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22685
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com