[issue41586] Allow to set pipe size on subprocess.Popen.

[Gregory P. Smith]

Change by Gregory P. Smith :


--
resolution:  -> fixed
stage: patch review -> commit review
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41586] Allow to set pipe size on subprocess.Popen.

[Gregory P. Smith]

Gregory P. Smith  added the comment:


New changeset 786addd9d07b6c712b8ea9ee06e1f9f41c1b67a1 by Gregory P. Smith in 
branch 'master':
bpo-41586: Attempt to make the pipesize tests more robust. (GH-22839)
https://github.com/python/cpython/commit/786addd9d07b6c712b8ea9ee06e1f9f41c1b67a1


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41586] Allow to set pipe size on subprocess.Popen.

[Gregory P. Smith]

Change by Gregory P. Smith :


--
pull_requests: +21793
stage: commit review -> patch review
pull_request: https://github.com/python/cpython/pull/22839

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41586] Allow to set pipe size on subprocess.Popen.

[Gregory P. Smith]

Gregory P. Smith  added the comment:

this caused a variety of buildbot failures.  investigating.

--
resolution: fixed -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41586] Allow to set pipe size on subprocess.Popen.

[Gregory P. Smith]

Gregory P. Smith  added the comment:

Thanks Ruben!

--
resolution:  -> fixed
stage: patch review -> commit review
status: open -> closed
type:  -> enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41586] Allow to set pipe size on subprocess.Popen.

[Gregory P. Smith]

Gregory P. Smith  added the comment:


New changeset 23c0fb8edd16fe6d796df2853a5369fd783e05b7 by Ruben Vorderman in 
branch 'master':
bpo-41586: Add pipesize parameter to subprocess & F_GETPIPE_SZ and F_SETPIPE_SZ 
to fcntl. (GH-21921)
https://github.com/python/cpython/commit/23c0fb8edd16fe6d796df2853a5369fd783e05b7


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41586] Allow to set pipe size on subprocess.Popen.

[Gregory P. Smith]

Change by Gregory P. Smith :


--
assignee:  -> gregory.p.smith
nosy: +gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41586] Allow to set pipe size on subprocess.Popen.

[Ruben Vorderman]

Change by Ruben Vorderman :


--
keywords: +patch
pull_requests: +21035
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/21921

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41586] Allow to set pipe size on subprocess.Popen.

[Ruben Vorderman]

New submission from Ruben Vorderman :

Pipes block if reading from an empty pipe or when writing to a full pipe. When 
this happens the program waiting for the pipe still uses a lot of CPU cycles 
when waiting for the pipe to stop blocking.

I found this while working with xopen. A library that pipes data into an 
external gzip process. (This is more efficient than using python's gzip module, 
because the subprocess escapes the GIL, so your main algorithm can fully 
utilize one CPU core while the compression is offloaded to another).

It turns out that increasing the pipe size on Linux from the default of 64KB to 
the maximum allowed pipe size in /proc/sys/fs/max-pipe-size (1024KB) 
drastically improves performance: https://github.com/marcelm/xopen/issues/35. 
TLDR: full utilization of CPU cores, a 40%+ decrease in wall-clock time and a 
20% decrease in the number of compute seconds (indicating that 20% was wasted 
waiting on blocking pipes).

However, doing this with subprocess is quite involved as it is now.

1. You have to find out which constants to use in fcntl for setting the 
pipesize (these constants are not in python). 
2. You have to start the Popen process with routing stdout to subprocess.Pipe. 
3. You have to get my_popen_process.stdout.fileno() 
4. Use fcntl.fcntl to modify the pipe size.

It would be much easier to do `subprocess.Popen(args, pipesize=1024 *1024)` for 
example.

I am currently working on a PR implementing this. It will also make 
F_GETPIPE_SZ and F_SETPIPE_SZ available to the fcntl module.

--
components: Library (Lib)
messages: 375636
nosy: rhpvorderman
priority: normal
severity: normal
status: open
title: Allow to set pipe size on subprocess.Popen.
versions: Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com