[issue37790] subprocess.Popen() is extremely slow (with close_fds=True which is the default) on Illumos

[STINNER Victor]

STINNER Victor  added the comment:

I merged FreeBSD changes:

New changeset 162c567d164b5742c0d1f3f8bd8c8bab9c117cd0 by Victor Stinner in 
branch 'master':
bpo-38061: os.closerange() uses closefrom() on FreeBSD (GH-19696)
https://github.com/python/cpython/commit/162c567d164b5742c0d1f3f8bd8c8bab9c117cd0

New changeset e6f8abd500751a834b6fff4f107ecbd29f2184fe by Victor Stinner in 
branch 'master':
bpo-38061: subprocess uses closefrom() on FreeBSD (GH-19697)
https://github.com/python/cpython/commit/e6f8abd500751a834b6fff4f107ecbd29f2184fe

Free feel to propose a more generic way to detect when closefrom() function is 
available ;-) Or just a patch to add "|| defined(sun)".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37790] subprocess.Popen() is extremely slow (with close_fds=True which is the default) on Illumos

[Antoine Pitrou]

Antoine Pitrou  added the comment:

I think we would accept a PR since it would probably be trivial, but someone 
has to submit it :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37790] subprocess.Popen() is extremely slow (with close_fds=True which is the default) on Illumos

[John Levon]

John Levon  added the comment:

closefrom() is on both Solaris and illumos too - and might even have originated 
there as an API -  so if that's the issue, it should be trivially fixable

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37790] subprocess.Popen() is extremely slow (with close_fds=True which is the default) on Illumos

[STINNER Victor]

STINNER Victor  added the comment:

> We've moved illumos-gate wsdiff python tool from Python 2 to Python 3.

I guess that you're talking about 
https://github.com/illumos/illumos-gate/blob/master/usr/src/tools/scripts/wsdiff.py

This project is part of illumos: "An open-source Unix operating system". It's a 
fork of Solaris.

Python does not officially support Solaris not Illumos. If someone provides a 
patch, we can review it. But I'm not aware of anyone working actively on 
supporting Illumos.

In the meanwhile, there are two workarounds:

* Use subprocess.Popen(close_fds=False): it should be safe since PEP 446 has 
been implemented in Python 3.4... except for C extension modules and C 
libraries which don't implement it.
* Reduce the maximum number of file descriptors.

On Linux and FreeBSD (if fdescfs kernel module is loaded and mounted), Python 3 
iterates on /proc/self/fd/. On FreeBSD, the closefrom() syscall can be used: PR 
19697. But I'm not aware of any more efficient solution for Illumos.

Since the latest comment of the original poster was last August, I close the 
issue.

--
resolution:  -> out of date
stage:  -> resolved
status: open -> closed
title: subprocess.Popen() is extremely slow -> subprocess.Popen() is extremely 
slow (with close_fds=True which is the default) on Illumos

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37790] subprocess.Popen() is extremely slow

[STINNER Victor]

STINNER Victor  added the comment:

I create bpo-38061 "FreeBSD: Optimize subprocess.Popen(close_fds=True) using 
closefrom()".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37790] subprocess.Popen() is extremely slow

[Nick Henderson]

Nick Henderson  added the comment:

We have recently bumped into a similar problem. Using FreeBSD, subprocess calls 
were taking more than 10 times the usual time to execute after migrating to 
python3.6. After some digging, the default for 'close_fds' was changed to 
'True'. On linux, this actually made things faster, but for unix, much slower. 
Passing 'close_fds=False' solved this for us.

--
nosy: +nickhendo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37790] subprocess.Popen() is extremely slow

[STINNER Victor]

STINNER Victor  added the comment:

This issue lacks a lot of information:

* What is your operating system (name and version)? On Linux, what is your 
Linux kernel version?
* Which Python version did you try?
* Which command are you running?
* Do you use a shell?
* Do you use bytes (default) or Unicode (universal_newlines=True or text=True)?
* Can you provide a minimum reproducer? I don't know how to use msg349894: cmd 
is not defined.

Attached 1.py uses 2 commands, appararently both use a shell:

* "find /usr/bin -type f 2>/dev/null"
* "objdump '%s'"

"objdump '%s'" is unsafe and can lead to shell injection: try to avoid the 
usage of a shell. Use subprocess.Popen directly, or an helper which doesn't use 
shell=True.

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37790] subprocess.Popen() is extremely slow

[Antoine Pitrou]

Antoine Pitrou  added the comment:

I don't seen any significant difference here (Ubuntu 18.04):

$ time python2 sp.py 
10.3433089256

real0m10,362s
user0m6,565s
sys 0m4,372s

$ time python3 sp.py 
11.746907234191895

real0m11,781s
user0m7,356s
sys 0m5,239s

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37790] subprocess.Popen() is extremely slow

[Alexander Pyhalov]

Alexander Pyhalov  added the comment:

Even if I use 
import subprocess
process = subprocess.Popen(cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=True,
universal_newlines=True)
return process.stdout.read()
difference in times are the same.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37790] subprocess.Popen() is extremely slow

[Alexander Pyhalov]

Alexander Pyhalov  added the comment:

I couldn't reproduce entire test, as wsdiff script is rather large, but here's 
the simplified version. If Popen is used more often, difference is much more 
significant.

# Using workaround
$ python3.5 ~/tmp/1.py 1 
10.780487537384033
# Without workaround
$ python3.5 ~/tmp/1.py
13.347045660018921
# Using python2.7
$ python2.7 ~/tmp/1.py
9.83385586739

--
Added file: https://bugs.python.org/file48549/1.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37790] subprocess.Popen() is extremely slow

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Can you post a simple script showcasing the performance degradation?

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37790] subprocess.Popen() is extremely slow

[John Levon]

Change by John Levon :


--
nosy: +movement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37790] subprocess.Popen() is extremely slow

[Alexander Pyhalov]

Alexander Pyhalov  added the comment:

I've tried to rewrite subporcess.getstatusoutput() usage with 
subprocess.Popen() and switch to shell=False, it didn't help, so I doubti it 
getstatusoutput() overhead, it's Popen() issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37790] subprocess.Popen() is extremely slow

[Josh Rosenberg]

Josh Rosenberg  added the comment:

Have you tried switching to using Popen itself (or run, which keeps it to one 
layer of convenience wrapping)?

subprocess.getstatusoutput is three layers of wrapping (it calls check_output, 
which in turn calls run, which in turn calls Popen), and (unlike Python 2) has 
to decode all the output. run would avoid two layers of wrapping, can 
optionally receive the raw bytes instead of decoding to str, and avoids needing 
to wrap the whole thing in a shell (which system, older popen, and 
getstatusoutput all do).

Beyond that, it looks like when 3.8 releases, Popen should get *much* faster if 
the call meets certain conditions, see 
https://docs.python.org/3.8/whatsnew/3.8.html#optimizations for details. If you 
can make your use cases conform to those requirements (e.g. by using 
shutil.which to determine the absolute paths to your utilities instead of 
relying on PATH lookup), the speed up should eliminate (or more than eliminate) 
the performance regression you're seeing (per #35537, on macOS, which got the 
least benefit, it was still a 1.7x improvement; on other OSes, the multiplier 
hits 61x or 106x).

--
nosy: +josh.r

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37790] subprocess.Popen() is extremely slow

[Alexander Pyhalov]

New submission from Alexander Pyhalov :

We've moved illumos-gate wsdiff python tool from Python 2 to Python 3.
The tool does the following - for each file from old and new proto area 
compares file attributes to find differences in binary otput (spawning elfdump, 
dump and other utilities). 
Performance has degraded in two times when commands.getstatusoutput() was 
replaced by subprocess.getstatusoutput().
os.popen() now is Popen() wrapper, so it also has poor performance.
Even naive popen() implementation using os.system() and os.mkstemp() behaved 
much more efficiently (about two times faster e.g. 20 minuts vs 40 minutes for 
single tool pass).

--
components: Library (Lib)
messages: 349197
nosy: Alexander.Pyhalov
priority: normal
severity: normal
status: open
title: subprocess.Popen() is extremely slow
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com