New submission from Kirill Kolyshkin <kolysh...@gmail.com>:
In case close_fds=True is passed to subprocess.Popen()
or its users (subprocess.call() etc), it might spend
some considerable time closing non-opened file descriptors,
as demonstrated by the following snippet from strace:
close(3) = -1 EBADF (Bad file descriptor)
close(5) = -1 EBADF (Bad file descriptor)
close(6) = -1 EBADF (Bad file descriptor)
close(7) = -1 EBADF (Bad file descriptor)
...
close(1021) = -1 EBADF (Bad file descriptor)
close(1022) = -1 EBADF (Bad file descriptor)
close(1023) = -1 EBADF (Bad file descriptor)
This happens because the code in _close_fds() iterates from 3 up to
MAX_FDS = os.sysconf("SC_OPEN_MAX").
Now, syscalls are cheap, but SC_OPEN_MAX (also known as RLIMIT_NOFILE
or ulimit -n) can be quite high, for example:
$ docker run --rm python python3 -c \
$'import os\nprint(os.sysconf("SC_OPEN_MAX"))'
1048576
This means a million syscalls before spawning a child process, which
can result in a major delay, like 0.1s as measured on my fast and mostly
idling laptop. Here is the comparison with python3 (which does not have this
problem):
$ docker run --rm python python3 -c $'import subprocess\nimport time\ns =
time.time()\nsubprocess.check_call([\'/bin/true\'],
close_fds=True)\nprint(time.time() - s)\n'
0.0009245872497558594
$ docker run --rm python python2 -c $'import subprocess\nimport time\ns =
time.time()\nsubprocess.check_call([\'/bin/true\'],
close_fds=True)\nprint(time.time() - s)\n'
0.0964419841766
----------
components: Library (Lib)
messages: 333819
nosy: Kirill Kolyshkin
priority: normal
pull_requests: 11269
severity: normal
status: open
title: slow subprocess.Popen(..., close_fds=True)
type: performance
versions: Python 2.7
_______________________________________
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue35757>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
