STINNER Victor <[email protected]> added the comment:
Oh, _posixsubprocess uses /dev/fd/ on FreeBSD, but only if it's mounted file
system (expected to be fdescfs filesystem):
#if defined(__FreeBSD__) || (defined(__APPLE__) && defined(__MACH__))
# define FD_DIR "/dev/fd"
#else
# define FD_DIR "/proc/self/fd"
#endif
#if defined(__FreeBSD__)
/* When /dev/fd isn't mounted it is often a static directory populated
* with 0 1 2 or entries for 0 .. 63 on FreeBSD, NetBSD and OpenBSD.
* NetBSD and OpenBSD have a /proc fs available (though not necessarily
* mounted) and do not have fdescfs for /dev/fd. MacOS X has a devfs
* that properly supports /dev/fd.
*/
static int
_is_fdescfs_mounted_on_dev_fd(void)
{
struct stat dev_stat;
struct stat dev_fd_stat;
if (stat("/dev", &dev_stat) != 0)
return 0;
if (stat(FD_DIR, &dev_fd_stat) != 0)
return 0;
if (dev_stat.st_dev == dev_fd_stat.st_dev)
return 0; /* / == /dev == /dev/fd means it is static. #fail */
return 1;
}
#endif
On my FreeBSD 12 VM, MAXFD is around 230k which is quite large.
vstinner@freebsd$ uname -a
FreeBSD freebsd 12.0-RELEASE-p10 FreeBSD 12.0-RELEASE-p10 GENERIC amd64
vstinner@freebsd$ ./python
Python 3.9.0a0 (heads/master:4db25d5c39, Sep 9 2019, 07:52:01)
>>> import os; os.sysconf("SC_OPEN_MAX")
229284
It's easy to measure the overhead of 230k close() syscalls:
vstinner@freebsd$ python3 -m pyperf timeit -s 'import subprocess;
args=["/usr/bin/true"]' 'subprocess.Popen(args, close_fds=False).wait()'
.....................
Mean +- std dev: 1.22 ms +- 0.12 ms
vstinner@freebsd$ python3 -m pyperf timeit -s 'import subprocess;
args=["/usr/bin/true"]' 'subprocess.Popen(args, close_fds=True).wait()'
.....................
Mean +- std dev: 53.3 ms +- 1.4 ms
The overhead is 52.08 ms per Popen().wait() call (with close_fds=True).
If I mount manually the fdescfs filesystem, suddenly, subprocess is efficient
again:
vstinner@freebsd$ sudo mount -t fdescfs /dev/fd
vstinner@freebsd$ python3 -m pyperf timeit -s 'import subprocess;
args=["/usr/bin/true"]' 'subprocess.Popen(args, close_fds=True).wait()'
.....................
Mean +- std dev: 1.20 ms +- 0.06 ms
close_fds=True becomes as efficient as close_fds=False.
----------
_______________________________________
Python tracker <[email protected]>
<https://bugs.python.org/issue38061>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
