Марк Коренберг added the comment:
Calling getdents()/readdir64() repeatedly while closing descriptors provides
unexpected behaviour. Reading directory while it modified is not safe by
default. For example: http://en.it-usenet.org/thread/18514/15719/.
So, we should re-open directory if we
STINNER Victor added the comment:
Calling getdents()/readdir64() repeatedly while closing descriptors provides
unexpected behaviour.
Please open a new issue.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8052
Марк Коренберг added the comment:
Also, it is not said in manual if getdents() may be interrupted by signal.
Assuming current code, error is not checked, so some (or all) descriptors will
be skipped in case of error.
--
___
Python tracker
STINNER Victor added the comment:
Also, it is not said in manual if getdents() may be interrupted by signal.
Assuming current code, error is not checked, so some (or all) descriptors will
be skipped in case of error.
This issue is closed, please open a new issue to discuss that.
--
Roundup Robot devn...@psf.upfronthosting.co.za added the comment:
New changeset 61aa484a3e54 by Gregory P. Smith in branch '3.2':
Fixes issue #8052: The posix subprocess module's close_fds behavior was
http://hg.python.org/cpython/rev/61aa484a3e54
New changeset 8879874d66a2 by Gregory P. Smith
Roundup Robot devn...@psf.upfronthosting.co.za added the comment:
New changeset 780992c9afea by Gregory P. Smith in branch '3.2':
Add a Misc/NEWS entry for issue 8052.
http://hg.python.org/cpython/rev/780992c9afea
New changeset 1f0a01dc723c by Gregory P. Smith in branch 'default':
A Misc/NEWS
Roundup Robot devn...@psf.upfronthosting.co.za added the comment:
New changeset d0acd8169c2a by Gregory P. Smith in branch '3.2':
Bugfix for issue #8052 fix on *BSD variants.
http://hg.python.org/cpython/rev/d0acd8169c2a
New changeset 5be3dadd2eef by Gregory P. Smith in branch '3.2':
Another
Roundup Robot devn...@psf.upfronthosting.co.za added the comment:
New changeset 754c2eb0a92c by Gregory P. Smith in branch '3.2':
Fix FreeBSD, NetBSD and OpenBSD behavior of the issue #8052 fix.
http://hg.python.org/cpython/rev/754c2eb0a92c
New changeset 7d4658a8de96 by Gregory P. Smith in
Gregory P. Smith g...@krypto.org added the comment:
For FreeBSD, Python 3.2 and 3.3 now check to see if /dev/fd is valid. Be sure
and mount -t fdescfs none /dev/fd on FreeBSD if you want faster subprocess
launching. Run a FreeBSD buildbot? Please do it!
For Python 3.1 the fix for #13788
Gregory P. Smith g...@krypto.org added the comment:
attaching a patch that implements this safely.
--
keywords: +patch
Added file:
http://bugs.python.org/file24255/subprocess-close-open-fds-gps01.diff
___
Python tracker rep...@bugs.python.org
Gregory P. Smith g...@krypto.org added the comment:
attaching a better formatted one for review.
--
Added file:
http://bugs.python.org/file24256/subprocess-close-open-fds-gps02.diff
___
Python tracker rep...@bugs.python.org
Ferringb ferri...@gmail.com added the comment:
In #13788, I've uploaded a patch modifying closerange along the same lines as
this discussion; someone w/ appropriate rights should set dependencies as
needed.
Either way, here's a question: does anyone actually know of a unix that does
procfs,
Charles-François Natali neolo...@free.fr added the comment:
Either way, here's a question: does anyone actually know of a unix that does
procfs, and has a daft opendir implementation as described below? Aka, are
we actually worrying about something relevant, or just hypotheticals?
I think
Ferringb ferri...@gmail.com added the comment:
The only question is: do other Unix also have /proc/pid/fd? e.g.
FreeBSD, OpenBSD. That's especially important because FreeBSD can have
a huge RLIMIT_NOFILE by default.
Unless the OS gives some way to optimize the process (whether inferring from
Ross Lagerwall rosslagerw...@gmail.com added the comment:
FreeBSD has a /dev/fd as well as a procfs (deprecated AFAIK).
However, both may not be mounted so a patch would *need* to at least fallback
to the current functionality.
--
___
Python tracker
STINNER Victor victor.stin...@haypocalc.com added the comment:
test_subprocess.test_leaking_fds_on_error takes more than 5 minutes on x86
FreeBSD 7.2 build slave! This tests creates 1,024 subprocesses, and subprocess
has to close 655,000 file descriptors just to create on child process
Charles-Francois Natali neolo...@free.fr added the comment:
I wonder whether the Java people are simply unaware of the potential problem?
Or perhaps they have checked the Linux and Solaris implementations of
readdir()
and confirmed that it is in fact safe on those platforms. Even if this is
Nadeem Vawda nadeem.va...@gmail.com added the comment:
For what its worth, an strace of Java's Process class appears to cheat by
opening /proc/self/fd inbetween fork exec.
Looking at the OpenJDK source confirms this:
STINNER Victor victor.stin...@haypocalc.com added the comment:
I like the approach suggested by Gregory in issue11284 (msg129912) - if we are
sure the program is not multithreaded, get the list of fds before forking, and
use that; otherwise, do it the slow way.
A signal handler can also
Ross Lagerwall rosslagerw...@gmail.com added the comment:
See #11284 (a duplicate) for more discussion about this issue.
--
nosy: +haypo, loewis, neologix, s7v7nislands
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8052
STINNER Victor victor.stin...@haypocalc.com added the comment:
Can we use FD_CLOEXEC to archive this goal? Example: open all files with
FD_CLOEXEC set and don't close explicitly files on fork. preexec_fn will get
access to the files, but the problem is exec(), not preexec_fn.
I suppose that
Charles-Francois Natali neolo...@free.fr added the comment:
If you're suggesting to set FDs CLOEXEC by default, I think it's neither
possible nor reasonable:
- you have to take into account not only files, but also pipes, sockets, etc
- there's no portable way to e.g. open a file and set it
Charles-Francois Natali neolo...@free.fr added the comment:
Ooops, it's of course not going to break code containing accept + fork or pipe
+ fork, you obviously also need an execve ;-)
But the point is that you can't change the semantics of FDs being inheritable
across an execve (think about
Ross Lagerwall rosslagerw...@gmail.com added the comment:
I don't think setting the cloexec flag is a viable solution, especially since
fds can be opened in custom c modules.
For what its worth, an strace of Java's Process class appears to cheat by
opening /proc/self/fd inbetween fork exec.
Martin v. Löwis mar...@v.loewis.de added the comment:
Can we use FD_CLOEXEC to archive this goal?
I think we should use FD_CLOEXEC in all places where it's reasonable.
As others have pointed out, we shouldn't set FD_CLOEXEC for file
descriptors where the application hasn't explicitly
Ross Lagerwall rosslagerw...@gmail.com added the comment:
Does this mean that it's better to call the close() syscall 1000 or 1000
times rather than listing the open fds closing say a handful?
On Linux, the listdir function results in an open() syscall, 2 or so getdents()
calls and a
Nadeem Vawda nadeem.va...@gmail.com added the comment:
According to POSIX [1], if a multi-threaded program calls fork(), the child
process may only use async-signal-safe system calls between fork() and exec*().
readdir() is not required to be async-safe [2], so reading /proc/self/fds in
the
Antoine Pitrou pit...@free.fr added the comment:
Of course, procfs isn't standard in any case; would it be necessary to
have a fallback for systems without it? Or do all *nix systems that we
care about provide it?
/proc/self doesn't exist under OpenSolaris. You have to use /proc/pid.
And I
Antoine Pitrou pit...@free.fr added the comment:
Is it still a problem now that there's a C path?
Furthermore, how do you plan to get a list of open fds?
--
nosy: +pitrou
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8052
Gregory P. Smith g...@krypto.org added the comment:
its still a problem, even the C path has to call close() a million times in
that case. thats a huge number of wasted syscalls. fixing this is blocking on
a good way to get the list of open fds.
I have seen other subprocess code do it using
Changes by Mark Lawrence breamore...@yahoo.co.uk:
--
stage: - needs patch
versions: +Python 3.1
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8052
___
31 matches
Mail list logo
