New submission from Jan-Wijbrand Kolman:
An internal library that heavily uses subprocess.Popen() started failing its
automated tests when we upgraded from Python 2.7.3 to Python 2.7.5. This
library is used in a threaded environment. After debugging the issue, I was
able to create a short Python script that demonstrates the error seen as in the
failing tests.
This is the script (called "threadedsubprocess.py"):
===
import time
import threading
import subprocess
def subprocesscall():
p = subprocess.Popen(
['ls', '-l'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
time.sleep(2) # simulate the Popen call takes some time to complete.
out, err = p.communicate()
print 'succeeding command in thread:', threading.current_thread().ident
def failingsubprocesscall():
try:
p = subprocess.Popen(
['thiscommandsurelydoesnotexist'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
except Exception as e:
print 'failing command:', e, 'in thread:',
threading.current_thread().ident
print 'main thread is:', threading.current_thread().ident
subprocesscall_thread = threading.Thread(target=subprocesscall)
subprocesscall_thread.start()
failingsubprocesscall()
subprocesscall_thread.join()
===
Note: this script does not exit with an IOError when ran from Python 2.7.3. It
does fail at least 50% of the times when ran from Python 2.7.5 (both on the
same Ubuntu 12.04 64-bit VM).
The error that is raised on Python 2.7.5 is this:
==
/opt/python/2.7.5/bin/python ./threadedsubprocess.py
main thread is: 139899583563520
failing command: [Errno 2] No such file or directory 139899583563520
Exception in thread Thread-1:
Traceback (most recent call last):
File "/opt/python/2.7.5/lib/python2.7/threading.py", line 808, in
__bootstrap_inner
self.run()
File "/opt/python/2.7.5/lib/python2.7/threading.py", line 761, in run
self.__target(*self.__args, **self.__kwargs)
File "./threadedsubprocess.py", line 13, in subprocesscall
out, err = p.communicate()
File "/opt/python/2.7.5/lib/python2.7/subprocess.py", line 806, in communicate
return self._communicate(input)
File "/opt/python/2.7.5/lib/python2.7/subprocess.py", line 1379, in
_communicate
self.stdin.close()
IOError: [Errno 9] Bad file descriptor
close failed in file object destructor:
IOError: [Errno 9] Bad file descriptor
==
When comparing the subprocess module from Python 2.7.3 to Python 2.7.5 I see
the Popen()'s __init__() call indeed now explicitly closes the stdin, stdout
and stderr file descriptors in case executing the command somehow fails. This
seems to be an intended fix applied in Python 2.7.4 to prevent leaking file
descriptors (http://hg.python.org/cpython/file/ab05e7dd2788/Misc/NEWS#l629).
The diff between Python 2.7.3 and Python 2.7.5 that seems to be relevant to
this issue is in the Popen __init__():
==
@@ -671,12 +702,33 @@
c2pread, c2pwrite,
errread, errwrite) = self._get_handles(stdin, stdout, stderr)
- self._execute_child(args, executable, preexec_fn, close_fds,
- cwd, env, universal_newlines,
- startupinfo, creationflags, shell,
- p2cread, p2cwrite,
- c2pread, c2pwrite,
- errread, errwrite)
+ try:
+ self._execute_child(args, executable, preexec_fn, close_fds,
+ cwd, env, universal_newlines,
+ startupinfo, creationflags, shell,
+ p2cread, p2cwrite,
+ c2pread, c2pwrite,
+ errread, errwrite)
+ except Exception:
+ # Preserve original exception in case os.close raises.
+ exc_type, exc_value, exc_trace = sys.exc_info()
+
+ to_close = []
+ # Only close the pipes we created.
+ if stdin == PIPE:
+ to_close.extend((p2cread, p2cwrite))
+ if stdout == PIPE:
+ to_close.extend((c2pread, c2pwrite))
+ if stderr == PIPE:
+ to_close.extend((errread, errwrite))
+
+ for fd in to_close:
+ try:
+ os.close(fd)
+ except EnvironmentError:
==
Note: I think to see a similar change in the subprocess.Popen() __init__() from
Python-3.3.0 to Python-3.3.1.
----------
components: Library (Lib)
messages: 196276
nosy: janwijbrand
priority: normal
severity: normal
status: open
title: subprocess's Popen closes stdout/stderr filedescriptors used in another
thread when Popen errors
type: behavior
versions: Python 2.7, Python 3.3
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue18851>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
