New submission from Andriy Maletsky <[email protected]>:
When communicate() is called in a loop, it crashes when the child process has
already closed any piped standard stream, but still continues to be running.
How this happens:
1) the parent waits for the child events inside communicate() call
2) the child closes its side of any attached pipes long before exiting (in my
case there is some complex c++ application which had messed with its
termination)
3) communicate() receives an epoll event, tries to read/write, receives SIGPIPE
(for stdin) or EOF (for stdout), decides to close corresponding file
descriptors from its side
4) communicate() waits for the death of the child, but a timeout is fired
5) parent handles timeout exception and calls communicate() again
6) an exception is raised when communicate() tries to register closed file in
epoll
I think there may be a simple solution: before registering file descriptors in
epoll, we may check whether any of them is already closed, and don't register
it in that case.
Here is a simple reproducible example, ran on Linux 4.15.0-1021-aws x86_64:
import subprocess
child = subprocess.Popen(
['/usr/local/bin/python3.7', '-c', 'import os, time; os.close(1),
time.sleep(30)'],
stdout=subprocess.PIPE,
)
while True:
try:
child.communicate(timeout=3)
break
except subprocess.TimeoutExpired:
# do something useful here
pass
Here is a stacktrace:
Traceback (most recent call last):
File "test.py", line 10, in <module>
child.communicate(timeout=3)
File "/usr/local/lib/python3.7/subprocess.py", line 933, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/usr/local/lib/python3.7/subprocess.py", line 1666, in _communicate
selector.register(self.stdout, selectors.EVENT_READ)
File "/usr/local/lib/python3.7/selectors.py", line 352, in register
key = super().register(fileobj, events, data)
File "/usr/local/lib/python3.7/selectors.py", line 238, in register
key = SelectorKey(fileobj, self._fileobj_lookup(fileobj), events, data)
File "/usr/local/lib/python3.7/selectors.py", line 225, in _fileobj_lookup
return _fileobj_to_fd(fileobj)
File "/usr/local/lib/python3.7/selectors.py", line 40, in _fileobj_to_fd
"{!r}".format(fileobj)) from None
ValueError: Invalid file object: <_io.BufferedReader name=3>
----------
messages: 329412
nosy: and800
priority: normal
severity: normal
status: open
title: Popen.communicate() breaks when child closes its side of pipe but not
exits
type: crash
versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8
_______________________________________
Python tracker <[email protected]>
<https://bugs.python.org/issue35182>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
