[issue35182] Popen.communicate() breaks when child closes its side of pipe but not exits
Change by Andriy Maletsky : -- pull_requests: +16540 pull_request: https://github.com/python/cpython/pull/17023 ___ Python tracker <https://bugs.python.org/issue35182> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35182] Popen.communicate() breaks when child closes its side of pipe but not exits
Change by Andriy Maletsky : -- keywords: +patch pull_requests: +16538 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/17020 ___ Python tracker <https://bugs.python.org/issue35182> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35182] Popen.communicate() breaks when child closes its side of pipe but not exits
Andriy Maletsky added the comment: @josh.r but you’re correct regarding cached data that isn’t sent on subsequent communicate() calls. If the child consumes the input too slowly, and timeout occurs before sending all input, the remaining part will be lost. Maybe it is not a bug, but it’s quite a confusing behavior, and I think it should be mentioned in the doc. -- ___ Python tracker <https://bugs.python.org/issue35182> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36434] Zipfile breaks if signalled during write()
New submission from Andriy Maletsky :
Consider a simple write to a zip file:
import zipfile
with zipfile.ZipFile('/workdir/archive.zip', 'w',
compression=zipfile.ZIP_DEFLATED) as zip_archive:
zip_archive.write('/workdir/data.csv', arcname='data.csv')
print('exiting from context manager...')
If a signal handler is fired and raises an exception during certain points of
write() execution, such an error occurs (py 3.7.2):
Traceback (most recent call last):
File "zipissue.py", line 4, in
zip_archive.write('/workdir/data.csv', arcname='data.csv')
File "/usr/local/lib/python3.7/zipfile.py", line 1744, in write
shutil.copyfileobj(src, dest, 1024*8)
File "/usr/local/lib/python3.7/zipfile.py", line 1107, in close
buf = self._compressor.flush()
KeyboardInterrupt
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "zipissue.py", line 5, in
print('exiting from context manager...')
File "/usr/local/lib/python3.7/zipfile.py", line 1265, in __exit__
self.close()
File "/usr/local/lib/python3.7/zipfile.py", line 1798, in close
raise ValueError("Can't close the ZIP file while there is "
ValueError: Can't close the ZIP file while there is an open writing handle on
it. Close the writing handle before closing the zip.
Exception ignored in:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/zipfile.py", line 1789, in __del__
File "/usr/local/lib/python3.7/zipfile.py", line 1798, in close
ValueError: Can't close the ZIP file while there is an open writing handle on
it. Close the writing handle before closing the zip.
Before any write the `ZipFile._writing` flag is set, and that flag is cleared
at `_ZipWriteFile.close()`. But if signalled inside `_ZipWriteFile.close()` we
are moving to a broken state: we don't write anything anymore, but
`ZipFile._writing` is still set. Therefore we cannot clearly close ZipFile. As
ZipFile contextmanager swallows KeyboardInterrupt and produces an exception of
`Exception` type, this leads to the impossibility of proper program shutdown.
I believe that by simply moving `ZipFile._writing = False` in
`_ZipWriteFile.close()` to some finally block, this issue will be solved safely.
--
components: Library (Lib)
messages: 338863
nosy: and800
priority: normal
severity: normal
status: open
title: Zipfile breaks if signalled during write()
type: behavior
versions: Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9
___
Python tracker
<https://bugs.python.org/issue36434>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35182] Popen.communicate() breaks when child closes its side of pipe but not exits
Change by Andriy Maletsky : -- components: +Library (Lib) ___ Python tracker <https://bugs.python.org/issue35182> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35182] Popen.communicate() breaks when child closes its side of pipe but not exits
Change by Andriy Maletsky : -- type: crash -> behavior ___ Python tracker <https://bugs.python.org/issue35182> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35182] Popen.communicate() breaks when child closes its side of pipe but not exits
New submission from Andriy Maletsky :
When communicate() is called in a loop, it crashes when the child process has
already closed any piped standard stream, but still continues to be running.
How this happens:
1) the parent waits for the child events inside communicate() call
2) the child closes its side of any attached pipes long before exiting (in my
case there is some complex c++ application which had messed with its
termination)
3) communicate() receives an epoll event, tries to read/write, receives SIGPIPE
(for stdin) or EOF (for stdout), decides to close corresponding file
descriptors from its side
4) communicate() waits for the death of the child, but a timeout is fired
5) parent handles timeout exception and calls communicate() again
6) an exception is raised when communicate() tries to register closed file in
epoll
I think there may be a simple solution: before registering file descriptors in
epoll, we may check whether any of them is already closed, and don't register
it in that case.
Here is a simple reproducible example, ran on Linux 4.15.0-1021-aws x86_64:
import subprocess
child = subprocess.Popen(
['/usr/local/bin/python3.7', '-c', 'import os, time; os.close(1),
time.sleep(30)'],
stdout=subprocess.PIPE,
)
while True:
try:
child.communicate(timeout=3)
break
except subprocess.TimeoutExpired:
# do something useful here
pass
Here is a stacktrace:
Traceback (most recent call last):
File "test.py", line 10, in
child.communicate(timeout=3)
File "/usr/local/lib/python3.7/subprocess.py", line 933, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/usr/local/lib/python3.7/subprocess.py", line 1666, in _communicate
selector.register(self.stdout, selectors.EVENT_READ)
File "/usr/local/lib/python3.7/selectors.py", line 352, in register
key = super().register(fileobj, events, data)
File "/usr/local/lib/python3.7/selectors.py", line 238, in register
key = SelectorKey(fileobj, self._fileobj_lookup(fileobj), events, data)
File "/usr/local/lib/python3.7/selectors.py", line 225, in _fileobj_lookup
return _fileobj_to_fd(fileobj)
File "/usr/local/lib/python3.7/selectors.py", line 40, in _fileobj_to_fd
"{!r}".format(fileobj)) from None
ValueError: Invalid file object: <_io.BufferedReader name=3>
--
messages: 329412
nosy: and800
priority: normal
severity: normal
status: open
title: Popen.communicate() breaks when child closes its side of pipe but not
exits
type: crash
versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8
___
Python tracker
<https://bugs.python.org/issue35182>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32102] Add "capture_output=True" option to subprocess.run
Change by Andriy Maletsky : -- pull_requests: +8204 ___ Python tracker <https://bugs.python.org/issue32102> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34074] Asyncio breaks coroutine finalization process
Andriy Maletsky added the comment: I meant that my example is a complete python script, and after `loop.run_forever()` the interpreter stops. So I expected every python object to be destructed on interpreter shutdown, but coro's `close()` is not called. To make example more practical, let's change `loop.stop()` to `raise KeyboardInterrupt()`. Program stops without calling `close()` -- ___ Python tracker <https://bugs.python.org/issue34074> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34074] Asyncio breaks coroutine finalization process
New submission from Andriy Maletsky :
Source: https://stackoverflow.com/q/51245011/6275324
Asyncio somehow breaks coroutine finalization. I believe there may be a bug in
C implementation (_asyncio) of tasks or futures. Reproducible within version
3.7.0@python:3.7 docker container.
Consider this example (except and finally blocks will never execute):
import asyncio
async def work():
try:
print('started working')
await asyncio.sleep(3600)
except BaseException as e:
print('caught ' + str(type(e)))
finally:
print('finalization completed')
async def stopper():
await asyncio.sleep(5)
loop.stop()
loop = asyncio.get_event_loop()
loop.create_task(work())
loop.create_task(stopper())
loop.run_forever()
And there is asyncio-free piece of code, which works properly, catching
GeneratorExit, thrown by coro destructor:
import asyncio
async def work():
try:
print('started working')
await asyncio.sleep(3600)
except BaseException as e:
print('caught ' + str(type(e)))
finally:
print('finalization completed')
coro = work()
coro.send(None)
del coro
--
components: asyncio
messages: 321321
nosy: and800, asvetlov, yselivanov
priority: normal
severity: normal
status: open
title: Asyncio breaks coroutine finalization process
type: behavior
versions: Python 3.6, Python 3.7
___
Python tracker
<https://bugs.python.org/issue34074>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
