[issue21595] asyncio: Creating many subprocess generates lots of internal BlockingIOError
Roundup Robot added the comment: New changeset 46c251118799 by Victor Stinner in branch '3.4': Closes #21595: asyncio.BaseSelectorEventLoop._read_from_self() now reads all http://hg.python.org/cpython/rev/46c251118799 New changeset 513eea89b80a by Victor Stinner in branch 'default': (Merge 3.4) Closes #21595: asyncio.BaseSelectorEventLoop._read_from_self() now http://hg.python.org/cpython/rev/513eea89b80a -- nosy: +python-dev resolution: - fixed stage: - resolved status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21595 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21595] asyncio: Creating many subprocess generates lots of internal BlockingIOError
STINNER Victor added the comment: I commited asyncio_read_from_self.patch into Tulip, Python 3.4 and 3.5. If someone is interested to work on more advanced enhancement, please open a new issue. Oh by, a workaround is to limit the number of concurrent processes. Without the patch, ./python test_subprocess_error.py 5 1000 (max: 5 concurrenet processes) emits a lot of BlockingIOError: [Errno 11] Resource temporarily unavailable message. With the patch, I start getting messages with 140 concurrent processes, which is much better :-) IMO more than 100 concurrent processes is crazy, don't do that at home :-) I mean processes with a very short lifetime. The limit is the number of SIGCHLD per second, so the number of processes which end at the same second. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21595 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21595] asyncio: Creating many subprocess generates lots of internal BlockingIOError
STINNER Victor added the comment: Can someone please review asyncio_read_from_self.patch? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21595 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21595] asyncio: Creating many subprocess generates lots of internal BlockingIOError
Changes by STINNER Victor victor.stin...@gmail.com: -- components: +Asyncio title: Creating many subprocess generates lots of internal BlockingIOError - asyncio: Creating many subprocess generates lots of internal BlockingIOError ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21595 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21595] asyncio: Creating many subprocess generates lots of internal BlockingIOError
STINNER Victor added the comment: Hum, maybe I need to add a unit test for it. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21595 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21595] asyncio: Creating many subprocess generates lots of internal BlockingIOError
STINNER Victor added the comment: Can someone please review asyncio_read_from_self.patch? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21595 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21595] asyncio: Creating many subprocess generates lots of internal BlockingIOError
STINNER Victor added the comment: asyncio_read_from_self_test.patch: Unit test to check that running the loop once reads all bytes. The unit test is ugly: it calls private methods, and it is completly different on UNIX (selector) and on Windows (proactor). I would prefer to *not* add such test, and just enhance the code (apply asyncio_read_from_self.patch). -- Added file: http://bugs.python.org/file35495/asyncio_read_from_self_test.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21595 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
