[issue23095] [Windows] asyncio: race condition when cancelling a _WaitHandleFuture
STINNER Victor added the comment: Pseudo-code: --- handle = ... fut = proactor.wait_for_handle(handle) fut.cancel() loop.close() --- Windows functions called in this code: - A: call RegisterWaitForSingleObject() on an handle - A: call UnregisterWaitEx() with an event - B: call RegisterWaitForSingleObject() on this event - B: the wait on this event completes: the completion is signaled with PostToQueueCallback() (our internal callback passed to RegisterWaitForSingleObject()) - A: _unregister_wait_cb() is called on the future A - (but the wait on the object A is never signaled by PostToQueueCallback()) In short, UnregisterWaitEx() is called before the callback passed to RegisterWaitForSingleObject() is called. In this case, the callback is never called, and so the completion of the wait is never signaled. Internally, RegisterWaitForSingleObject() is implemented as a pool of threads calling WaitForMultipleObjects(). The default limit for the pool is 500 threads. WaitForMultipleObjects() is limited to 64 objects. Each thread uses a timer and computes a timeout. The timer is awaken to cancel a wait. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23095 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23095] [Windows] asyncio: race condition when cancelling a _WaitHandleFuture
Roundup Robot added the comment: New changeset 36e80c6599aa by Victor Stinner in branch '3.4': Issue #23095, asyncio: Fix _WaitHandleFuture.cancel() https://hg.python.org/cpython/rev/36e80c6599aa -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23095 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23095] [Windows] asyncio: race condition when cancelling a _WaitHandleFuture
STINNER Victor added the comment: I fixed many other issues related to the IocpProactor. This time it should be ok. I ran the Tulip and Trollius test suite a lot of times, in release and debug mode. I didn't see any warning nor hang. I also ran test_asyncio of Python as well. I close this issue again. -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23095 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23095] [Windows] asyncio: race condition when cancelling a _WaitHandleFuture
STINNER Victor added the comment: test_asyncio hangs on AMD64 Windows7 SP1 3.x buildbot: http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/5562 The most significant change of this build is the changeset d3804307cce4: IocpProactor.close() must not cancel pending _WaitCancelFuture futures. [391/391] test_asyncio Timeout (1:00:00)! Current thread 0x13f8 (most recent call first): File C:\buildbot.python.org\3.x.kloth-win64\build\lib\asyncio\windows_events.py, line 617 in _unregister File C:\buildbot.python.org\3.x.kloth-win64\build\lib\asyncio\windows_events.py, line 193 in _unregister_wait_cb File C:\buildbot.python.org\3.x.kloth-win64\build\lib\asyncio\windows_events.py, line 209 in _unregister_wait File C:\buildbot.python.org\3.x.kloth-win64\build\lib\asyncio\windows_events.py, line 152 in set_result File C:\buildbot.python.org\3.x.kloth-win64\build\lib\asyncio\windows_events.py, line 671 in _poll File C:\buildbot.python.org\3.x.kloth-win64\build\lib\asyncio\windows_events.py, line 386 in select File C:\buildbot.python.org\3.x.kloth-win64\build\lib\asyncio\base_events.py, line 1081 in _run_once File C:\buildbot.python.org\3.x.kloth-win64\build\lib\asyncio\base_events.py, line 258 in run_forever File C:\buildbot.python.org\3.x.kloth-win64\build\lib\asyncio\base_events.py, line 286 in run_until_complete File C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_asyncio\test_events.py, line 1695 in test_subprocess_stderr_redirect_to_stdout File C:\buildbot.python.org\3.x.kloth-win64\build\lib\unittest\case.py, line 577 in run File C:\buildbot.python.org\3.x.kloth-win64\build\lib\unittest\case.py, line 625 in __call__ File C:\buildbot.python.org\3.x.kloth-win64\build\lib\unittest\suite.py, line 125 in run File C:\buildbot.python.org\3.x.kloth-win64\build\lib\unittest\suite.py, line 87 in __call__ File C:\buildbot.python.org\3.x.kloth-win64\build\lib\unittest\suite.py, line 125 in run File C:\buildbot.python.org\3.x.kloth-win64\build\lib\unittest\suite.py, line 87 in __call__ File C:\buildbot.python.org\3.x.kloth-win64\build\lib\unittest\suite.py, line 125 in run File C:\buildbot.python.org\3.x.kloth-win64\build\lib\unittest\suite.py, line 87 in __call__ File C:\buildbot.python.org\3.x.kloth-win64\build\lib\unittest\suite.py, line 125 in run File C:\buildbot.python.org\3.x.kloth-win64\build\lib\unittest\suite.py, line 87 in __call__ File C:\buildbot.python.org\3.x.kloth-win64\build\lib\unittest\suite.py, line 125 in run File C:\buildbot.python.org\3.x.kloth-win64\build\lib\unittest\suite.py, line 87 in __call__ File C:\buildbot.python.org\3.x.kloth-win64\build\lib\unittest\runner.py, line 168 in run File C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\support\__init__.py, line 1770 in _run_suite File C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\support\__init__.py, line 1804 in run_unittest File C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\regrtest.py, line 1283 in test_runner File C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\regrtest.py, line 1284 in runtest_inner File C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\regrtest.py, line 967 in runtest File C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\regrtest.py, line 532 in main File C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\regrtest.py, line 1568 in main_in_temp_cwd File C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\regrtest.py, line 1593 in module File C:\buildbot.python.org\3.x.kloth-win64\build\lib\runpy.py, line 85 in _run_code File C:\buildbot.python.org\3.x.kloth-win64\build\lib\runpy.py, line 170 in _run_module_as_main -- resolution: fixed - status: closed - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23095 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23095] [Windows] asyncio: race condition when cancelling a _WaitHandleFuture
STINNER Victor added the comment: A different hang on AMD64 Windows7 SP1 3.4/ buildbot: http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.4/builds/805/steps/test/logs/stdio This build is also related to the changeset d3804307cce44f7f02e38166daf6d8227aa45021: IocpProactor.close() must not cancel pending _WaitCancelFuture futures. IMO it's not a new bug, it's just that it was not seen before. I may be related to the issue #23293. [390/390/1] test_asyncio Timeout (1:00:00)! Current thread 0x05c8 (most recent call first): File C:\buildbot.python.org\3.4.kloth-win64\build\lib\asyncio\windows_events.py, line 620 in _unregister File C:\buildbot.python.org\3.4.kloth-win64\build\lib\asyncio\windows_events.py, line 196 in _unregister_wait_cb File C:\buildbot.python.org\3.4.kloth-win64\build\lib\asyncio\windows_events.py, line 212 in _unregister_wait File C:\buildbot.python.org\3.4.kloth-win64\build\lib\asyncio\windows_events.py, line 152 in set_result File C:\buildbot.python.org\3.4.kloth-win64\build\lib\asyncio\windows_events.py, line 674 in _poll File C:\buildbot.python.org\3.4.kloth-win64\build\lib\asyncio\windows_events.py, line 389 in select File C:\buildbot.python.org\3.4.kloth-win64\build\lib\asyncio\base_events.py, line 1081 in _run_once File C:\buildbot.python.org\3.4.kloth-win64\build\lib\asyncio\base_events.py, line 258 in run_forever File C:\buildbot.python.org\3.4.kloth-win64\build\lib\asyncio\base_events.py, line 286 in run_until_complete File C:\buildbot.python.org\3.4.kloth-win64\build\lib\test\test_asyncio\test_subprocess.py, line 211 in test_pause_reading File C:\buildbot.python.org\3.4.kloth-win64\build\lib\unittest\case.py, line 577 in run File C:\buildbot.python.org\3.4.kloth-win64\build\lib\unittest\case.py, line 625 in __call__ File C:\buildbot.python.org\3.4.kloth-win64\build\lib\unittest\suite.py, line 125 in run File C:\buildbot.python.org\3.4.kloth-win64\build\lib\unittest\suite.py, line 87 in __call__ File C:\buildbot.python.org\3.4.kloth-win64\build\lib\unittest\suite.py, line 125 in run File C:\buildbot.python.org\3.4.kloth-win64\build\lib\unittest\suite.py, line 87 in __call__ File C:\buildbot.python.org\3.4.kloth-win64\build\lib\unittest\suite.py, line 125 in run File C:\buildbot.python.org\3.4.kloth-win64\build\lib\unittest\suite.py, line 87 in __call__ File C:\buildbot.python.org\3.4.kloth-win64\build\lib\unittest\suite.py, line 125 in run File C:\buildbot.python.org\3.4.kloth-win64\build\lib\unittest\suite.py, line 87 in __call__ File C:\buildbot.python.org\3.4.kloth-win64\build\lib\unittest\runner.py, line 168 in run File C:\buildbot.python.org\3.4.kloth-win64\build\lib\test\support\__init__.py, line 1769 in _run_suite File C:\buildbot.python.org\3.4.kloth-win64\build\lib\test\support\__init__.py, line 1803 in run_unittest File C:\buildbot.python.org\3.4.kloth-win64\build\lib\test\regrtest.py, line 1279 in test_runner File C:\buildbot.python.org\3.4.kloth-win64\build\lib\test\regrtest.py, line 1280 in runtest_inner File C:\buildbot.python.org\3.4.kloth-win64\build\lib\test\regrtest.py, line 967 in runtest File C:\buildbot.python.org\3.4.kloth-win64\build\lib\test\regrtest.py, line 532 in main File C:\buildbot.python.org\3.4.kloth-win64\build\lib\test\regrtest.py, line 1564 in main_in_temp_cwd File C:\buildbot.python.org\3.4.kloth-win64\build\lib\test\regrtest.py, line 1589 in module File C:\buildbot.python.org\3.4.kloth-win64\build\lib\runpy.py, line 85 in _run_code File C:\buildbot.python.org\3.4.kloth-win64\build\lib\runpy.py, li -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23095 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23095] [Windows] asyncio: race condition when cancelling a _WaitHandleFuture
STINNER Victor added the comment: Running runtests.py test_cancel_post_init test_shell and runtests.py test_wait_for_handle test_wait_for_handle_cancel of Tulip show a different behaviour of _WaitHandleFuture. In one case, the cancelled wait is still signaled, on other case, it's never signaled. Currently, a cancelled _WaitHandleFuture always unregisters the overlapped operation, which causes unexpected event or may lead tests to hang. Never unregisters causes a different issue. Unregistering the overlapped indirectly delete it in memory, which is bad if the completion is still signaled. A workaround is to keep a reference to the unregistered overlopped, but it's an ugly workaround. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23095 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23095] [Windows] asyncio: race condition when cancelling a _WaitHandleFuture
STINNER Victor added the comment: To wait for the exit of the subprocess, we use RegisterWaitForSingleObject(). To cancel this wait, we can use UnregisterWait() which returns immediatly. Problem: UnregisterWait() doesn't tell us if the wait was cancelled or not, the cancellation is asynchronous. Second problem: the wait may have been signaled to the IOCP... or not. The wait may be signaled after the call to UnregisterWait(), since the cancellation is asynchronous (I'm not sure of that, but it doesn't change everything). This can be explained by the implementation: RegisterWaitForSingleObject() is implemented with a pool of threads. Windows XP introduced UnregiterWaitEx() which can be used to be notified when the wait has been cancelled. Cool. But the notification requires an Event object. And how can we asynchronously wait for this Event? Using RegisterWaitForSingleObject()! Wait, what? We were cancelling another RegisterWaitForSingleObject(). To be fully asynchronous (no performance impact), cancelling a RegisterWaitForSingleObject() wait requires a new Event object and call RegisterWaitForSingleObject() on it. -- In Python, we must ensure that the Overlapped object used by RegisterWaitForSingleObject() is kept alive until the wait is signalled, or until we are sure that the wait was cancelled. Otherwise, the program may crash. To keep the Overlapped object alive, we keep indirectly in a _WaitHandleFuture object, and this future is registered in IocpProactor._cache. I'm working on a change to use UnregiterWaitEx(). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23095 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23095] [Windows] asyncio: race condition when cancelling a _WaitHandleFuture
Roundup Robot added the comment: New changeset fb8a093db8b1 by Victor Stinner in branch '3.4': Issue #23095, asyncio: Rewrite _WaitHandleFuture.cancel() https://hg.python.org/cpython/rev/fb8a093db8b1 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23095 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23095] [Windows] asyncio: race condition when cancelling a _WaitHandleFuture
Roundup Robot added the comment: New changeset d3804307cce4 by Victor Stinner in branch '3.4': Issue #23095, asyncio: IocpProactor.close() must not cancel pending https://hg.python.org/cpython/rev/d3804307cce4 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23095 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23095] [Windows] asyncio: race condition when cancelling a _WaitHandleFuture
STINNER Victor added the comment: It took me several months to understand this issue. For the beginning of the story, see: https://code.google.com/p/tulip/issues/detail?id=196 But I think that *this* issue can be closed: UnregisterWaitEx() really do what we need in asyncio. I don't like the complex IocpProactor._unregister() function and _WaitCancelFuture class, but it looks that it's how we are supposed to wait until a wait for a handle is cancelled... Windows IOCP API is much complex that what I expected. It's probably because some parts (especially RegisterWaitForSingleObject()) are implemented with threads in user land, not in the kernel. In short, I'm very happy that have fixed this very complex but also very annoying IOCP bug in asyncio. -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23095 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23095] [Windows] asyncio: race condition when cancelling a _WaitHandleFuture
Guido van Rossum added the comment: Congrats with the fix, and thanks for your perseverance! -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23095 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23095] [Windows] asyncio: race condition when cancelling a _WaitHandleFuture
STINNER Victor added the comment: IocpProactor.close() must not cancel pending _WaitCancelFuture futures FYI I found this bug when running the trollius test suite. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23095 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23095] [Windows] asyncio: race condition when cancelling a _WaitHandleFuture
Changes by STINNER Victor victor.stin...@gmail.com: -- title: asyncio: race condition when cancelling a _WaitHandleFuture - [Windows] asyncio: race condition when cancelling a _WaitHandleFuture ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23095 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
