[issue43144] test_unicodedata: test_normalization uses network but doesn't depend on network resource
Arkadiusz Miśkiewicz added the comment: Sorry. Feel free to commit as own patch. It's trivial. -- ___ Python tracker <https://bugs.python.org/issue43144> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43144] test_unicodedata: test_normalization uses network but doesn't depend on network resource
Change by Arkadiusz Miśkiewicz : -- keywords: +patch Added file: https://bugs.python.org/file49792/unicodedata-no-network.patch ___ Python tracker <https://bugs.python.org/issue43144> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43144] test_unicodedata: test_normalization uses network but doesn't depend on network resource
New submission from Arkadiusz Miśkiewicz : Hello. test_normalization uses network but doesn't depend on network resource, so it tries to run even when network is disabled when running tests (-u-network) FAIL: test_normalization (test.test_unicodedata.NormalizationTest) socket.gaierror: [Errno -3] Temporary failure in name resolution urllib.error.URLError: AssertionError: Could not retrieve http://www.pythontest.net/unicode/13.0.0/NormalizationTest.txt -- components: Tests messages: 386554 nosy: arekm priority: normal severity: normal status: open title: test_unicodedata: test_normalization uses network but doesn't depend on network resource type: compile error versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue43144> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever
Arkadiusz Miśkiewicz added the comment: So only this test (from first commit) should be added on master to check for further regressions in the area, right? -- ___ Python tracker <https://bugs.python.org/issue39360> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever
Arkadiusz Miśkiewicz added the comment: This one is also needed on 3.8 to get it not hang with 1) test case. So 3.8 branch + 9ad58acbe8b90b4d0f2d2e139e38bb5aa32b7fb6 + 4d96b4635aeff1b8ad41d41422ce808ce0b971c8 is working for me. commit 4d96b4635aeff1b8ad41d41422ce808ce0b971c8 Author: Victor Stinner Date: Sat Feb 1 02:30:25 2020 +0100 bpo-39511: PyThreadState_Clear() calls on_delete (GH-18296) PyThreadState.on_delete is a callback used to notify Python when a thread completes. _thread._set_sentinel() function creates a lock which is released when the thread completes. It sets on_delete callback to the internal release_sentinel() function. This lock is known as Threading._tstate_lock in the threading module. The release_sentinel() function uses the Python C API. The problem is that on_delete is called late in the Python finalization, when the C API is no longer fully working. The PyThreadState_Clear() function now calls the PyThreadState.on_delete callback. Previously, that happened in PyThreadState_Delete(). The release_sentinel() function is now called when the C API is still fully working. -- ___ Python tracker <https://bugs.python.org/issue39360> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever
Arkadiusz Miśkiewicz added the comment: On master test 1) hangs before commit below and works after commit below. Unfortunately applying that commit to 3.8 branch doesn't help - 3.8 still hangs. Some other fix is also needed I guess commit 9ad58acbe8b90b4d0f2d2e139e38bb5aa32b7fb6 Author: Victor Stinner Date: Mon Mar 9 23:37:49 2020 +0100 bpo-19466: Py_Finalize() clears daemon threads earlier (GH-18848) Clear the frames of daemon threads earlier during the Python shutdown to call objects destructors. So "unclosed file" resource warnings are now emitted for daemon threads in a more reliable way. Cleanup _PyThreadState_DeleteExcept() code: rename "garbage" to "list". -- ___ Python tracker <https://bugs.python.org/issue39360> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever
Arkadiusz Miśkiewicz added the comment: master works fine 3.8 branch hangs so there is some other/related issue. -- ___ Python tracker <https://bugs.python.org/issue39360> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever
Arkadiusz Miśkiewicz added the comment: http://ixion.pld-linux.org/~arekm/python-gdb1.txt it's a result of thread apply all bt, thread apply all bt full, where pystack command shows nothing. ps. I'm arekm@freenode and can give ssh account for playing with the issue -- ___ Python tracker <https://bugs.python.org/issue39360> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever
Arkadiusz Miśkiewicz added the comment: Ok, so two test cases 1) from multiprocessing.pool import ThreadPool class A(object): def __init__(self): self.pool = ThreadPool() def __del__(self): self.pool.close() self.pool.join() a = A() print(a) 2) from multiprocessing import Pool class A: def __init__(self): self.pool = Pool(processes=1) def test(): problem = A() problem.pool.map(float, tuple(range(10))) if __name__ == "__main__": test() On two my test environments with Linux, 5.5.2 and 4.9.208 kernel, glibc 2.31 (userspace is mostly the same on both) test 1) hangs, 2) doesn't hang. Testing done on 3.8 branch + https://patch-diff.githubusercontent.com/raw/python/cpython/pull/19023.patch For test 1) no traceback on ctrl+c and also no trace logged with faulthandler installed, so no idea where it hangt. strace shows [...] [pid 232031] munmap(0x7f1baaffe000, 8392704 [pid 232017] write(5, "\0\0\0\4\200\4N.", 8 [pid 232031] <... munmap resumed>) = 0 [pid 232017] <... write resumed>) = 8 [pid 232031] exit(0)= ? [pid 232017] futex(0x7f1b94000b60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY [pid 232031] +++ exited with 0 +++ ^C * ctrl + c pressed [pid 232017] <... futex resumed>) = ? ERESTARTSYS (To be restarted if SA_RESTART is set) strace: Process 232017 detached strace: Process 232032 detached strace: Process 232033 detached strace: Process 232034 detached strace: Process 232035 detached -- ___ Python tracker <https://bugs.python.org/issue39360> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever
Arkadiusz Miśkiewicz added the comment: (note, testing on 3.8 branch + pull request patch) -- ___ Python tracker <https://bugs.python.org/issue39360> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever
Arkadiusz Miśkiewicz added the comment:
I've applied pull request patch (which looks a bit weird to me, removes
something then adds the same thing back etc
https://patch-diff.githubusercontent.com/raw/python/cpython/pull/19009.patch).
Essentially the change is:
diff --git a/Lib/multiprocessing/pool.py b/Lib/multiprocessing/pool.py
index b223d6aa72..8bd9608b0e 100644
--- a/Lib/multiprocessing/pool.py
+++ b/Lib/multiprocessing/pool.py
@@ -682,7 +682,15 @@ class Pool(object):
# this is guaranteed to only be called once
util.debug('finalizing pool')
-worker_handler._state = TERMINATE
+# Explicitly do the cleanup here if it didn't come from terminate()
+# (required for if the queue will block, if it is already closed)
+if worker_handler._state != TERMINATE:
+# Notify that the worker_handler state has been changed so the
+# _handle_workers loop can be unblocked (and exited) in order to
+# send the finalization sentinel all the workers.
+worker_handler._state = TERMINATE
+change_notifier.put(None)
+
task_handler._state = TERMINATE
util.debug('helping task handler/workers to finish')
Unfortunately test case from first message in this bug report still hangs for
me with this applied.
--
___
Python tracker
<https://bugs.python.org/issue39360>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever
Change by Arkadiusz Miśkiewicz : -- nosy: +arekm ___ Python tracker <https://bugs.python.org/issue39360> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38744] python 3.8 hang in multiprocessing.Pool() locking on FreeBSD / Linux
Arkadiusz Miśkiewicz added the comment: bpo-35493 -- ___ Python tracker <https://bugs.python.org/issue38744> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit
Arkadiusz Miśkiewicz added the comment: And also https://bugs.python.org/issue38744 on Linux and FreeBSD -- nosy: +arekm ___ Python tracker <https://bugs.python.org/issue35493> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38744] python 3.8 hang in multiprocessing.Pool() locking on FreeBSD / Linux
Arkadiusz Miśkiewicz added the comment: Bisecting led to this commit and reverting it on 3.8 branch makes the hang go away. 7c994549dcffd0d9d3bb37475e6374f356e7240e is the first bad commit commit 7c994549dcffd0d9d3bb37475e6374f356e7240e Author: Pablo Galindo Date: Sat Mar 16 22:34:24 2019 + bpo-35493: Use Process.sentinel instead of sleeping for polling worker status in multiprocessing.Pool (#11488) * bpo-35493: Use Process.sentinel instead of sleeping for polling worker status in multiprocessing.Pool * Use self-pipe pattern to avoid polling for changes * Refactor some variable names and add comments * Restore timeout and poll * Use reader object only on wait() * Recompute worker sentinels every time * Remove timeout and use change notifier * Refactor some methods to be overloaded by the ThreadPool, document the cache class and fix typos Lib/multiprocessing/pool.py| 88 +++--- .../2019-01-09-23-43-08.bpo-35493.kEcRGE.rst | 3 + 2 files changed, 80 insertions(+), 11 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2019-01-09-23-43-08.bpo-35493.kEcRGE.rst -- ___ Python tracker <https://bugs.python.org/issue38744> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
