[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Arkadiusz Miśkiewicz added the comment: So only this test (from first commit) should be added on master to check for further regressions in the area, right? -- ___ Python tracker

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

STINNER Victor added the comment: If you want to get a reliable behavior, don't rely on destructors. Python finalization is not determistic and destructors can be called while Python is not longer fully functional. Release ressources explicitly. For example, use multiprocessing.Pool with a

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Pablo Galindo Salgado added the comment: Ok, so sadly it seems that we need to close this issue as is fixed in master but the backport is risky. -- stage: patch review -> resolved status: open -> closed ___ Python tracker

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Change by Pablo Galindo Salgado : -- resolution: -> fixed ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

STINNER Victor added the comment: commit 4d96b4635aeff1b8ad41d41422ce808ce0b971c8 "bpo-39511: PyThreadState_Clear() calls on_delete (GH-18296)" "(...) The release_sentinel() function is now called when the C API is still fully working." This change is "safer" than the other one, but there is

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

STINNER Victor added the comment: > Victor, are you OK if we backport both changes to 3.8? Let me look at commit 9ad58acbe8b90b4d0f2d2e139e38bb5aa32b7fb6: "bpo-19466: Py_Finalize() clears daemon threads earlier (GH-18848)" Calling _PyThreadState_DeleteExcept() in Py_FinalizeEx() is really

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Pablo Galindo Salgado added the comment: > So 3.8 branch + 9ad58acbe8b90b4d0f2d2e139e38bb5aa32b7fb6 + > 4d96b4635aeff1b8ad41d41422ce808ce0b971c8 is working for me. I can confirm that these commits together fix the problem. Victor, are you OK if we backport both changes to 3.8? --

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Pablo Galindo Salgado added the comment: Can you try applying 4f384af067d05b16a554bfd976934fca9f87a1cf and 4d96b4635aeff1b8ad41d41422ce808ce0b971c8 together? -- ___ Python tracker

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Pablo Galindo Salgado added the comment: Oh, sorry, I missed your last message. -- ___ Python tracker ___ ___ Python-bugs-list

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Arkadiusz Miśkiewicz added the comment: This one is also needed on 3.8 to get it not hang with 1) test case. So 3.8 branch + 9ad58acbe8b90b4d0f2d2e139e38bb5aa32b7fb6 + 4d96b4635aeff1b8ad41d41422ce808ce0b971c8 is working for me. commit 4d96b4635aeff1b8ad41d41422ce808ce0b971c8 Author: Victor

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Arkadiusz Miśkiewicz added the comment: On master test 1) hangs before commit below and works after commit below. Unfortunately applying that commit to 3.8 branch doesn't help - 3.8 still hangs. Some other fix is also needed I guess commit 9ad58acbe8b90b4d0f2d2e139e38bb5aa32b7fb6 Author:

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Arkadiusz Miśkiewicz added the comment: master works fine 3.8 branch hangs so there is some other/related issue. -- ___ Python tracker ___

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Pablo Galindo Salgado added the comment: I tested it on the master branch with commit 5b66ec166b81c8a77286da2c0d17be3579c3069a -- ___ Python tracker ___

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Pablo Galindo Salgado added the comment: I still cannot reproduce it in your system: [test@ixion-pld cpython]$ hostname ixion-pld [test@ixion-pld cpython]$ uname -a Linux ixion-pld 5.5.2-1 #1 SMP Wed Feb 5 19:26:43 CET 2020 x86_64 Common_KVM_processor PLD Linux [test@ixion-pld cpython]$ cat

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Pablo Galindo Salgado added the comment: > ps. I'm arekm@freenode and can give ssh account for playing with the issue Can you send me an email to pablog...@gmail.com to set up ssh access? -- ___ Python tracker

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Arkadiusz Miśkiewicz added the comment: http://ixion.pld-linux.org/~arekm/python-gdb1.txt it's a result of thread apply all bt, thread apply all bt full, where pystack command shows nothing. ps. I'm arekm@freenode and can give ssh account for playing with the issue --

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

miss-islington added the comment: New changeset 7f5302fed4ff0cc8042e639b29a0664a16bc2702 by Miss Islington (bot) in branch '3.8': bpo-39360: Ensure all workers exit when finalizing a multiprocessing Pool (GH-19009)

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Pablo Galindo Salgado added the comment: Sadly, I am unable to reproduce, so without more clarification or a reliable reproducer, we cannot start debugging the issue. > For test 1) no traceback on ctrl+c and also no trace logged with faulthandler > installed, so no idea where it hangt. Can

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Arkadiusz Miśkiewicz added the comment: Ok, so two test cases 1) from multiprocessing.pool import ThreadPool class A(object): def __init__(self): self.pool = ThreadPool() def __del__(self): self.pool.close() self.pool.join() a = A() print(a) 2) from

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Pablo Galindo Salgado added the comment: I tried and I cannot reproduce the hang with the code provided in the first message here after PR 19009. (Tested on Linux, Windows and MacOS) -- ___ Python tracker

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Pablo Galindo Salgado added the comment: > Hummm, the test case from this bug is precisely the test case in PR 19009 so > it should not hang. Oh, actually I am wrong as this is a different issue I had in mind. PR 19009 should fix the case reported in https://bugs.python.org/issue38744 and

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Change by miss-islington : -- nosy: +miss-islington nosy_count: 5.0 -> 6.0 pull_requests: +18372 pull_request: https://github.com/python/cpython/pull/19023 ___ Python tracker

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Pablo Galindo Salgado added the comment: New changeset ac10e0c93218627d1a639db0b7b41714c5f6a883 by Batuhan Taşkaya in branch 'master': bpo-39360: Ensure all workers exit when finalizing a multiprocessing Pool (GH-19009)

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Pablo Galindo Salgado added the comment: > which looks a bit weird to me Is explained in the message in https://bugs.python.org/msg364211: What happens is that is moving that code so is executed in both code paths: explicit termination and multiprocessing finalization. > Unfortunately test

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Arkadiusz Miśkiewicz added the comment: (note, testing on 3.8 branch + pull request patch) -- ___ Python tracker ___ ___

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Arkadiusz Miśkiewicz added the comment: I've applied pull request patch (which looks a bit weird to me, removes something then adds the same thing back etc https://patch-diff.githubusercontent.com/raw/python/cpython/pull/19009.patch). Essentially the change is: diff --git

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Pablo Galindo Salgado added the comment: Check out also https://bugs.python.org/issue38744 -- ___ Python tracker ___ ___

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Change by Batuhan Taskaya : -- keywords: +patch nosy: +BTaskaya nosy_count: 4.0 -> 5.0 pull_requests: +18354 stage: -> patch review pull_request: https://github.com/python/cpython/pull/19009 ___ Python tracker

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Change by Arkadiusz Miśkiewicz : -- nosy: +arekm ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Pablo Galindo Salgado added the comment: This is not a regression: the code only worked on 3.7 because there is a sleep in the pool maintainance loop that makes the pool more likely being more consistent, but is the same problem as there is a race condition. For example, reducing the sleep

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

gaborbernat added the comment: Reported the issue downstream too under https://github.com/swagger-api/swagger-codegen/issues/9991 -- ___ Python tracker ___

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

STINNER Victor added the comment: This code snippet seems to trigger a crash in the _struct module: https://bugs.python.org/issue38076#msg360127 -- nosy: +vstinner ___ Python tracker

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

gaborbernat added the comment: An FYI update, the code is used by swagger-codegen to generate HTTP clients (https://github.com/swagger-api/swagger-codegen/blob/master/modules/swagger-codegen/src/main/resources/python/api_client.mustache#L73). The example below basically replicates creating

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Change by STINNER Victor : -- nosy: -vstinner ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

Change by gaborbernat : -- nosy: +pablogsal ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

New submission from gaborbernat : Assume the following code: ```python from multiprocessing.pool import ThreadPool class A(object): def __init__(self): self.pool = ThreadPool() def __del__(self): self.pool.close() self.pool.join() a = A() print(a) ``` The code